LiteLLM solves the model management problem by acting as a universal proxy, while Elasticsearch delivers high-performance Vector Search. By combining them, you can build a search architecture that is both flexible and powerful.

In this guide, we will walk through hosting an OpenAI-compatible embedding model using LiteLLM on Docker and consuming it directly from Elasticsearch to perform seamless vector search.

Prerequisites

• OS: Ubuntu 24.04.3 LTS (or similar Linux environment)
• Docker: Ensure Docker is installed and running.
• Elasticsearch: An Elastic Cloud instance or a self-managed cluster.

Step 1: Configure and Run LiteLLM

First, we need to configure LiteLLM. This tool will act as our gateway, proxying requests to various models (like GPT-4 or specific embedding models) while presenting a unified OpenAI-compatible API to the outside world.

1. Create the config.yaml file:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: os.environ/OPENAI_API_KEY

  - model_name: text-embedding-3-small
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY

2. Run LiteLLM with Docker:

We will run the container with the config mounted. We also define a LITELLM_MASTER_KEY (sk-my-admin-key-123), which will serve as the static API key for our internal services to authenticate against this proxy.

docker run -d \
  -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -e OPENAI_API_KEY="enter your openai api key" \
  -e LITELLM_MASTER_KEY="sk-my-admin-key-123" \
  docker.litellm.ai/berriai/litellm:main-latest \
  --config /app/config.yaml --detailed_debug

Step 2: Verify the Proxy

let’s ensure our LiteLLM proxy is correctly handling requests.

Test Chat Completion:

curl http://0.0.0.0:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-my-admin-key-123" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      { "role": "user", "content": "Hello from LiteLLM Docker!" }
    ]
  }'

Test Embedding Generation:

curl http://0.0.0.0:4000/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-my-admin-key-123" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "LiteLLM makes proxying easy."
  }'

Step 3: Configure Elasticsearch Inference

Now we connect the two pieces. Let’s create elatic deployment instance from Elastic cloud.

Since LiteLLM provides an OpenAI-compatible interface, we can use the standard OpenAI inference in Elasticsearch but point it to our custom LiteLLM URL.

For more details on configuring these connectors, refer to the API doc

Create the Inference Endpoint

Run the following in your Kibana Dev Tools. Note that the url points to our LiteLLM endpoint and the api_key matches our LiteLLM master key.

PUT _inference/text_embedding/litellm-embeddings
{
   "service": "openai",
   "service_settings": {
       "api_key": "sk-my-admin-key-123",
       "model_id": "text-embedding-3-small",
       "url": "http://litellm_endpoint:4000/v1/embeddings",
       "dimensions": 128
   }
}

Output

{
  "inference_id": "litellm-embeddings",
  "task_type": "text_embedding",
  "service": "openai",
  "service_settings": {
    "model_id": "text-embedding-3-small",
    "url": "http://litellm_endpoint:4000/v1/embeddings",
    "similarity": "dot_product",
    "dimensions": 128,
    "rate_limit": {
      "requests_per_minute": 3000
    }
  },
  "chunking_settings": {
    "strategy": "sentence",
    "max_chunk_size": 250,
    "sentence_overlap": 1
  }
}

Test the Inference Endpoint

Let’s verify that Elasticsearch can generate embeddings via LiteLLM:

POST _inference/text_embedding/litellm-embeddings
{
 "input": "The sky above the port was the color of television tuned to a dead channel."
}

Output:

{
  "text_embedding": [
    {
      "embedding": [
        -0.02372423,
        -0.016345346,
        -0.269933,
        -0.038996283,
        0.06547082,
        -0.04299877,
        0.059388835,
        -0.13049445,
        -0.019900626,
        0.09131928,
        0.024954043,
        0.08608698,
        -0.072268344,
        0.09623853,
        0.06260871,
        0.0264969,
        0.21161744,
        0.00086855615,
        -0.2389864,
        -0.07410188,
        -0.00018866465,
        -0.0704348,
        -0.08724971,
        0.14078017,
        -0.121192575,
        0.011660873,
        0.07177641,
        0.070211194,
        -0.01770932,
        0.13067332,
        0.02551305,
        -0.16734414,
        0.0144000035,
        -0.033562742,
        0.12226587,
        -0.097311825,
        0.08868077,
        -0.011236028,
        -0.08631058,
        -0.13541369,
        0.007585716,
        -0.08742859,
        0.19337147,
        0.072447225,
        -0.025155285,
        -0.07025592,
        -0.0071552815,
        0.03591057,
        0.118509345,
        0.028598765,
        -0.094628595,
        0.051115543,
        -0.03497144,
        0.053888213,
        0.046196286,
        -0.091945365,
        -0.013449693,
        -0.025759013,
        0.13764973,
        0.046554048,
        0.08841244,
        0.05460374,
        -0.025826093,
        0.030611187,
        -0.012197519,
        -0.1722634,
        -0.02372423,
        -0.0021340067,
        -0.025222367,
        0.06158014,
        0.09740127,
        -0.010352798,
        -0.05701865,
        -0.06381617,
        -0.0127341645,
        0.042149078,
        0.034144107,
        -0.012432301,
        0.1711901,
        -0.13112053,
        -0.0789317,
        0.12879506,
        -0.14677271,
        -0.02915777,
        -0.0157528,
        0.007390064,
        -0.04659877,
        -0.10473543,
        0.04152299,
        0.16662861,
        0.017060874,
        0.12333916,
        -0.06256399,
        -0.09167704,
        -0.023120502,
        0.03872796,
        -0.17852427,
        -0.020314291,
        0.023232304,
        -0.050578896,
        0.05983604,
        0.020470813,
        -0.14417891,
        -0.0002360055,
        -0.0051400634,
        0.07334163,
        0.12655903,
        0.0008245344,
        -0.057644736,
        -0.067080766,
        -0.02989566,
        -0.04223852,
        -0.14838265,
        0.1329988,
        0.009531058,
        0.038638517,
        0.08076524,
        -0.0014939444,
        0.24846715,
        -0.012711804,
        -0.003342858,
        0.014411184,
        -0.077232316,
        0.033629823,
        0.082598776,
        -0.020850938,
        0.08376151,
        0.12772177
      ]
    }
  ]
}

Step 4: End-to-End Vector Search

Now that the pipeline is ready, we can perform Semantic Search. We will use the (semantic_text)[https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text] field type, which abstracts the embedding generation process—Elasticsearch calls LiteLLM automatically during data ingestion and query time.

1. Create the Index

PUT /litellm-test-index
{
 "mappings": {
   "properties": {
     "content": {
       "type": "semantic_text",
       "inference_id": "litellm-embeddings"
     },
     "category": {
       "type": "keyword"
     }
   }
 }
}

2. Ingest Data

POST /litellm-test-index/_bulk
{ "index": {} }
{ "content": "The quick brown fox jumps over the lazy dog.", "category": "animals" }
{ "index": {} }
{ "content": "Artificial intelligence is transforming software development.", "category": "tech" }
{ "index": {} }
{ "content": "Docker containers make deployment consistent and easy.", "category": "tech" }

3. Verify Data

You can now search the index.

GET litellm-test-index/_search

Note: You will not see the embedding vectors in the _source output. This is by design to save storage space. To learn more about this behavior, check out the Elastic Search Labs blog on excluding vectors from source.

4. Perform Semantic Query

Finally, let’s run a natural language search. The query “tools for deploying software” does not exist in our text, but the vector search identifies the semantic relationship with “Docker containers.”

GET /litellm-test-index/_search
{
 "query": {
   "semantic": {
     "field": "content",
     "query": "tools for deploying software"
   }
 }
}

Output

{
  "took": 501,
  "timed_out": false,
  "_shards": {
    "total": 10,
    "successful": 10,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 0.7422092,
    "hits": [
      {
        "_index": "litellm-test-index",
        "_id": "UZR4pJsBGjew_vztklFP",
        "_score": 0.7422092,
        "_source": {
          "content": "Docker containers make deployment consistent and easy.",
          "category": "tech"
        }
      },
      {
        "_index": "litellm-test-index",
        "_id": "UJR4pJsBGjew_vztklFP",
        "_score": 0.68897665,
        "_source": {
          "content": "Artificial intelligence is transforming software development.",
          "category": "tech"
        }
      },
      {
        "_index": "litellm-test-index",
        "_id": "T5R4pJsBGjew_vztklFP",
        "_score": 0.5035514,
        "_source": {
          "content": "The quick brown fox jumps over the lazy dog.",
          "category": "animals"
        }
      }
    ]
  }
}

Conclusion

This architecture demonstrates the power of decoupling your model provider from your search engine. By using LiteLLM as a standardized proxy, you gain the flexibility to swap underlying models without breaking your application. Simultaneously, Elasticsearch’s Vector Search leverages these embeddings to provide deep semantic understanding.

Together, LiteLLM and Elasticsearch create a robust, future-proof foundation for building intelligent search applications that go far beyond simple keyword matching.

Happy searching!

Everyone’s talking about hybrid search right now. But here’s the uncomfortable truth:

👉 Just because you glued vector search onto your database and added metadata filters doesn’t mean you’ve built true hybrid search.

That’s like duct-taping a spoiler on a hatchback and calling it a race car. 🚗💨

Hybrid search is more than just “keyword + vector + filter.” It’s about field-level design, reranking, scoring, and scale.

Let’s break this down.

Field-Level Truth: Not Every Field Deserves Semantic Search

The biggest mistake I see: Teams running semantic + lexical search on every field in their JSON docs.

That’s how you kill relevance.

Lexical-Only Fields

These work best with exact match:

• title
• name
• issue_id
• category
• tags

If I search for issue_id: 1245, I want 1245 — not “similar looking IDs.”

Semantic-Only Fields

These benefit from embeddings:

• product_description
• movie_storyline
• customer_feedback
• reviews

This is where meaning > keywords. “Battery dies too quickly” should match with “poor battery life.”

The Formula That Works

Final Score = RRF(
   Lexical(title, name, category) +
   Semantic(description, feedback, reviews)
)

Hybrid works when each field is treated for what it is.

Quick Reference Table

Metadata Filters: Keep Them Lexical

One of the biggest anti-patterns: vectorizing filters.

Filters are not semantic. They should remain exact.

• Price → numeric filters
• Stock availability → boolean
• Categories, IDs → keyword
• Timestamps, geo → structured fields

Search is about precision here. If you fuzzy up filters with embeddings, you’ll tank user trust.

Hybrid ≠ Metadata Filters + Vectors

Here’s the thing. Some vector DBs shout “we support keyword search now!” But if you look closer, it’s often:

• A bolted-on library
• Or patched-in metadata filters

Yeah, technically it works. But that’s not hybrid.

Search is not just filters.

What Real Hybrid Search Looks Like

Let’s talk the real stuff beyond filters + vectors:

• Autocomplete — search-as-you-type, phrase suggesters, boosting, user-friendly completion.
• Facets — powerful aggregations: geo boundaries, date histograms, bucketing. Not just string counts.
• Native rescoring — query rescorer, rank features, boosting for freshness, popularity, personalization.
• Rich documents — nested JSON, arrays, geo points, IPs. Search isn’t flat.
• Geo Search — aggregations, polygons, proximity scoring. Beyond just “in radius.”
• Access control — index → document → field level security. For lexical and vector fields.
• Data enrichment pipelines — entity extraction, tagging, embeddings, LLM-based enrichment.
• Scalability — petabytes of data, tiered storage, lifecycle management. Search doesn’t stop at 10M docs.

This is hybrid.

Vectors: Ask the Hard Questions

Not all vector support is equal. Check:

• Is vector search native to the engine or bolted on?
• Query types: KNN, ANN, hybrid filtering?
• Can it run at scale with good recall + low latency?
• Quantization: int4, int8, binary, advanced methods like BBQ (Better Binary Quantization)?
• Filters on HNSW: is it pre-filtering or filter-aware indexing (like ACORN)?
• Can it blend semantic + lexical scoring at query time?
  • RRF (Reciprocal Rank Fusion)
  • Linear retrievers
  • Semantic reranking with LLMs

If your engine can’t do these, it’s not serious about hybrid.

Implementation Note: One Query Should Do It

If you need to stitch 3–4 services together just to get:

• Facets
• Filters
• Semantic results
• Lexical results
• Reranking

…you’re adding latency + complexity.

A true hybrid engine should give you everything in one structured query.

Practical Takeaways

1. Not all fields deserve semantic. Treat fields differently.
2. Filters are lexical. Stop semantic-izing metadata.
3. Hybrid = lexical + semantic + rescoring + scale. Not just bolted-on keyword support.
4. Evaluate vectors deeply. Native support, quantization, filtering, reranking.
5. One query > stitched services. Hybrid done right = clean pipeline, low latency.

Closing Thoughts

Hybrid search is not a checkbox. It’s about designing for relevance + scale + control.

If your “hybrid” is just vectors + filters, you’ve built a demo — not a real search engine.

True hybrid is Lexical + Semantic + Scoring + Access + Scale + Control. That’s when search feels natural, accurate, and production-ready.

Speaker: Ashish Diwali (Senior Developer Advocate, Elastic)

🔑 Introduction

• Topic: Integrating Generative AI (LLMs) with PHP.
• Goal: Show how to build chat assistants, semantic search, and vector search without heavy ML expertise.
• Demo focus: Using Elasticsearch + PHP + LLM (LLaMA 3.1).

🧩 Core Concepts

1. Prompt Engineering

• LLMs generate responses based on prompts → predicting next words.
• Techniques:
  • Zero-shot inference → direct classification or tagging.
  • One-shot inference → provide one example in the prompt.
  • Few-shot inference → multiple examples → useful for structured outputs (SQL, JSON, XML).
• Iteration + context = In-context learning (ICL).

2. LLM Limitations

• ❌ Hallucinations (wrong answers).
• ❌ Complex to build/train from scratch.
• ❌ No real-time / private data access.
• ❌ Privacy & security concerns (especially in banking, public sector).

3. RAG (Retrieval-Augmented Generation)

• Solution to limitations.
• Workflow:
  1. User query → hits database/vector DB (e.g., Elasticsearch).
  2. Retrieve top 5–10 relevant docs.
  3. Pass as context window → LLM generates accurate answer.
• Benefits:
  • Grounded responses.
  • Works with private data.
  • Avoids retraining large models.

🔍 Semantic & Vector Search

• Semantic Search: Understands meaning, not just keywords.
  • Example: “best city” ↔ “beautiful city.”
• Vector Search: Text, images, and audio converted into embeddings (arrays of floats).
  • Enables image search, recommendation systems, music search (via humming).
• Similarity algorithms: cosine similarity, dot product, nearest neighbors.

🛠️ Tools & Demo

Elephant Library (PHP)

• Open-source PHP library for GenAI apps.
• Supports:
  • LLMs: OpenAI, Mistral, Anthropic, LLaMA.
  • Vector DBs: Elasticsearch, Pinecone, Chroma, etc.
  • Features: document chunking, embedding generation, semantic retrieval, Q&A (RAG).

Demo Flow

1. Ingestion:

   • Chunk PDF into smaller pieces (800 chars).
   • Generate embeddings with LLaMA.
   • Store text + vectors in Elasticsearch.

2. Querying:

   • User question → hits Elasticsearch.
   • Retrieve top 10 docs.
   • Send docs + query → LLaMA → response.

3. Examples:

   • “Who won the Nobel Prize in Physics 2024?” → Retrieved correct answer from PDF context.
   • “How do brain neural networks work?” → Summarized based on provided docs.
   • “Who won ICC Championship 2025?” → No irrelevant hallucination (kept within context).

🎯 Key Takeaways

• Don’t train your own LLM → use RAG + search to build assistants on private data.
• Elasticsearch is a powerful vector DB for semantic + hybrid search.
• PHP + Elephant makes building RAG chatbots accessible for web developers.
• RAG powers most modern chat assistants today.

Speaker: Ashish (Senior Developer Advocate, Elastic)
Event: AWS Community Day Mumbai 2024

🔑 Why Search Still Matters with LLMs

• LLMs (like ChatGPT) are powerful but face:
  • ❌ Hallucinations
  • 💰 High cost per query
  • 🔒 No access to private / real-time data
• ✅ Search grounds LLMs in reliable, domain-specific info.

⚡ Elasticsearch Capabilities

• Traditional keyword search + modern vector search.
• Real-world use cases:
  • 📍 Geospatial queries (ride-sharing, food delivery)
  • ❤️ Matchmaking
  • 📊 Observability dashboards
  • 📝 Centralized logging (Elastic Stack: Elasticsearch, Kibana, Beats, Logstash)

🤖 Retrieval-Augmented Generation (RAG)

Workflow:

1. User query →
2. Elasticsearch retrieves relevant documents →
3. Context passed to AWS Bedrock LLM →
4. LLM generates grounded answers

👉 Reduces hallucination, enables chatbots on private data.

🛠️ Demo Stack

• Flowise AI → No-code, drag & drop RAG builder
• Steps:
  1. Ingest data (JSON, PDFs, web crawl)
  2. Chunk text
  3. Generate embeddings (Amazon Titan)
  4. Store embeddings + text in Elasticsearch
  5. Connect Bedrock LLM for Q&A

🌐 Extensions

• Text search ✅
• Image search 🖼️
• Audio search (e.g., humming a tune) 🎵
• Multimodal experiences 🤝

🎯 Key Takeaway

With Elasticsearch + AWS Bedrock + Flowise AI, developers can build domain-specific, no-code RAG chatbots that combine:

• The precision of search 🔍
• The fluency of LLMs 💬

Talk video

Note - This gist is not recommended for production multinode cluster. As I only tested on single node cluster. If you have a multi-node cluster please follow the official guide.

Elasticsearch Installation

1. I created a VM instance on GCP. Here are the details -

OS: Ubuntu 22.04
Machine type: e2-standard-4
Configuration: 4 CPU, 16 GB RAM

2. Configure static external IP for newly created VM or ensure your VM is accessible from the public IP.

3. Installed Elasticsearch from Archive.

You can verify if Elasticsearch is running or not.

Get into the $ES_HOME folder -

cd elasticsearch-{version}/

In my case it was

cd elasticsearch-8.15.3/

Default TLS setup

When you install Elasticsearch, By default it will generate a TLS certificate for the host localhost and IP 127.0.0.1.

ls config/certs/

http.p12  http_ca.crt  transport.p12

CURL call -

curl --cacert $ES_HOME/config/certs/http_ca.crt -u elastic:pass https://localhost:9200

OR

curl --cacert $ES_HOME/config/certs/http_ca.crt -u elastic:pass https://127.0.0.1:9200

Response

{
  "name" : "benchmark1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "kqZsQ5WrTAWnlFmYbyMQWw",
  "version" : {
    "number" : "8.15.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "f97532e680b555c3a05e73a74c28afb666923018",
    "build_date" : "2024-10-09T22:08:00.328917561Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

If you run the same CURL command using the public IP of the VM, It won’t work.

curl --cacert $ES_HOME/config/certs/http_ca.crt -u elastic:pass https://x.x.x.x:9200

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

It is giving an error because the certificate is not valid for the IP / Domain. However, you can call insecurely by adding -k which is not recommended for production use.

curl -k --cacert $ES_HOME/config/certs/http_ca.crt -u elastic:pass https://x.x.x.x:9200

Generate TLS certificate for the Public IP / Domain

1. Let’s generate the new local certificate authority CA)

./bin/elasticsearch-certutil ca

This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]: 
Enter password for elastic-stack-ca.p12 :

I kept blank for the last two questions. But you can change the output file and set the Password.

It must have created a file elastic-stack-ca.p12 in $ES_HOME directory.

2. Generate the Certificate using newly created CA

./bin/elasticsearch-certutil http

It will ask some questions -

1. Generate a CSR? [y/N] n
2. Use an existing CA? [y/N] y
3. CA Path: $ES_HOME/elastic-stack-ca.p12

You need to provide the absolute path for elastic-stack-ca.p12.

4. Password for elastic-stack-ca.p12:

Type the password if you had set in Step 1. In my case, I’ll keep it blank.

5. For how long should your certificate be valid? [5y]

6. Generate a certificate per node? [y/N] y

7. node #1 name: benchmakr1.node

8. Which hostnames will be used to connect to benchmakr1.node?

yourdomain.name  
localhost

9. Which IP addresses will be used to connect to benchmakr1.node?

127.0.0.1  
x.x.x.x

Add your public IP list.

10. Do you wish to change any of these options? [y/N] n
11. Generate additional certificates? [Y/n] n
12. Provide a password for the “http.p12” file:

Type the password which you want to set and save it. I kept it as a blank.

13. What filename should be used for the output zip file?

It must have created elasticsearch-ssl-http.zip file

3. Set http.p12

Lets unzip

mkdir new-ssl
mv elasticsearch-ssl-http.zip new-ssl/
cd new-ssl/
unzip elasticsearch-ssl-http.zip

This will create two directory -

1. elasticsearch
2. kibana

go to the $ES_HOME directory

cd ..

Replace old http.p12 with new.

cp new-ssl/elasticsearch/http.p12 config/certs/http.p12

4. Change password for your private key http.p12

./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password

Setting xpack.security.http.ssl.keystore.secure_password already exists. Overwrite? [y/N] y

Enter value for xpack.security.http.ssl.keystore.secure_password:

Kept it as a blank in my case. But if you have set any password on Step 2 (Question no. 12), Please enter the same password.

5. Restart the Elasticsearch and verify

Restart the Elasticsearch.

Let’s make the same CURL call with the newly generated CA in kibana folder

curl --cacert $ES_HOME/new-ssl/kibana/elasticsearch-ca.pem -u elastic:pass https://x.x.x.x:9200

{
  "name" : "benchmark1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "kqZsQ5WrTAWnlFmYbyMQWw",
  "version" : {
    "number" : "8.15.3",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "f97532e680b555c3a05e73a74c28afb666923018",
    "build_date" : "2024-10-09T22:08:00.328917561Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

This gist for single node cluster. But if you’re looking to secure your cluster end to end, Go through the complete guide.

Serverless architectures take on-demand tasks to the next level with event-driven scheduling of workloads. Elastic Observability gives you the same insights into your serverless activities as the rest of your environment. Gather logs and metrics from your serverless invocations and tie them together with traces from your serverless functions.

For example Identify AWS Lambda latency issues, cold starts, and other invocation issues. Logs are collected with the rest of your telemetry data, so you can look at all your data in context, in one place.

Talk video

In today’s data-driven world, just having a search engine is not enough; the key is making it smart. Enter Elasticsearch Relevance Engine (ESRE) augmented with Retrieval Augmented Generation (RAG), a powerful solution that marries Elasticsearch’s superior search capabilities with Large Language Models (LLMs) like ChatGPT for precise, contextual querying over proprietary datasets. This session is a hands-on guide that will show you how to amplify the power of Elasticsearch with advanced LLMs.

Key Takeaways:

• Learn how to supercharge Elasticsearch’s BM25 algorithm with semantic search for results that are not just relevant but contextually accurate.
• Discover how to plug in Large Language Models like OpenAI’s ChatGPT to enable context-aware question-answering over your proprietary data.
• Gain insights into the latest advancements in vector search within Lucene and Elasticsearch.
• A quick live demo: Experience first-hand how ESRE, empowered by RAG, transforms a basic search query into a context-rich, highly relevant result..

This talk is for you if you’re grappling with search relevance issues and are looking for innovative ways to make your search smarter and more efficient. Whether you’re a software developer, data engineer, or ML enthusiast, this session will equip you with the skills you need to build next-generation search capabilities.

Talk video

movies.json

[
  {
    "title_x": "Student of the Year 2",
    "imdb_id": "tt7255568",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/thumb/3/3c/Student_of_the_year_2_Poster.jpg/220px-Student_of_the_year_2_Poster.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/Student_of_the_Year_2",
    "title_y": "Student of the Year 2",
    "original_title": "Student of the Year 2",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "146",
    "genres": "Drama|Romance|Sport",
    "imdb_rating": 2.5,
    "imdb_votes": 12671,
    "story": "A student must face off against bullies and overcome hurdles  both academic and romantic  to win his college's coveted Student of the Year trophy.",
    "summary": "A student must face off against bullies and overcome hurdles  both academic and romantic  to win his college's coveted Student of the Year trophy.",
    "tagline": "",
    "actors": "Tiger Shroff|Ananya Panday|Tara Sutaria|Aditya Seal|Manoj Pahwa|Ayesha Raza|Rajesh Kumar|Manasi Joshi Roy|Samir Soni|Gul Panag|Manjot Singh|Sahil Anand|Harsh Beniwal|Daljeet Singh Gujral|",
    "wins_nominations": "",
    "release_date": "10 May 2019 (USA)"
  },
  {
    "title_x": "De De Pyaar De",
    "imdb_id": "tt8647400",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/thumb/c/c4/De_De_Pyaar_De_Poster.jpg/220px-De_De_Pyaar_De_Poster.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/De_De_Pyaar_De",
    "title_y": "De De Pyaar De",
    "original_title": "De De Pyaar De",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "135",
    "genres": "Comedy|Romance",
    "imdb_rating": 6.6,
    "imdb_votes": 5387,
    "story": "a 50 year old Aashish living London away from his family fro many years He falls in love with a 26 year Ayesha.Thou officially not divorced from his first wife Manju.Aashish wants her permission to get married to Ayesha.Upon reaching his house in India  Ayesha finds that Aashish kids are of the same age as her.Aashish introduces Ayesha as his secretary to his family he also finds that his daughter is about to get married and VK whose become close family friend is trying to woo her Manju.",
    "summary": "A 50-year-old single father faces disapproval from his family and his ex-wife when he falls in love with a 26-year-old woman.",
    "tagline": "",
    "actors": "Ajay Devgn|Tabu|Rakul Preet Singh|Jimmy Sheirgill|Alok Nath|Madhumalti Kapoor|Harry Anton|Gergo Baranyi|Sammy John Heaney|Bhavin Bhanushali|Gandharv Dewan|Gulnaaz Khan|Kumud Mishra|Sunny Singh Nijjar|",
    "wins_nominations": "",
    "release_date": "17 May 2019 (USA)"
  },
  {
    "title_x": "India's Most Wanted (film)",
    "imdb_id": "tt8484942",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/a/ae/India%27s_Most_Wanted_poster.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/India%27s_Most_Wanted_(film)",
    "title_y": "India's Most Wanted",
    "original_title": "India's Most Wanted",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "123",
    "genres": "Action|Thriller",
    "imdb_rating": 4.4,
    "imdb_votes": 1449,
    "story": "India's Most Wanted is a Bollywood action thriller film directed by Raj Kumar Gupta and starring Arjun Kapoor. The film is about tracking a terrorist in a secret mission and arresting him without firing bullets. It pays tribute to unsung heroes of our society.",
    "summary": "A group of intelligence officers embark on a top secret mission to track down a wanted international criminal.",
    "tagline": "",
    "actors": "Arjun Kapoor|Sudev Nair|Rajesh Sharma|Prasanth|Santilal Mukherjee|",
    "wins_nominations": "",
    "release_date": "24 May 2019 (USA)"
  },
  {
    "title_x": "Yeh Hai India",
    "imdb_id": "tt5525846",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/thumb/d/db/Yeh_Hai_India_Official_Poster.jpg/220px-Yeh_Hai_India_Official_Poster.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/Yeh_Hai_India",
    "title_y": "Yeh Hai India",
    "original_title": "Yeh Hai India",
    "is_adult": 0,
    "year_of_release": "2017",
    "runtime": "128",
    "genres": "Action|Adventure|Drama",
    "imdb_rating": 5.7,
    "imdb_votes": 169,
    "story": "Yeh Hai India  follows the story of a 25 years old NRI  who is born and brought up in U.K and shares the same stereotype views of India  which is known for its vast population  pollution and poverty. However protagonist finds new development in media or probably an 'other side of same coin' of India  which is also known for successful mars mission in first attempt  a nation which proudly holds title 'God of Cricket' for Sachin Tendulkar  who is again an Indian and a nation which is known for its holy generosity with icons like Mother Teresa\".",
    "summary": "Yeh Hai India  follows the story of a 25 years old NRI  who is born and brought up in UK. He shares the same stereotypical views about India that most NRIs have and how his perception changes.",
    "tagline": "A Film for Every Indian",
    "actors": "Gavie Chahal|Mohan Agashe|Mohan Joshi|Lom Harsh|",
    "wins_nominations": "2 wins & 1 nomination",
    "release_date": "24 May 2019 (India)"
  },
  {
    "title_x": "Kabir Singh",
    "imdb_id": "tt8983202",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/thumb/d/dc/Kabir_Singh.jpg/220px-Kabir_Singh.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/Kabir_Singh",
    "title_y": "Kabir Singh",
    "original_title": "Kabir Singh",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "173",
    "genres": "Drama|Romance",
    "imdb_rating": 7.2,
    "imdb_votes": 20535,
    "story": "This Sandeep Vanga directorial is a remake of the 2017 Telugu movie Arjun Reddy. The plot revolves around an alcoholic surgeon battling temper issues. Things get worse as he watches the love of his life marry another man. High on emotions coupled with impressive action the film promises an intense watch.",
    "summary": "Kabir Singh is a remake of a Telugu movie Arjun Reddy (2017)  where a short-tempered house surgeon gets used to drugs and drinks when his girlfriend is forced to marry another person.",
    "tagline": "",
    "actors": "Shahid Kapoor|Kiara Advani|Soham Majumdar|Arjan Bajwa|Suresh Oberoi|Kamini Kaushal|Adil Hussain|Nikita Dutta|Amit Sharma|Anurag Arora|Dolly Mattoo|Aanchal Chauhan|Suparna Marwah|Swati Seth|",
    "wins_nominations": "",
    "release_date": "20 June 2019 (USA)"
  },
  {
    "title_x": "Article 15 (film)",
    "imdb_id": "tt10324144",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/thumb/1/11/Article_15_Poster.jpg/220px-Article_15_Poster.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/Article_15_(film)",
    "title_y": "Article 15",
    "original_title": "Article 15",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "130",
    "genres": "Crime|Drama",
    "imdb_rating": 8.3,
    "imdb_votes": 13417,
    "story": "In the rural heartlands of India  an upright police officer sets out on a crusade against violent caste-based crimes and discrimination.",
    "summary": "In the rural heartlands of India  an upright police officer sets out on a crusade against violent caste-based crimes and discrimination.",
    "tagline": "Farq Bahut Kar Liya| Ab Farq Laayenge.",
    "actors": "Ayushmann Khurrana|Nassar|Manoj Pahwa|Kumud Mishra|Isha Talwar|Sayani Gupta|Mohammed Zeeshan Ayyub|Subhrajyoti Barat|Sushil Pandey|Aakash Dabhade|Ashish Verma|Ronjini Chakraborty|Veen|Sumbul Touqueer|",
    "wins_nominations": "1 win",
    "release_date": "28 June 2019 (USA)"
  },
  {
    "title_x": "One Day: Justice Delivered",
    "imdb_id": "tt8130558",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/f/f7/One_Day-_Justice_Delivered.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/One_Day:_Justice_Delivered",
    "title_y": "One Day: Justice Delivered",
    "original_title": "One Day: Justice Delivered",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "124",
    "genres": "Action|Crime|Thriller",
    "imdb_rating": 6.8,
    "imdb_votes": 590,
    "story": "A Crime Branch Special Officer Investigates the Serial disappearance of high profile individuals in a state capital.",
    "summary": "A Crime Branch Special Officer Investigates the Serial disappearance of high profile individuals in a state capital.",
    "tagline": "",
    "actors": "Anupam Kher|Kumud Mishra|Esha Gupta|Zakir Hussain|Rajesh Sharma|Murli Sharma|Deepshika Nagpal|",
    "wins_nominations": "",
    "release_date": "5 July 2019 (India)"
  },
  {
    "title_x": "Super 30 (film)",
    "imdb_id": "tt7485048",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/thumb/2/29/Super_30_The_Film.jpg/220px-Super_30_The_Film.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/Super_30_(film)",
    "title_y": "Super 30",
    "original_title": "Super 30",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "154",
    "genres": "Biography|Drama",
    "imdb_rating": 8.2,
    "imdb_votes": 13972,
    "story": "Anand Kumar  a Mathematics genius from a modest family in Bihar who is made to believe that only a King's son can become a king is on a mission to prove that even the poor man can create some of the world's most genius minds. He starts a training program named 'Super 30' to help 30 IIT aspirants crack the entrance test and make them highly successful professionals.",
    "summary": "Based on life of Patna-based mathematician Anand Kumar who runs the famed Super 30 program for IIT aspirants in Patna.",
    "tagline": "Inspired by the Life of Anand Kumar & His Students",
    "actors": "Hrithik Roshan|Mrunal Thakur|Nandish Singh|Virendra Saxena|Sadhana Singh|Aditya Srivastava|Sanket Deshpande|Pankaj Tripathi|Vaibhav Gupta|Ali Haji|Rajesh Sharma|Deepali Kumar|Chittaranjan Giri|Ganesh Kumar|",
    "wins_nominations": "",
    "release_date": "12 July 2019 (USA)"
  },
  {
    "title_x": "Batla House",
    "imdb_id": "tt8869978",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/e/ed/Batla_House_poster.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/Batla_House",
    "title_y": "Batla House",
    "original_title": "Batla House",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "146",
    "genres": "Action|Drama|Thriller",
    "imdb_rating": 7.3,
    "imdb_votes": 5556,
    "story": "This action thriller is based on the real-life incident of 'Batla House Encounter'  officially known as Operation Batla House  to the silver screen. The incident took place on September 19  2008  against Indian Mujahideen (IM) terrorists in Batla House locality in Jamia Nagar  Delhi.",
    "summary": "After a deadly encounter  a police officer works to prove that the police acted lawfully.",
    "tagline": "The Story of India's Most Decorated / Controversial Cop",
    "actors": "John Abraham|Nora Fatehi|Mrunal Thakur|Rajesh Sharma|Ravi Kishan|Sonam Arora|Sahidur Rahaman|Manish Chaudhary|Anil Rastogi|Kranti Prakash Jha|Amruta Sant|Anil Khopkar|Aditi Gulati|Jitendra Trehan|",
    "wins_nominations": "",
    "release_date": "15 August 2019 (USA)"
  },
  {
    "title_x": "Mission Mangal",
    "imdb_id": "tt9248972",
    "poster_path": "https://upload.wikimedia.org/wikipedia/en/thumb/b/b7/Mission_Mangal.jpg/220px-Mission_Mangal.jpg",
    "wiki_link": "https://en.wikipedia.org/wiki/Mission_Mangal",
    "title_y": "Mission Mangal",
    "original_title": "Mission Mangal",
    "is_adult": 0,
    "year_of_release": "2019",
    "runtime": "130",
    "genres": "Drama|History",
    "imdb_rating": 6.5,
    "imdb_votes": 7165,
    "story": "Based on true events of the Indian Space Research Organisation (ISRO) successfully launching the Mars Orbiter Mission (Mangalyaan)  making it the least expensive mission to Mars.",
    "summary": "Based on true events of the Indian Space Research Organisation (ISRO) successfully launching the Mars Orbiter Mission (Mangalyaan)  making it the least expensive mission to Mars.",
    "tagline": "This Independence Day...The Sky is Not the Limit",
    "actors": "Akshay Kumar|Vidya Balan|Nithya Menon|Taapsee Pannu|Sonakshi Sinha|Kirti Kulhari|H.G. Dattatreya|Sharman Joshi|Sanjay Kapoor|Dalip Tahil|Vikram Gokhale|Mohammed Zeeshan Ayyub|Purab Kohli|",
    "wins_nominations": "",
    "release_date": "15 August 2019 (USA)"
  }
]

Install jq on your linux machine and hit below command

cat movies.json | \
jq -c '.[] | {"index": {"_index": "movies", "_id": .id}}, .' > nd-movies.json

Verify newly created nd-movies.json file.

In this gist we will quickly spin a sample Kubernetes cluster and deploying the nginx pod. Additionally, we will implement monitoring using Elastic.

Setup K8s cluster

Cluster architecture

3 Node cluster

Machine - Centos7, 4GB RAM

1. kube1.local - Control plane node
2. kube2.local - worker node
3. kube3.local - worker node

Here I am setting hostname kube1.local, kube2.local, kube3.local. Login into all of the servers and perform below command on all three nodes.

Swap off

sudo swapoff -a

Install docker

You can refer docker documentation but here are the quick steps:

Contanerd runtime

kubeadm automatically tries to detect an installed container runtime by scanning through a list of known endpoints.

Verify if containerd is running or not

ps -ef | grep containerd

In my system containerd was running. Install containerd if it is not installed.

Change below cofing to remove cri from disable_plugins

sudo vim /etc/containerd/config.toml
#disabled_plugins = ["cri"]
disabled_plugins = []

sudo systemctl restart containerd

Install kubeadm, kubelet & kubectl

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF


# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0

sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

Make sure you install same version on all of the nodes. In my case i am installing v1.28.

Need to install all three packages on all nodes

Kubeadm - Use to bootstrap the cluster

Kubelet - Will run all over the machin and take care of pods and container (General operation like stop / start / modify)

Kubectl - command line utility to talk with kubernetes APIs

On control plane node only (kube1.local)

Initialise kubeadm

kubeadm init

On success, It will print the below join command, which we will use to join any machine in the cluster.

kubeadm join control_plane_ip:6443 --token tlif0t.jq7r1nvdmltre7m3 --discovery-token-ca-cert-hash sha256:02eb1a7f249b2f2a2b8db2b8fef9b5564ac3db9d42da39db71c23c06df5cecb8

Save this command. We need to execute this command on every worker node to join the cluster.

On worker node only (kube2.local, kube3.local)

Perform join command.

kubeadm join control_plane_ip:6443 --token tlif0t.jq7r1nvdmltre7m3 --discovery-token-ca-cert-hash sha256:02eb1a7f249b2f2a2b8db2b8fef9b5564ac3db9d42da39db71c23c06df5cecb8

You should see the message - “This node has joined the cluster” with addition details.

Verify cluster (kube1.local)

Enable kubectl to run from non root user.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

List nodes

kubectl get nodes

NAME          STATUS   ROLES           AGE     VERSION
kube1.local   Ready    control-plane   199d    v1.28.1
kube2.local   Ready    <none>          8m22s   v1.28.8
kube3.local   Ready    <none>          199d    v1.28.1

Our cluster is up and running.

Deploy Nginx (kube1.local)

Create Nginx deployment

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
EOF

Expose the Nginx deployment on a NodePort 32000

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
    - port: 80
      targetPort: 80
      nodePort: 32000
EOF

Verify pod is running

Kubectl get pods

Just visit on browser - http://external_id:32000.

You should able to see the nginx’s default page.

Setup Observability with Elastic

Enable kube-state-matrics (kube1.local)

git clone https://github.com/kubernetes/kube-state-metrics
kubectl apply -f kube-state-metrics/examples/standard/

Verify endpoint

kubectl port-forward svc/kube-state-metrics -n kube-system 8080:8080

#login to anoter tab and hit
curl localhost:8080/metrics

It should return all metrics.

Monitoring using Elastic cloud

You can follow detailed doc - https://www.elastic.co/getting-started/observability/monitor-kubernetes-clusters

Blog - https://www.elastic.co/blog/kubernetes-cluster-metrics-logs-monitoring

Reference talk

Bring logs, metrics, and traces from your Kubernetes cluster and the workloads running on it into a single, unified solution. Elastic observability gives better visibility on your kubernetes ecosystem where you can monitor your pods, services, workload etc. Use a centrally managed Elastic Agent to gain visibility into your Kubernetes deployments on EKS, AKS, GKE or self-managed clusters.

Talk Video

ES|QL is a new query language for Elasticsearch. It is the unified language for all kinds of use cases like simple queries, aggregations, performing correlations, finding logs, etc. It provides simple easy syntax to perform complex queries. If you come from SQL background, You going to find this very handy.

It is a piped separated langugage with a combination of source commands and process commands. The Elasticsearch Query Language (ES|QL) makes use of “pipes” (|) to manipulate and transform data in a step-by-step fashion. This means output of the first step will go as an input for second step.

ES|QL is more than langugage. The execution engine is developed by considering performance in mind. Here ES|QL is not going to convert into Query DSL instead it will be directly executed within Elasticsearch. It operates on blocks at a time instead of per row, targets vectorization and cache locality, and embraces specialization and multi-threading.

ES|QL - Filter, Transform and Analyze

Example

Below is a few examples of ES|QL. I am considering you have an Elasticsearch and kibana is installed and running. Please import the sample dataset (Sample web logs) from kibana. Navigate in side menu -> Management -> Dev Tools to perform the below query.

Source commands

FROM

# Format

POST /_query?format=csv
{
  "query": """
    from kibana_sample_data_logs
  """
}

ROW

POST _query?format=txt
{
  "query":"""
    row a = "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1"
    | dissect a "%{browser}/%{version}"
    | keep browser
  """
}

SHOW

POST /_query?format=txt
{
  "query": """
    show info
  """
}

Processing commands

keep

POST _query?format=txt
{
  "query": """
    from kibana_sample_data_logs
    | keep @timestamp, clientip, host, tags, bytes
  """
}

where, limit, sort

POST _query?format=txt
{
  "query": """
    from kibana_sample_data_logs
    | keep @timestamp, clientip, host, tags, bytes
    | where bytes > 1000
    | sort bytes desc
    | limit 5
  """
}

like

POST _query?format=txt
{
  "query":"""
    FROM sample_data
    | where message like "*error*"
  """
}

GROK, STATS…BY

POST _query?format=txt
{
  "query":"""
    from kibana_sample_data_logs
    | grok agent "%{WORD:browser}/%{NUMBER:version}"
    | keep browser, version, @timestamp
  """
}

POST _query?format=txt
{
  "query":"""
    from kibana_sample_data_logs
    | grok agent "%{WORD:browser}/%{NUMBER:version}"
    | keep browser, version, @timestamp
    | stats count(*) by version
  """
}

DISSECT

POST _query?format=txt
{
  "query": """
    from kibana_sample_data_logs 
    | dissect message "%{ip} - - %{time} %{web_call} "
    | keep ip, time, web_call
  """
}

EVAL

POST /_query?format=txt
{
  "query": """
    from kibana_sample_data_logs
    | eval t = replace(agent, "Mozilla", "Chrome")
    | eval l = length(agent)
    | dissect agent "%{browser}/%{version} "
    | eval lt = left(t, 6)
    | keep lt, browser, version, t, l
  """
}

You can check more string functions.

Data enrichment (ENRICH)

# Create mappings

PUT lang
{
  "mappings": {
    "properties": {
      "lang_id": {
        "type": "keyword"
      },
      "name": {
        "type": "keyword"
      }
    }
  }
}

PUT devs
{
  "mappings": {
    "properties": {
      "lang_id": {
        "type": "keyword"
      },
      "name": {
        "type": "keyword"
      }
    }
  }
}

# Create index "lang"

PUT lang/_bulk
{ "index" : {}}
{ "lang_id": "1x", "name": "java" }
{ "index" : {}}
{ "lang_id": "2x", "name": "php" }
{ "index" : {}}
{ "lang_id": "3x", "name": "node" }
{ "index" : {}}
{ "lang_id": "4x", "name": "python" }
{ "index" : {}}
{ "lang_id": "5x", "name": "ruby" }

# create index "devs"

PUT devs/_bulk
{ "index" : {}}
{ "lang_id": "5x", "developer": "bob" }
{ "index" : {}}
{ "lang_id": "3x", "developer": "mark" }
{ "index" : {}}
{ "lang_id": "1x", "developer": "max" }
{ "index" : {}}
{ "lang_id": "2x", "developer": "david" }
{ "index" : {}}
{ "lang_id": "4x", "developer": "ashish" }


# Create enrich policy

PUT /_enrich/policy/dev_lang
{
  "match": {
    "indices": "lang",
    "match_field": "lang_id",
    "enrich_fields": ["name"]
  }
}

PUT /_enrich/policy/dev_lang/_execute?wait_for_completion=true

POST _query?format=txt
{
  "query":"""
    from devs
    | keep lang_id, name, developer
    | enrich dev_lang on lang_id with name
  """
}

Search is not just traditional TF/IDF any more but the current trend of machine learning and models has opened another dimension for search.

This talk gives an overview of:

• Classic search and its limitations.
• What is a model and how can you use it.
• How to use vector search or hybrid search in Elasticsearch.
• Where OpenAI’s ChatGPT or similar LLMs come into play to with Elastic.

Check how to leverage Leverage ChatGPT with Elasticsearch.

Talk Video

Bring logs, metrics, and traces from your Kubernetes cluster and the workloads running on it into a single, unified solution. Elastic observability gives better visibility on your kubernetes ecosystem where you can monitor your pods, services, workload etc. Use a centrally managed Elastic Agent to gain visibility into your Kubernetes deployments on EKS, AKS, GKE or self-managed clusters.

Talk Video

In this hands-on workshop, We will learn how to connect ChatGPT to proprietary data stores using Elasticsearch and build question/answer capabilities for your data. In a demo, We will quickly convert your website, FAQ, or any documentation into prompt chat where your user can directly ask a question on your data.

Flow

Prerequisites

1. You have used ChatGPT :)

2. Good to have understanding around Elasticsearch (Not mandatory, Introduction will be cover)

3. System + Internet connection

4. OpenAI account with API key - Create new one from https://platform.openai.com/account/api-keys. Make sure it having free credits.

Without local setup

1. Google account to use google Colab.

2. Render account.

Local setup

1. Git - Install it from https://git-scm.com/downloads

2. Docker - Good to have. Install it from https://docs.docker.com/engine/install/.

3. Having basic python knowledge will be good.

For a workshop we going to follow without local setup.

1. Setup cluster

1. Visit cloud.elastic.co and signup.

2. Click on Create deployment. In the pop-up, you can change the settings or leave it default.

3. We need to add machine learning instance. For that, simply click on “advance settings” .

4. Go to “Machine Learning instances” -> click on “Add Capacity” and select at least 4GB ram capacity.

5. Finally click on “Create deployment”.

6. Download / Copy the deployment credentials.

7. Once deployment ready, click on “Continue” (or click on Open Kibana). It will redirect you on kibana dashboard.

2. Deploy Model

ELSER Model by Elastic (Recommended)

Go to the kibana panel. Navigate to Menu -> Machine Learning (In Analytics section). In left menu, Click on Trained Models (In Model Management Section).

1. ELSER can be found in the list of trained models.
2. Click the Download model button under Actions.
3. After the download is finished, start the deployment by clicking the Start deployment button.
4. Provide a deployment ID, select the priority, and set the number of allocations and threads per allocation values.
5. Click Start.

Third party model

We are going to use all-distilroberta-v1 model hosted on a hugging face. Lets import on an elastic cluster using eland.

Get your credentials ready

• cloud_id : Visit “cloud.elastic.co” -> Navigate to your deployment and click on “manage”. Simply copy Cloud ID and save it.
• cloud_user: elastic
• cloud_password: You will get it from step 1.6. If you forget to save, Simply click on “Action” -> “Reset password”. (Username will be elastic only)
• hf_model_id: sentence-transformers/all-distilroberta-v1 (Go to model page on huggingface & copy the ID sentence-transformers/all-distilroberta-v1)

Now there is two way, You can upload the model using docker as well as Google colab.

Using Google Colab (Recommended)

Simply click on below link. It will open ready made notebook. You just need to click on play button to run notebood.

Using Docker

2. We’re going to use docker for import model to the elastic cluster

   1.  git clone https://github.com/elastic/eland.git 
       cd eland
      

   2.  docker build -t elastic/eland .
      

   3.  docker run -it --rm elastic/eland eland_import_hub_model \
           --cloud-id <cloud_id> \
           -u elastic -p <elastic_cloud_password> \
           --hub-model-id sentence-transformers/all-distilroberta-v1 \
           --task-type text_embedding \
           --start
      

   4. Let’s wait till the model gets uploaded without any error.

   5. Exit from eland folder.

      cd ..
      

Verify uploaded model

Go to the kibana panel. Navigate to Menu -> Machine Learning (In Analytics section). In left menu, Click on Trained Models(Model Management Section). You must see your model here in the “Started” state.

In case if a warning message is displayed at the top of the page that says ML job and trained model synchronization required. Follow the link to Synchronize your jobs and trained models. Then click Synchronize.

3. Crawling private data

1. Click on Menu -> Enterprise Search -> “Create an Elasticsearch index” button
2. Click on Web crawler.
3. Add index name (It will add prefix search) and hit “Create index”. In my case index name is (search-ashish.one)
4. Go to “Pipelines” to create a pipeline.
5. Click “Copy and customize” in the Ingest Pipeline Box.
6. Click “Add Inference Pipeline” in the Machine Learning Inference Pipelines box.
7. Give the unique pipeline name e.g. “ml-inference-ashish-one”
8. Select a trained ML Model from the dropdown “sentence-transformers__all-distilroberta-v1” (For ELSER choose “.elser_model_1”)
9. Select “title” as the Source field and set “title-vector” as a destination. You can specify your own destination field name. (In case of ELSER, just select the “Source” field e.g title, body_content)
10. Let’s click on “Continue” and move to the Test(Optional) tab. Click on “Continue” again.
11. At the Review stage let’s click on “Create pipeline”.
12. (Skip this for ELSER) Go to Menu -> Management -> Dev Tools. Let’s create a mapping

POST <index_name>/_mapping
{
  "properties": {
    "<vector_field_name>": {
      "type": "dense_vector",
      "dims": 768,
      "index": true,
      "similarity": "dot_product"
    }
  }
}

In my case mapping will be:

POST search-ashish.one/_mapping
{
  "properties": {
    "title-vector": {
      "type": "dense_vector",
      "dims": 768,
      "index": true,
      "similarity": "dot_product"
    }
  }
}

Paste above query in cosole and hit on play button.

13. Go to Enterprise search -> indices -> your_index_name -> Manage Domains. Enter the domain (e.g. https://ashish.one. You can add your own domain) to crawl and hit “Validate Domain”.
14. If everything is fine, simply click on “Add domain” and start crawling by click on Crawl -> Crawl all domains on this index.
15. Go to Enterprise Search -> Indices. You should see your index name.

4. Setup Interface

** Get your credentials ready **

1. cloud_id : Visit “cloud.elastic.co” -> Navigate to your deployment and click on “manage”. Simply copy Cloud ID and save it.
2. cloud_user: elastic
3. cloud_password: You will get it from step 1.6. If you forget to save, Simply click on “Action” -> “Reset password”. (Username will be elastic)
4. openai_api: Create open ai api key from https://platform.openai.com/account/api-keys.
5. es_index: Index name which we created in step 3.3. (search-ashish.one)
6. vector_field: The field which we’ve set for destination at step 3.9. i.e. title-vector

Setup on local with Docker

1. Clone

git clone https://github.com/ashishtiwari1993/elasticsearch-chatgpt.git
cd elasticsearch-chatgpt

2. Replace credentials in Dockerfile

Open Dockerfile and change below creds

ENV openai_api="<open_api_key>"
ENV cloud_id="<elastic cloud id>"
ENV cloud_user="elastic"
ENV cloud_pass="<elastic_cloud_password>"
ENV es_index="<elasticsearch_index_name>"
ENV chat_title="<Any title for your page e.g. ashish.one GPT>"
ENV vector_field="< specify vector field where embedding will be save. e.g. title-vector>"

3. Build

docker build -t es-gpt .

4. Run

docker run -p 8501:8501 es-gpt

Simply visit on localhost:8501

Setup on Render with Docker

1. Signup on https://render.com.

2. Create Web Service.

3. Go to Public Git repository section and add below repo url

https://github.com/ashishtiwari1993/elasticsearch-chatgpt

Hit on Continue.

4. Add Name and select Free Instance Type.

5. Click on Advanced and Add Environment Variable

openai_api="<open_api_key>"
cloud_id="<elastic cloud id>"
cloud_user="elastic"
cloud_pass="<elastic_cloud_password>"
es_index="<elasticsearch_index_name>"                                                 
chat_title="<Any title for your page e.g. ashish.one GPT>"
vector_field="< specify vector field where embedding will be save. e.g. title-vector>"

6. Finally click on Create Web Service

Output

Reference

Blog - ChatGPT and Elasticsearch: OpenAI meets private data

What is Webhook ?

Webhook enables the two programs to communicate or transfer the data with the help of callback functions / hooks. Now in the modern tech world it is also known as Reverse API, Push API etc. Mostly it is used to send small amounts of data from source to destination. It is a one way data transfer procedure. It works over the HTTP protocol using REST API. It is simple like client and server communication. Most of the saas allow you to integrate their product with your system with the help of APIs and Webhook only. E.g. Slack and discord allows you to push messages with the help of webhooks. To accept the webhook event, You need to expose one HTTP endpoint lets say

POST /message

In the above example, your system will accept messages from a third party system. Here you can setup authentication, Method, required parameters etc. and convey to users who are going to use this webhook.

So I have to use the POST method, and send data on the above endpoint. You can hit this Endpoint using curl, postman or any programming language.

What is ELK ?

Well from elastic.co -

E - Elasticsearch - Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Check more here.

L - logstash - Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash.” check more here.

K - Kibana - Kibana is an free and open frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch. Check more here.

The Flow

Events -> Logstash -> Elasticsearch <- Kibana

Logstash works with three plugins - input, filter, output.

You can choose any source as a input. For example you can use jdbc input plugin to read data from mysql.

Transform your data with the help of filter plugin.

Push the data on destination which you have specified in output plugin.

We will use below plugins to process the webhooks requests.

• Input plugin -> HTTP
• Output plugin -> Elasticsearch

Logstash will process the event and push to the Elasticsearch. Visualise data from kibana.

Implementation

Installation

You can simply follow the official documentation for installation.

• Elasticsearch - https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html
• Kibana - https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
• Logstash - https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

I am spinning ELK instances using docker. You can choose any method mentioned on above links.

Verify

Lets verify everything is up and running properly.

Elasticsearch

Hit below curl command:

curl --cacert path/to/ca.crt -u elastic:pass@123 https://localhost:9200

Response

{
  "name" : "es01",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "_CDp3XgbQUKTuQxZWVLh6A",
  "version" : {
    "number" : "8.6.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "f67ef2df40237445caa70e2fef79471cc608d70d",
    "build_date" : "2023-01-04T09:35:21.782467981Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

Logstash

Simply check the log file if there is any error.

tail -f logs/logstash-plain.log

Kibana

Simply visit https://localhost:5601. Try to login in kibana with credentials.

Now the ELK stack is up and running. Lets create the logstash configuration file to receive the webhook request and push to elasticsearch.

Logstash pipeline

Create file webhook.conf on the path which you have specified at path.config settings. You can set this setting at config/logstash.yml or config/pipeline.yml.

webhook.conf

input {
  http {
    port => 4000
  }
}

filter {
  json {
    source => "message"
  }
}

output {   
  elasticsearch {
    hosts => ["https://es01:9200"]
    cacert => '/usr/share/logstash/pipeline/certs/ca.crt'
    user => 'elastic'
    password => 'pass@123'
    index => 'webhook'
  }
}

We are configuring HTTP endpoint on port 4000. So whenever anyone calls the webhook endpoint, they need to specify port like http://mydomain.com:4000

Here i am only defining the index but you can configure data stream as well. Check here for more options.

Test configuration

Run pipeline with below command:

./bin/logstash -f webhook-receiver.conf

Check if any errors are there.

If everything seems fine, Let’s start the logstash service.

Test

Push sample data

curl -XPOST -H 'Content-type:applicaton/json' http://localhost:4000 -d '{"test_key1":"test_value1","info":{"name":"Ashish","last_name":"Tiwari"},"my_list":["el1","el2"]}'

Response

ok%

Verify data in elasticsearch

Login to kibana by visiting localhost:5601.

Navigate to Menu -> Management -> Dev Tools

Lets see if Index is created or not.

GET _cat/indices?v

Response

health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   webhook LiXxWLy5QvKkpYpmLLbYsw   1   1          1            0     12.5kb         12.5kb

I can see the index webhook has been created. Also the docs.count is 1 which means data has been inserted.

Lets see the data.

GET webhook/_search

Response

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "webhook",
        "_id": "-ok63YUBrNdmvLyIE5Ue",
        "_score": 1,
        "_source": {
          "host": {
            "ip": "192.168.192.1"
          },
          "my_list": [
            "el1",
            "el2"
          ],
          "message": """{"test_key1":"test_value1","info":{"name":"Ashish","last_name":"Tiwari"},"my_list":["el1","el2"]}""",
          "event": {
            "original": """{"test_key1":"test_value1","info":{"name":"Ashish","last_name":"Tiwari"},"my_list":["el1","el2"]}"""
          },
          "@version": "1",
          "@timestamp": "2023-01-23T06:04:08.664398594Z",
          "http": {
            "version": "HTTP/1.1",
            "method": "POST",
            "request": {
              "mime_type": "applicaton/json",
              "body": {
                "bytes": "97"
              }
            }
          },
          "info": {
            "last_name": "Tiwari",
            "name": "Ashish"
          },
          "user_agent": {
            "original": "curl/7.79.1"
          },
          "url": {
            "path": "/",
            "domain": "localhost",
            "port": 4000
          },
          "test_key1": "test_value1"
        }
      }
    ]
  }
}

If you are able to see data like above, Your webhook receiver is all set to accept requests.

Data Visualisation

Simply navigate to Menu -> Analytics -> Discover

Create a Data View on the index webhook.

To create the dashboard, go to Menu -> Analytics -> Dashboard. You can create a dashboard according to your requirement.

Conclusion

We have successfully set up the webhook receiver with the help of ELK stack. Though ELK use cases are very vast. There are various input and output plugins available in logstash for data pipelines.

Error: pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService

System configuration

centos 9 / 2GB RAM / 2CPU

Master Node

Same issue on master node.

Command

[root@kube-master-1 ~]# kubeadm config images pull
failed to pull image "registry.k8s.io/kube-apiserver:v1.26.0": output: E0107 14:52:09.997544    4134 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService" image="registry.k8s.io/kube-apiserver:v1.26.0"
time="2023-01-07T14:52:09Z" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

✅ Solved

Remove below file:

rm /etc/containerd/config.toml

Try again.

Worker Node

Same issue on worker node while joining to master.

Command

[root@kube-worker-2 ~]# kubeadm join x.x.x.x:6443 --token ga8bqg.01azxe9avjx2n6jr        --discovery-token-ca-cert-hash sha256:d57699d74721094e5f921d48a0f9f895a0d7def7e1977e95ce0027a03e7f7d39
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR CRI]: container runtime is not running: output: E0107 17:46:12.269694   11160 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
time="2023-01-07T17:46:12Z" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

✅ Solved

Same remove the config.toml file and restart the containerd service.

rm /etc/containerd/config.toml
systemctl restart containerd

Elastic Stack (Elasticsearch, Logstash, Kibana and Beats) is such a platform which is built for scalability, performance and “You know… for Search”. When you have a system which scales to the horizons of your data, helps you in your data quest, shows you insights - imagine what you can do with it.

Talk Video

Feel free to comment below, If you have any doubts or suggestion about this talk.

CRUD

# Insert

POST meetup/_doc/
{
  "name":"Ashish Tiwari"
}

# Insert with id

POST meetup/_doc/1
{
  "name":"Ashish Tiwari"
}

# Search

GET meetup/_search

# Update

POST meetup/_doc/1
{
  "name":"Ashish",
  "company":"elastic",
  "address":"Navi Mumbai kharghar",
  "skills":{
    "language":["php","java","node"],
    "database":["mysql","mongodb"],
    "search":"elasticsearch"
  }
}

# search with query

GET meetup/_search
{
  "query": {
    "match": {
      "address": "navi"
    }
  }
}

# delete

DELETE meetup

BULK

POST _bulk
{"index":{"_index":"meetup"}}
{"user_id":1,"first_name":"Yvonne","last_name":"Willmott","email":"ywillmott0@live.com","gender":"Female","street_address":"38 Helena Avenue","ip_address":"104.221.25.110","company":"Flashset"}
{"index":{"_index":"meetup"}}
{"user_id":2,"first_name":"Immanuel","last_name":"Philbrick","email":"iphilbrick1@wunderground.com","gender":"Male","street_address":"01 Bunting Pass","ip_address":"9.20.164.27","company":"Babblestorm"}
{"index":{"_index":"meetup"}}
{"user_id":3,"first_name":"Clotilda","last_name":"Danelut","email":"cdanelut2@deliciousdays.com","gender":"Agender","street_address":"0 Crowley Trail","ip_address":"158.94.144.140","company":"Riffpedia"}
{"index":{"_index":"meetup"}}
{"user_id":4,"first_name":"Nahum","last_name":"Attfield","email":"nattfield3@blog.com","gender":"Male","street_address":"7 Garrison Court","ip_address":"225.144.148.44","company":"Chatterpoint"}
{"index":{"_index":"meetup"}}
{"user_id":5,"first_name":"Vaughan","last_name":"Middis","email":"vmiddis4@ted.com","gender":"Male","street_address":"7 Cody Way","ip_address":"66.198.31.108","company":"Mynte"}
{"index":{"_index":"meetup"}}
{"user_id":6,"first_name":"Nolie","last_name":"Alessandrucci","email":"nalessandrucci5@networksolutions.com","gender":"Male","street_address":"826 Brown Hill","ip_address":"96.77.221.95","company":"Feedfish"}
{"index":{"_index":"meetup"}}
{"user_id":7,"first_name":"Beverlie","last_name":"Ovitts","email":"bovitts6@tripod.com","gender":"Male","street_address":"6 Sycamore Pass","ip_address":"102.24.117.107","company":"Zazio"}
{"index":{"_index":"meetup"}}
{"user_id":8,"first_name":"Graeme","last_name":"Dopson","email":"gdopson7@free.fr","gender":"Female","street_address":"26 Dunning Avenue","ip_address":"198.33.215.93","company":"Flashset"}
{"index":{"_index":"meetup"}}
{"user_id":9,"first_name":"Mellisa","last_name":"Hurich","email":"mhurich8@nbcnews.com","gender":"Male","street_address":"6371 Browning Way","ip_address":"66.0.3.199","company":"Divanoodle"}
{"index":{"_index":"meetup"}}
{"user_id":10,"first_name":"Dyan","last_name":"Loude","email":"dloude9@berkeley.edu","gender":"Female","street_address":"9818 Reindahl Road","ip_address":"16.56.137.54","company":"Agivu"}
{"index":{"_index":"meetup"}}
{"user_id":11,"first_name":"Becky","last_name":"Shank","email":"bshanka@tinypic.com","gender":"Male","street_address":"1206 Warrior Terrace","ip_address":"90.63.35.111","company":"Izio"}
{"index":{"_index":"meetup"}}
{"user_id":12,"first_name":"Bar","last_name":"Bedburrow","email":"bbedburrowb@vistaprint.com","gender":"Male","street_address":"75 Onsgard Crossing","ip_address":"85.122.33.250","company":"Zoombox"}
{"index":{"_index":"meetup"}}
{"user_id":13,"first_name":"Dorey","last_name":"Isenor","email":"disenorc@privacy.gov.au","gender":"Female","street_address":"53682 Parkside Crossing","ip_address":"150.158.150.213","company":"Rhyzio"}
{"index":{"_index":"meetup"}}
{"user_id":14,"first_name":"Torrin","last_name":"Rangall","email":"trangalld@buzzfeed.com","gender":"Male","street_address":"24247 Old Shore Plaza","ip_address":"40.151.17.2","company":"Devpoint"}
{"index":{"_index":"meetup"}}
{"user_id":15,"first_name":"Genvieve","last_name":"Beslier","email":"gbesliere@yolasite.com","gender":"Male","street_address":"96214 Miller Trail","ip_address":"115.143.68.208","company":"Oyonder"}
{"index":{"_index":"meetup"}}
{"user_id":16,"first_name":"Arden","last_name":"Ramas","email":"aramasf@whitehouse.gov","gender":"Polygender","street_address":"390 Gulseth Alley","ip_address":"36.83.126.154","company":"Youbridge"}
{"index":{"_index":"meetup"}}
{"user_id":17,"first_name":"Alyosha","last_name":"Domm","email":"adommg@washingtonpost.com","gender":"Female","street_address":"32 Oxford Way","ip_address":"174.71.176.45","company":"Wikizz"}

Upload sample json data from kibana

Download movies.json and insert into elasticsearch by using below command:

Open Kibana -> Menu -> Home -> Upload a file

Create Data view in Kibana

Open Kibana -> Menu -> Stack Management -> Data Views (Kibana)

Create Dashboard

Open Kibana -> Menu -> Analytics -> Dashboard

Create mapping

PUT /devfest-raipur
{
  "mappings": {
    "properties": {
      "age":    { "type": "integer" },  
      "email":  { "type": "keyword"  }, 
      "name":   { "type": "text"  }     
    }
  }
}

Analyze

# analyze

GET /_analyze?pretty
{
  "text" : "Quick Brown Foxes!"
}

# Whitespace

GET _analyze
{
  "analyzer": "whitespace",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog\u0027s bone."
}

What is an analyzer?

An analyzer is made of character filters, tokenizer and token filters.

Let’s build one

POST _analyze
{
  "char_filter": [], 
  "tokenizer":   "standard",
  "filter":      [], 
  "text": [ 
    "I like when the <strong>quick</strong> foxes jumps over lazy DOGS!",
    "and <strong>fast</strong>"
  ]
}

Let’s remove the html code.

POST _analyze
{
  "char_filter": ["html_strip"], 
  "tokenizer":   "standard",
  "filter":      [], 
  "text": [ 
    "I like when the <strong>quick</strong> foxes jumps over lazy DOGS!",
    "and <strong>fast</strong>"
  ]
}

Some words don’t bring us any value. Let’s skip them.

POST _analyze
{
  "char_filter": ["html_strip"], 
  "tokenizer":   "standard",
  "filter":      [
    {
      "type":       "stop",
      "stopwords":  [ "_english_"]
    }
  ], 
  "text":
  [ 
    "I like when the <strong>quick</strong> foxes jumps over lazy DOGS!",
    "and <strong>fast</strong>"
  ]
}

We can also remove “I”, “when” and “over”.

POST _analyze
{
  "char_filter": ["html_strip"], 
  "tokenizer":   "standard",
  "filter":      [
    {
      "type":       "stop",
      "ignore_case":true, 
      "stopwords":  [ "_english_", "I", "when", "over"]
    }
  ], 
  "text":
  [ 
    "I like when the <strong>quick</strong> foxes jumps over lazy DOGS!",
    "and <strong>fast</strong>"
  ]
}

DOGS and dogs should match.

POST _analyze
{
  "char_filter": ["html_strip"], 
  "tokenizer":   "standard",
  "filter":      [
    {
      "type":       "stop",
      "ignore_case":true, 
      "stopwords":  [ "_english_", "I", "when", "over"]
    },
    "lowercase"
  ], 
  "text":
  [ 
    "I like when the <strong>quick</strong> foxes jumps over lazy DOGS!",
    "and <strong>fast</strong>"
  ]
}

dog, dogs and fox, foxes and jump, jumps, jumping, jumped should match. Let’s use a stemmer.

POST _analyze
{
  "char_filter": ["html_strip"], 
  "tokenizer":   "standard",
  "filter":      [
    {
      "type":       "stop",
      "ignore_case":true, 
      "stopwords":  [ "_english_", "I", "when", "over"]
    },
    "lowercase",
    {
      "type":       "stemmer",
      "language":   "english" 
    }
  ], 
  "text":
  [ 
    "jumping jumps jump jumped",
    "I like when the <strong>quick</strong> foxes jumps over lazy DOGS!",
    "and <strong>fast</strong>"
  ]
}

Language analyzer

GET /_analyze?pretty
{
  "analyzer": "hindi", 
  "text" : "चाणक्य ने चंद्रगुप्त और बिंदूसार दोनों के लिए प्रधानमंत्री और राजनयिक सलाहकार के रूप में काम किया।"
}

Let’s create Hindi search engine (गीतयंत्र)

Create mapping

PUT geetyantra
{
  "settings": {
    "analysis": {
      "analyzer": {
        "geetyantra-analyzer":{
          "type":"hindi"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "adhyay": {"type": "integer"},
      "updesh": {"type": "text","analyzer": "geetyantra-analyzer"}
    }
  }
}

Insert data

Taking data from 🙏विकिपीडिया-श्रीमद्भगवद्गीता

Lets insert some data like below:

POST geetyantra/_doc
{
  "adhyay":1,
  "updesh":"कृष्ण ने अर्जुन की वह स्थिति देखकर जान लिया कि अर्जुन का शरीर ठीक है किंतु युद्ध आरंभ होने से पहले ही उस अद्भुत क्षत्रिय का मनोबल टूट चुका है। बिना मन के यह शरीर खड़ा नहीं रह स।"
}

POST geetyantra/_doc
{
  "adhyay":2,
  "updesh":"दूसरे अध्याय का नाम सांख्ययोग है। इसमें जीवन की दो प्राचीन संमानित परंपराओं का तर्कों द्वारा वर्णन आया है। अर्जुन को उस कृपण स्थिति में रोते देखकर कृष्ण ने उनका ध्यान दिलाया है कि इस प्रकार का क्लैव्य और हृदय की क्षुद्र दुर्बलता अर्जुन जैसे वीर के लिए उचित नहीं।"
}

POST geetyantra/_doc
{
  "adhyay":3,
  "updesh":"नित्य कर्म करने वाले की श्रेष्ठता"
}

POST geetyantra/_doc
{
  "adhyay":4,
  "updesh":"चौथे अध्याय में, जिसका नाम ज्ञान-कर्म-संन्यास-योग है, यह बाताया गया है कि ज्ञान प्राप्त करके कर्म करते हुए भी कर्मसंन्यास का फल किस उपाय से प्राप्त किया जा सकता है।",
  "shloka":"यदा यदा हि धर्मस्य ग्लानिर्भवति भारत,अभ्युत्थानमधर्मस्य तदात्मानं सृजाम्यहम्"
}

POST geetyantra/_doc
{
  "adhyay":5,
  "updesh":"ज्ञानी महापुरुष विद्या-विनययुक्त ब्राह्मण में और चाण्डाल में तथा गाय, हाथी एवं कुत्ते में भी समरूप परमात्मा को देखने वाले होते हैं।",
  "shloka":"विद्याविनयसंपन्ने ब्राह्मणे गवि हस्तिनि,शुनि चैव श्वपाके च पंडिता: समदर्शिन"
}


POST geetyantra/_doc
{
  "adhyay":6,
  "updesh":"छठा अध्याय आत्मसंयम योग है जिसका विषय नाम से ही प्रकट है। जितने विषय हैं उन सबसे इंद्रियों का संयम-यही कर्म और ज्ञान का निचोड़ है। सुख में और दुख में मन की समान स्थिति, इसे ही योग कहते हैं।"
}

POST geetyantra/_doc
{
  "adhyay":7,
  "updesh":"पंचतत्व, मन, बुद्धि भी मैं हूँ| मैं ही संसार की उत्पत्ति करता हूँ और विनाश भी मैं ही करता हूँ। मेरे भक्त चाहे जिस प्रकार भजें परन्तु अंततः मुझे ही प्राप्त होते हैं। मैं योगमाया से अप्रकट रहता हूँ और मुर्ख मुझे केवल साधारण मनुष्य ही समझते हैं।",
  "shloka":"यो यो यां यां तनुं भक्तः श्रद्धयार्चितुमिच्छति,तस्य तस्याचलां श्रद्धां तामेव विदधाम्यहम्"
}

POST geetyantra/_doc
{
  "adhyay":8,
  "updesh":"की संज्ञा अक्षर ब्रह्मयोग है। उपनिषदों में अक्षर विद्या का विस्तार हुआ। गीता में उस अक्षरविद्या का सार कह दिया गया है-अक्षर ब्रह्म परमं, अर्थात् परब्रह्म की संज्ञा अक्षर है। मनुष्य, अर्थात् जीव और शरीर की संयुक्त रचना का ही नाम अध्यात्म है।"
}

POST geetyantra/_doc
{
  "adhyay":9,
  "updesh":"को राजगुह्ययोग कहा गया है, अर्थात् यह अध्यात्म विद्या विद्याराज्ञी है और यह गुह्य ज्ञान सबमें श्रेष्ठ है। राजा शब्दका एक अर्थ मन भी था। अतएव मन की दिव्य शक्तिमयों को किस प्रकार ब्रह्ममय बनाया जाय, इसकी युक्ति ही राजविद्या है।"
}

POST geetyantra/_doc
{
  "adhyay":10,
  "updesh":"दसवें अध्याय का नाम विभूतियोग है। इसका सार यह है कि लोक में जितने देवता हैं, सब एक ही भगवान, की विभूतियाँ हैं, मनुष्य के समस्त गुण और अवगुण भगवान की शक्ति के ही रूप हैं।"
}

POST geetyantra/_doc
{
  "adhyay":11,
  "updesh":"का नाम विश्वरूपदर्शन योग है। इसमें अर्जुन ने भगवान का विश्वरूप देखा। विराट रूप का अर्थ है मानवीय धरातल और परिधि के ऊपर जो अनंत विश्व का प्राणवंत रचनाविधान है, उसका साक्षात दर्शन। विष्णु का जो चतुर्भुज रूप है, वह मानवीय धरातल पर सौम्यरूप है।"
}

POST geetyantra/_doc
{
  "adhyay":12,
  "updesh":"का नाम भक्ति योग है। जो जानने योग्य है। जिसको जानकर मनुष्य परमानन्द को प्राप्त हो जाता है अर्थात वो परमात्मा ही सत्य है ।।"
}

POST geetyantra/_doc
{
  "adhyay":13,
  "updesh":"में एक सीधा विषय क्षेत्र और क्षेत्रज्ञ का विचार है। यह शरीर क्षेत्र है, उसका जाननेवाला जीवात्मा क्षेत्रज्ञ है।"
}

POST geetyantra/_doc
{
  "adhyay":14,
  "updesh":"का नाम गुणत्रय विभाग योग है। यह विषय समस्त वैदिक, दार्शनिक और पौराणिक तत्वचिंतन का निचोड़ है-सत्व, रज, तम नामक तीन गुण-त्रिको की अनेक व्याख्याएँ हैं।"
}

POST geetyantra/_doc
{
  "adhyay":15,
  "updesh":"का नाम पुरुषोत्तमयोग है। इसमें विश्व का अश्वत्थ के रूप में वर्णन किया गया है। यह अश्वत्थ रूपी संसार महान विस्तारवाला है। देश और काल में इसका कोई अंत नहीं है।"
}

POST geetyantra/_doc
{
  "adhyay":16,
  "updesh":"में देवासुर संपत्ति का विभाग बताया गया है। आरंभ से ही ऋग्देव में सृष्टि की कल्पना दैवी और आसुरी शक्तियों के रूप में की गई है। "
}

POST geetyantra/_doc
{
  "adhyay":17,
  "updesh":"की संज्ञा श्रद्धात्रय विभाग योग है। इसका संबंध सत, रज और तम, इन तीन गुणों से ही है, अर्थात् जिसमें जिस गुण का प्रादुर्भाव होता है, उसकी श्रद्धा या जीवन की निष्ठा वैसी ही बन जाती है।"
}

POST geetyantra/_doc
{
  "adhyay":18,
  "updesh":"की संज्ञा मोक्षसंन्यास योग है। इसमें गीता के समस्त उपदेशों का सार एवं उपसंहार है। यहाँ पुन: बलपूर्वक मानव जीवन के लिए तीन गुणों का महत्व कहा गया है। पृथ्वी के मानवों में और स्वर्ग के देवताओं में कोई भी ऐसा नहीं जो प्रकृति के चलाए हुए इन तीन गुणों से बचा हो।"
}

Search

GET geetyantra/_search?size=20

GET geetyantra/_search
{
  "query": {
    "match": {
      "updesh": "अर्जुन"
    }
  }
}

GET geetyantra/_search
{
  "query": {
    "match": {
      "shloka": "धर्मस्य"
    }
  }
}

GET geetyantra/_search
{
  "query": {
    "multi_match": {
      "query": "श्रद्धा",
      "fields": ["shloka","updesh"]
    }
  }
}

GET geetyantra/_search
{
  "query": {
    "match_phrase_prefix": {
      "updesh": "चौथे अध्याय में"
    }
  }
}

Search as you type

Refer article Search as you type.

As a Developer, we keep learning & developing stuff. Sometimes we also find the solutions. Our stuff is distributed the same as our System architecture. We maintain different platforms like GitHub for projects, Medium for blogs, LinkedIn for profile, etc.

All this stuff we want to share on a single platform but as a tech, I am lazy if you ask me to install some CMS and maintain all stuff over there. it is hard to switch from a black terminal 💻 window to some UI 😅

As a developer I was always thinking can I write my blogs, article or notes in Vim? What if we can publish the blog the same as we do releases by hitting some commands.

In this gist, I am quickly going to give you some suggestions for tools that will help you to build your platform.

The Requirement 📝

• Write a blog in any editor (For me its Vim)
• Easy to publish
• Developer Friendly
• Lightweight and speed
• Cost effective

Let’s Build 🔧

I was evaluating multiple tools while searching for a platform and found Hugo which is easy, fast, and handy as a developer. Hugo is an opensource static site generator which means you can write everything in Markdown and Hugo will generate the site accordingly. You need to know the basic Markdown syntax and you are good to go.

You can refer official hugo document to get started.

Below is some tool that helped me to set up my platform.

Theme

Lots of themes are present which you can configure with your Hugo site. There are some common features across the themes and some themes provide the special features also. You can explore all themes here themes.gohugo.io

I was looking for a simple theme which has a simple layout with menus, a dark theme, and tech friendly.

PaperMod

I have build my website in PaperMod. Few pointers why i choose:

1. Search 🔍

PaperMod uses Fuse.js Basic for search functionality.

2. Post Cover Image 🗼

It gives an easy option to add a cover image to your post.

3. Edit link for post ✏️

Suggest changes option to ask viewers to contribute or Raise PR.

You can check more details about all features here

hyde-hyde

hyde-hyde I found a simple and easy theme if you want to get started with a simple menu and post. I chooses this because of its simplicity and then migrated to PaperMod.

Comments 💬

Once your audience starts reading your article, they would like to give feedback, suggestion and sometime it could be a discussion. You need someplace like comments where viewers can add their points to a particular article. The theme does not come with comments, for that you need to integrate the comments tool. Below are some suggestions you can explore:

1. giscus

Giscus comments system powered by GitHub Discussions. Let visitors leave comments and reactions on your website via GitHub! As soon as your viewers comment on your article, it will create the discussion thread on Github Discussion. You can explore more about giscus on the official site.

2. Utterances

utterances lightweight comments system built powered by GitHub issues. It will create an issue per article once anyone comments. All comments will be associated with a particular Github issue.

3. DISQUS

Disqus is a blog comment hosting paid service for websites and online communities that use a networked platform. It also comes with social integration, social networks, user profiles, profile notifications, etc.

I migrated my comment system from utterances to giscus. As Github discussion is a proper tool for commenting, discussion, etc. Both are lightweight and you can choose accordingly.

Shortcodes

While developing the site with Hugo, Shortcodes are your friends. Always check if shortcodes are available for popular tools e.g. youtube, github gist, etc.

Hosting ☁️

After the site generation, you need to host your website somewhere.

Found Github pages best place to host your static site. It allows you to manage your website the same way you manage your projects on github. Get started by creating a simple repository and pushing your source directory.

Check out all steps here

Once you set up end to end, You can publish your blog or changes by just basic git commands i.e. git add, git commit, git push.

It will give you the same feeling as you are releasing some features for your project.

You can check more options about hosting and deployment

I have hosted current website on github.com/ashishtiwari1993.

Domain

By default, Github pages assign the domain like username.github.io and you can access your site by visiting that subdomain. You can also setup the custom domain.

Speed 🚀

Almost 100% Page speed.

Pagespeed insights

tl;dr

✅ Platform - Hugo
✅ Theme - PaperMod
✅ Comment - giscus
✅ Hosting - Github Pages
✅ PageSpeed - 100%
✅ Cost - 100% Free

In this gist, we will quickly try to spin Elastic stacks with Docker containers. We are going to use docker-compose. You can learn more about Docker & Docker Compose, Which will help you to understand the flow.

Prerequisite

Tested on the below configuration.

• docker:Docker version 20.10.16, build aa7e414
• docker-compose:Docker version 20.10.16, build aa7e414

Cluster

This setup will include

• Elasticsearch
• Kibana
• Logstash
• APM

Setup

Clone repo:

git clone https://github.com/ashishtiwari1993/elastic-docker.git
cd elastic-docker

Make changes in .env file.

Start the cluster

Start

docker-compose up -d

Just visit to localhost:5601. You should see a kibana login page.

Stop

docker-compose down

Stop with deleting network, containers and volumes

docker-compose down -v

Access stacks

Elasticsearch

Access via curl from host machine

Copy ca.crt file

docker cp elastic-docker_es01_1:/usr/share/elasticsearch/config/certs/ca/ca.crt /tmp/

Curl command

curl --cacert /tmp/ca.crt -u elastic:pass@123 https://localhost:9200

Logstash

1. You need to have pipeline configuration files on LOGSTASH_PIPELINE_PATH location. If there will be no file, Logstash will throw an error and get exit.

NOTE

You can simply comment other stacks which is not needed. For example if you want to just run Elasticsearch & Kibana, Just comment the APM or other stack specification.

As a developer, you need to log everything it may be info, error or debug logs, etc. There are multiple types of log formats like Common log, JSON log, etc. and there are already solutions available in an elastic stack like filebeat to read JSON logs and push them to elasticsearch.

There can be cases where you need to log the data according to your convenience which will not be any standard log format. Or sometimes you just need to dump the log in an unstructured way but you need to have it in a structured format if you want to analyze those.

In this article, we are going to see how we can parse custom logs of any format into Elasticsearch. Elasticsearch provides the ingest pipeline with grok processor which will be able to match any unstructured log.

Overview

Ingest Pipeline

Elasticsearch gives ingest pipeline where it let you perform different types of operations or data transformations before indexing the data. The pipeline consists of a series of various processors.

Each processor in the pipeline takes input and produces the output. That output goes as input to the next processor.

Like below, After converting into lowercase letters by “lowercase processor” it sends the same string as input to the “split processor”.

Example

{"date":"HELLO - WORLD"} Input → |  lowercase processor ({"date":"hello - world"}) → split processor ({"date":"hello world"})   |  → Index

You can check more about ingest pipeline and processors.

Grok Processor

Grok Processor allows you to extract structured data from the ingested value on the specific field. Grok Processor works with grok patterns.

Let’s suppose you have the following document

{
    "Ip":"1.2.3.4",
    "country":"India",
    "log":"LEVEL:ERROR,method:fetchUser(),message:user not found"
}

You can mention the field name (Ex. log) where you want to apply a grok processor. The Grok processor uses the Grok pattern.

Grok pattern

Grok pattern is the regular expression which is the alias of some of the predefined expressions which can be reused.

Below is some sample grok patterns:

HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
SPACE \s*

Here HOSTNAME, USERNAME, USER & SPACE are Grok pattern names.

I can directly use SPACE label to define regex for space instead of a regular pattern like \s*. Also, the Grok pattern can be reused as the same as USERNAME.

Pattern list

You can find all pattern list on the below link:

https://github.com/elastic/elasticsearch/tree/8.1/libs/grok/src/main/resources/patterns

You can select any folder and explore all the patterns.

Grok pattern: https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/ecs-v1/grok-patterns

The flow

1. Create & Test Ingest Pipeline
2. Index custom data

Lets create the pipeline with a grok processor which is going to parse the message field with the document below.

{
    "Ip":"1.2.3.4",
    "country":"India",
    "message":"LEVEL:ERROR,method:fetchUser(),message:user not found,code:123"
}

I would like to extract the below data from message field and index into elasticsearch,

"method_name" : "fetchUser",
"log_level" : "ERROR",
"response_code" : "123",
"message" : "user not found"

Create & Test Ingest Pipeline

Syntax

%{SYNTAX:ID}

SYNTAX: It is the pattern name.
ID: It is the key name of the document.

Data need to parse

LEVEL:ERROR,method:fetchUser(),message:user not found

Let’s match some of the grok patterns from the below file https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/ecs-v1/grok-patterns

ERROR = %{LOGLEVEL:log_level}
fetchUser = %{WORD:method_name}
user not found = %{GREEDYDATA:message}
123 = code:(?<response_code>(?:[+-]?(?:[0-9]+)))

Data

LEVEL:ERROR,method:fetchUser(),message:user not found

Final Pattern

"LEVEL:%{LOGLEVEL:log_level},method:%{WORD:method_name}\\(\\),message:%{GREEDYDATA:message},code:(?<response_code>(?:[+-]?(?:[0-9]+)))"

Note: You can add your own regex as well. For example, I have added regex for code.

Test pipeline

For testing above pattern or pipeline we can use Simulate Pipeline API like below:

_simulate

POST /_ingest/pipeline/_simulate?pretty
{
  "pipeline": {
    "description" : "testing grok processor",
    "processors": [
      {
        "grok": {
          "field": "log",
          "patterns": ["LEVEL:%{LOGLEVEL:log_level},method:%{WORD:method_name}\\(\\),message:%{GREEDYDATA:message},code:(?<response_code>(?:[+-]?(?:[0-9]+)))"]
        }
      }
    ]
  },
  "docs":[
    {
      "_source": {
        "log": "LEVEL:ERROR,method:fetchUser(),message:user not found,code:123"
      }
    }
  ]
}

Response

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "response_code" : "123",
          "log" : "LEVEL:ERROR,method:fetchUser(),message:user not found,code:123",
          "method_name" : "fetchUser",
          "log_level" : "ERROR",
          "message" : "user not found"
        },
        "_ingest" : {
          "timestamp" : "2022-04-29T14:20:59.454111422Z"
        }
      }
    }
  ]
}

As you can see we got the desired output. Now let’s save the pipeline.

Save pipeline

Pipeline can be saved using _ingest/pipeline API.

PUT /_ingest/pipeline/custom_log?pretty
{
  "description" : "testing grok processor",
  "processors": [
    {
      "grok": {
        "field": "log",
        "patterns": ["LEVEL:%{LOGLEVEL:log_level},method:%{WORD:method_name}\\(\\),message:%{GREEDYDATA:message},code:(?<response_code>(?:[+-]?(?:[0-9]+)))"]
      }
    }
  ]
}

Custom_log: It is the pipeline name. We need to specify the pipeline name while indexing the data.

Index custom data

Now let’s index some data with the same pipeline which we created.

POST data-stream/_doc?pipeline=custom_log
{
  "log" : "LEVEL:INFO,method:addUser(),message:user added successfully.,code:200"
}

POST data-stream/_doc?pipeline=custom_log
{
  "log" : "LEVEL:DEBUG,method:deleteUser(),message:user_id notfound.,code:433"
}

POST data-stream/_doc?pipeline=custom_log
{
  "log" : "LEVEL:ERROR,method:fetchUser(),message:Database connection timeout,code:567"
}

Check all data

GET data-stream/_search?pretty

{
  "took" : 898,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "data-stream",
        "_id" : "3460dYABYeDE95hG23XJ",
        "_score" : 1.0,
        "_source" : {
          "response_code" : "200",
          "log" : "LEVEL:INFO,method:addUser(),message:user added successfully.,code:200",
          "method_name" : "addUser",
          "log_level" : "INFO",
          "message" : "user added successfully."
        }
      },
      {
        "_index" : "data-stream",
        "_id" : "Kwi2dYABbuPmoOiCSUjL",
        "_score" : 1.0,
        "_source" : {
          "response_code" : "433",
          "log" : "LEVEL:DEBUG,method:deleteUser(),message:user_id notfound.,code:433",
          "method_name" : "deleteUser",
          "log_level" : "DEBUG",
          "message" : "user_id notfound."
        }
      },
      {
        "_index" : "data-stream",
        "_id" : "LAi4dYABbuPmoOiCmUgb",
        "_score" : 1.0,
        "_source" : {
          "response_code" : "567",
          "log" : "LEVEL:ERROR,method:fetchUser(),message:Database connection timeout,code:567",
          "method_name" : "fetchUser",
          "log_level" : "ERROR",
          "message" : "Database connection timeout"
        }
      }
    ]
  }
}

Parsing common log format

Common log format is standard format for logging which is used by webservers and system log generators etc.

Let’s say we having below log:

97.92.139.153 1254 ashishtiwari [04/Mar/2022:15:18:55 +0530] "GET /niches HTTP/1.1" 201 2322

Grok Pattern

Final Grok Pattern

%{IPORHOST:ip} %{USER:user_id} %{USERNAME:username} \\[%{HTTPDATE:date}\\] \"%{WORD:request.method} %{URIPATH:request.path} %{URIPROTO:request.proto}/%{NUMBER:request.http_version}\" %{NUMBER:request.response} %{NUMBER:request.size_bytes}

_simulate

Lets simulate the pipeline with above grok pattern

POST /_ingest/pipeline/_simulate?pretty
{
  "pipeline": {
    "description" : "testing grok processor",
    "processors": [
      {
        "grok": {
          "field": "log",
          "patterns": ["%{IPORHOST:ip} %{USER:user_id} %{USERNAME:username} \\[%{HTTPDATE:date}\\] \"%{WORD:request.method} %{URIPATH:request.path} %{URIPROTO:request.proto}/%{NUMBER:request.http_version}\" %{NUMBER:request.response} %{NUMBER:request.size_bytes}"]
        }
      }
    ]
  },
  "docs":[
    {
      "_source": {
        "log": "97.92.139.153 1254 ashishtiwari [04/Mar/2022:15:18:55 +0530] \"GET /niches HTTP/1.1\" 201 2322"
      }
    }
  ]
}

Response

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_id" : "_id",
        "_source" : {
          "date" : "04/Mar/2022:15:18:55 +0530",
          "request" : {
            "path" : "/niches",
            "size_bytes" : "2322",
            "method" : "GET",
            "response" : "201",
            "proto" : "HTTP",
            "http_version" : "1.1"
          },
          "log" : """97.92.139.153 1254 ashishtiwari [04/Mar/2022:15:18:55 +0530] "GET /niches HTTP/1.1" 201 2322""",
          "user_id" : "1254",
          "ip" : "97.92.139.153",
          "username" : "ashishtiwari"
        },
        "_ingest" : {
          "timestamp" : "2022-04-29T14:42:51.765427484Z"
        }
      }
    }
  ]
}

As we can see it successfully parsed the data. You can save the pipeline and index the data as shown on the above steps.

Similarly you can parse any log format with Grok pattern.

Demo

You can also check the demo on the below link where I presented how you can parse common log format using ingest pipeline and grok processor.

https://ashish.one/talks/devops-conf-2022/

Conclusion

We have seen how we can parse any types of log format into elasticsearch with the help of ingest pipeline & grok processor. You can also use this ingest pipeline with filebeat or logstash. Where you just need to specify the pipeline name.

Feel free to put in comments if you have any doubts.

In this blog, we will try to understand how “Search as you type” works and Quickly setup one demo using some sample data. You must have seen various websites like eCommerce, food apps, etc. where you just start typing & simultaneously relevant options start displaying as suggestions and autocomplete. We will try to achieve somewhat the same feature. Search as you type Elasticsearch gives this specific mapping type which you can simply set to a specific field where you want to perform this kind of search.

Why search_as_you_type?

No need to think about what kind of functionality like analyzer, tokenizer, etc. you have to apply to achieve this. It automatically handles everything in the backend by producing necessary terms on which you can query efficiently.

You can simply create mapping like the below example:

Create Index

PUT products
{
  "mappings": {
    "properties": {
      "description": {
        "type": "search_as_you_type"
      }
    }
  }
}

Insert sample data

POST products/_doc/
{
  "description": "best jogging shoes for men"
}

How is data indexed ?

search_as_you_type mapping creates 4 types of fields in the backend.

Field 1: description

It will produce the terms according to the default analyzer if no analyzer is defined i.e. standard analyzer.

[
  "best",
  "jogging",
  "shoes",
  "for",
  "men"
]

Field 2: description._2gram

This will use a shingle token filter and produce the terms with shingle size 2. This means a shingle token filter produces the token by concatenating the adjacent token. You can find more here.

This operation will perform on all the terms which are created on the description field and it will produce the below terms.

[ 
  "best jogging",
  "jogging shoes",
  "shoes for",
  "for men"
]

Field 3: description._3gram

This will also use a shingle token filter and produce the terms with shingle size 3. This means it will concatenate 3 adjacent tokens like below.

[
  "best jogging shoes",
  "jogging shoes for",
  "shoes for men"
]

Field 4: description._index_prefix

This will apply an edge n gram token filter on the field description._3gram which means it will split terms (words) of description._3gram to a small substring that will start from the edge. You can have a look at the terms below.

[
  "b",
  "be",
  "bes",
  "best",
  "best ",
  "best j",
  "best jo",
  "best jog",
  "best jogg",
  "best joggi",
  "best joggin",
  "best jogging",
  "best jogging ",
  "best jogging s",
  "best jogging sh",
  "best jogging sho",
  "best jogging shoe",
  "best jogging shoes",
  "j",
  "jo",
  "jog",
  "jogg",
  "joggi",
  "joggin",
  "jogging",
  "jogging ",
  "jogging s",
  "jogging sh",
  "jogging sho",
  "jogging shoe",
  "jogging shoes",
  "jogging shoes ",
  "jogging shoes f",
  "jogging shoes fo",
  "jogging shoes for",
  "s",
  "sh",
  "sho",
  "shoe",
  "shoes",
  "shoes ",
  "shoes f",
  "shoes fo",
  "shoes for",
  "shoes for ",
  "shoes for m",
  "shoes for me",
  "shoes for men",
  "f",
  "fo",
  "for",
  "for ",
  "for m",
  "for me",
  "for men",
  "for men ",
  "m",
  "me",
  "men",
  "men ",
  "men  "
]

As you noticed the token limit is up to 3 words only because description._3gram has generated 3 word tokens only.

Search Query

multi_match

We will use a multi_match query here. Because we want to look up on each subfield for a perfect match.

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "jogging",
      "fields": [
        "description",
        "description._2gram",
        "description._3gram"
        ]
    }
  }
}

The above query is going to search the term "jogging" on all 3 subfields which are specified in fields[].

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "products",
        "_id" : "TSNNiX8BYh0NLleBiu4u",
        "_score" : 0.2876821,
        "_source" : {
          "description" : "best jogging shoes for men"
        }
      }
    ]
  }
}

Lets try with any substring (jog).

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "jog",
      "fields": [
        "description",
        "description._2gram",
        "description._3gram"
        ]
    }
  }
}

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

The result is empty. Because there is no term created with the name jog if you closely look at the above generated tokens on respective fields.

To solve this we need to use the bool_prefix query.

bool_prefix analyze the input and constructs the bool query from the terms. But it puts the last term in the prefix query. For example, input is given as men jogging s, So it will produce terms like ["men","jogging","s"] but it will always perform a prefix query on the last term which is "s". So documents will return where terms will match with "men" or "jogging" or any term which is starting with "s".

Below is the query which will give you the desired output.

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "jog",
      "type": "bool_prefix", 
      "fields": [
        "description",
        "description._2gram",
        "description._3gram"
        ]
    }
  }
}

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "products",
        "_id" : "TSNNiX8BYh0NLleBiu4u",
        "_score" : 1.0,
        "_source" : {
          "description" : "best jogging shoes for men"
        }
      }
    ]
  }
}

Here when we make a prefix query on the root field (description) or any subfields, It will rewrite the query as a term query on description._index_prefix field.

This matches more efficiently because prefixes up to 3 words are already created as the terms as shown in the above.

Note: This query will search for terms irrespective of order. For example, if we search for jogging men, This will also give the result because it will search for both the terms jogging or men. In most of the cases this query (multi_match + bool_prefix) is recommended because the end user can search for any string like shoes or shoes for men or jogging shoes etc.

What if you want to search with strict prefix order?

You can use match_phrase_prefix, It will strictly match input from prefix in the same order only. So input like “men best” won’t return anything. Whereas you will get results with the previous one.

GET products/_search
{
  "query": {
    "match_phrase_prefix": {
      "description": "best jogging s"
    }
  }
}

It will return documents where the term’s prefix will be matched with “best jogging s”. Sometimes it can provide confusing results. You can check more about match_phrase_prefix.

Demo

Let’s take a practical experience of how it is going to work.

Search As You Type (Elasticsearch)

Demo code and sample employees data to implement the “Search as you type” feature on elasticsearch.

Written the middleware API in python using flask. Used JQuery for javascript operations.

Installation

Assuming you have successfully installed Elasticsearch and Kibana on your machine and it is working perfectly. Kindly refer respective installation document.

OR

You can run Elasticsearch on the cloud with a few clicks.

Install Python3 & pip3

1. Refer Document to install python3 & pip3 on your system.
2. Install flask

pip3 install flask

3. Install elasticsearch package

pip3 install elasticsearch

git Clone

git clone https://github.com/ashishtiwari1993/search_as_you_type.git
cd search_as_you_type

Create Index and load data

Make sure Elasticsearch and kibana are up and running fine on your machine.

Create Index

PUT /sayt?pretty
{
  "mappings": {
    "properties": {
      "first_name": {
        "type": "search_as_you_type"
      },
      "last_name": {
        "type": "search_as_you_type"
      },
      "street_address": {
        "type": "search_as_you_type"
      },
      "company": {
        "type": "search_as_you_type"
      },
      "email": {
        "type": "search_as_you_type"
      }
    }
  }
}

Load sample data

Sample data.json file is given which need to load with the help of bulk API.

curl -s -H "Content-Type: application/x-ndjson" -XPOST "localhost:9200/_bulk" --data-binary "@data.json"

Do not forget to change the elasticsearch’s endpoint.

Run api.py & test

Open api.py and change elasticsearch endpoint accordingly.

es = Elasticsearch("http://localhost:9200")

Start API Server

python3 api.py

This will start the API service on port 5001.

Open index.html on your browser.

In the talk, Quickly showed how you can use the metricbeat to check system health. With a few clicks, you can start monitoring your infra.

Another demo shows, How you can parse your custom unstructured logs to Elasticsearch. There is some utility already available for predefined logs format like json, apache, nginx and system logs etc.

But if you have different requirement where you need to parse a different types of log format , You can parse using Grok processor.

Also explained how you can read custom logs from log files in real time and ingest to Elasticsearch.

Slides

Talk Video

Feel free to comment below, If you have any doubts or suggestion about this talk.

What this talk is all about ?

The purpose of the talk is to give a short overview of Elastic solutions & Elastic stacks. In the demo shown, how you can deploy elasticsearch instance on Microsoft Azure.

Also, it gives an idea to use the elastic cloud to manage the elasticsearch instance which deployed on the Azure cloud. You can also create deployment on elastic cloud (cloud.elastic.co).

In the demo, Successfully shipped the metric data of the local system (my MacBook) to the newly deployed elasticsearch instance and explored the dashboard on kibana.

Slides

Talk Video

Feel free to comment below, If you have any doubts or suggestion about this talk.

Introduction

This article is not containing the detailed procedure for installation but listed some challenges which you can face while installing the arch Linux. Arch Linux is one of the Linux distributions, Which gives you full control over your application and OS. You have complete freedom to install or what to keep. Unlike the other distribution like Ubuntu, Centos, etc. It doesn’t come with pre-loaded applications or software.

Though Arch wiki has a detailed explanation of the installation process I faced some challenges whose solution was not easily available. I am listing a few here and will keep updating this post as I move forward.

In this post I am not going to cover the installation steps, You will get a better explanation on the arch wiki.

Machine Configuration

Hardware

RAM: 2GB
Disk: 500

BIOS configuration

Secure boot: disabled
Legacy boot: enabled
Boot Mode: UEFI

Challenges

1. What should be my disk partition ?

1.1 Check Boot mode

You need to check installation guide, Refer step Verify the boot mode.

In a live environment, you need to check first what is your boot mode.

ls /sys/firmware/efi/efivars

If the command shows the directory without error, then the system is booted in UEFI mode. If the directory does not exist, the system may be booted in BIOS (or CSM) mode.

So my machine’s boot mode is UEFI.

1.2 How to create partitions?

I would recommend using cfdisk command to create the partition. Which gives you good visualization.

My SCSI Disk is dev/sda.

1.3 Types of Partition

You need to create 3 types of partition as shown below:

Here on my machine i have created 3 partition name dev/sda1, dev/sda2, dev/sda3. This name can be different.

This is how you need to create the partition with the help of the cfdisk command.

1.4 Mount the partition

Format the partition

mkfs.ext4 /dev/sda3
mkswap /dev/sda2
mkfs.fat -F 32 /dev/sda1

Mount the partition

mount /dev/sda3 /mnt
mount /dev/sda1 /mnt/boot
swapon /dev/sda2

2. Install Bootloader

I have decided to install a grub bootloader on my machine. You can decide on this wiki page.

2.1 Install GRUB

pacman -S grub

2.2 Generate Grub config

grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=GRUB

2.3 Reboot the system

exit
umount /dev/sda3
reboot

3. How to login again with Live bootable Arch device (USB/CD etc.)

Sometimes we need to install some package with a live environment only. In such a case, you can prefer these steps.

Attach the live bootable device and choose the specific device in the boot order. Once you are logged in, Just perform the below commands.

mount /dev/sda3 /mnt
mount /dev/sda1 /mnt/boot
swapon /dev/sda2

arch-chroot /mnt

Once you are in Than you can perform any operation on your installed Arch OS.

4. Install NetworkManager

Please refer to challenge #3 to log in with the live environment. Once you successfully log in, Perform the below commands:

pacman -S NetworkManager
systemctl start NetworkManager
systemclt enable NetworkManager

Simply check with ping if you connected with a LAN connection.

ping google.com

Note: Please start and enable the Network Manager after reboot. It will not start on first time login.

5. Install Broadcom Wireless

pacman -S linux-headers
pacman -S broadcom-wl-dkms

Reboot the system.

6. Desktop Environment Arch

To run any desktop environment, Need to install Display Manager. First, we will install LXDM (display manager).

6.1 LXDM

pacman -S lxdm
systemctl start lxdm
systemctl enable lxdm

6.2 LXDE

LXDE is a lightweight desktop. As my system has 2GB RAM, I have tried this.

pacman -S lxde

Reboot the system.

6.3 XFCE4

Xfce is a lightweight and modular desktop environment currently based on GTK 3. To provide a complete user experience, it includes a window manager, a file manager, desktop and panel.

pacman -S xfce4 xfce4-goodies

Edit lxdm.conf

Edit below line

session=/usr/bin/startlxfce4

Reboot the system.

7. Connect to WIFI

You need to install Network Manager and wireless driver first. You can refer step 4 & 5.

Once done with proper installation you can connect to wifi using:

nmtui

8. Stuck on GRUB window while booting on arch linux

It seems my bootloader was not configured properly. Re Attached the live bootable device and perform steps 3 and then 2.

Reboot the system.

9. Error: [ TIME ] Timed out waiting for device /dev/disk/by-uuid/

Please perform step 3.

Check UUID of partitions

lsblk -l

Cross-check whether the above UUIDs and mentioned UUIDs in /etc/fstab should be the same.

vim /etc/fstab

Update UUID accordingly on the above file. Save and exit.

Reboot the system.

The Fact

The fact is there is no hard and fast rule or there is no standard course by following which you will get the tag of a security expert.

There is no defined way to become a security researcher.

Here at the end, I am also going to give you some resources about the get started but to be honest it will not help until you start with a practice or use it in your day-to-day life for learning purposes.

So what missing?

Resources will get the directions for learning. Let’s say you got one blog where the blogger explains everything steps by step and You decided to complete it in 7 days. Your approach SHOULD NOT be that once you finish with the whole tutorial you should start with researching it won’t help or probably you must have forgotten the things which you learned on day 1.

The Approach

Start performing practical from the first day. Whatever your learning start implementing those. Try to find the smallest loopholes. And trust me you won’t get it on Day 1. But with the practice, you will become to know how to find or where to find.

Where to do practice?

You can use any Bugbounty side where lots of products, apps, websites get listed to find bugs, vulnerabilities, etc. You can also use CTF (Capture the flag). I will not go in deep with these terminologies because you can find better explanations on google. You will also get a better guide on how to start with these platforms.

Don’t Forget your practice will give you a real security expert badge. Keep practicing.

Some resources to get start

• https://github.com/nahamsec/Resources-for-Beginner-Bug-Bounty-Hunters

• https://whoami.securitybreached.org/2019/06/03/guide-getting-started-in-bug-bounty-hunting/

• https://infosecwriteups.com/guide-to-basic-recon-bug-bounties-recon-728c5242a115

• Pentesting for n00bs is ELI5 hacking, designed for beginners who have never walked through a hack before. All 10 episodes:

Ep 1 - Legacy: https://youtu.be/JZN3JhoAdWo

Ep 2 - Lame: https://youtu.be/ntBkyid_u8Y

Ep 3 - Blue: https://youtu.be/xLI7OialKk4

Ep 4 - Devel: https://youtu.be/ODUDau7BPSY

Ep. 5 - Jerry: https://youtu.be/nF14K2VAVtw

Ep. 6 - Nibbles: https://youtu.be/8ulnQVFHcOE

Ep. 7 - Optimum: https://youtu.be/bTxnobhJ_b8

Ep. 8 - Bashed: https://youtu.be/5406MfOfXBc

Ep. 9 - Grandpa: https://youtu.be/3aASluoJ-iM

Ep. 10 - Netmon: https://youtu.be/8k-8aVwS0fk

Please feel free to post your doubt or suggestions below :slight_smile:

Thanks

Prerequisite

Go version

$ go version
go version go1.13 linux/amd64

Dependency

DNS Library(https://github.com/miekg/dns)

Install dependency

$ go get github.com/miekg/dns 

spfLookup.go

Here you can change nameserver according to your requirement. I have specified here google’s name server (8.8.8.8). You can also use cloudflare’s nameserver (1.1.1.1)

Conclusion

You can make any DNS query with miekg/dns library. In the above script, we have looked up TXT Records and then we have searched for a string containing v=spf1.

Goal of this blog

In this blog, I am going to show you how easily we can write logs to the files in Golang. As well as we are going to store all logs on elasticsearch with EKB (Elasticsearch, Kibana, Beats).

Why ELKB stack ?

Logs are very important for debugging, reporting, insights etc. In today’s tech world, We uses multiple cloud servers, private servers etc. Which consist of lots of different applications, scripts, programs, daemons, services and they generate their logs too. It is very difficult to go to each server and check all log files in case of debugging or to generate any insights or reporting.

In my case I used to go to every server and perform grep on log files.

ELKB Gives you an easy setup to move all yours logs into one central Place which is Elasticsearch & With a beautiful kibana interface, You can visualize all your logs. You can also make various types of dashboards.

This is a very small use case I am going to convert but you can find lots of big problem solving use cases on the internet about ELKB.

I will not go much deeper about ELKB stack. You can find a wonderful explanation on elastic.co.

Two Parts:

1. Write JSON logs to the file in Go
2. Shift All JSON logs on Elasticsearch

My Configuration

OS: ubuntu 18.04
Go version: go version go1.13 linux/amd64
Elasticsearch version: 7.7.1
Filebeat version: filebeat version 7.7.1 (amd64)
Kibana version: 7.7.1

1. Write JSON logs to the file in Go

I am assuming you have already installed Go on your machine. If not then you can refer to the installation guide here.

My Golang version:

go version go1.13 linux/amd64

Logrus

This is a wonderful package available to write logs. Below is the short example of using logrus package:

1. Install Logrus package

go install github.com/sirupsen/logru

2. Create log.go and paste the below code

package main

import (
  "os"
  log "github.com/sirupsen/logrus"
)

func init() {

  f, _ := os.OpenFile("/tmp/go_logs/example.log", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
  log.SetFormatter(&log.JSONFormatter{})
  log.SetOutput(f)

}

func main() {

  log.WithFields(log.Fields{
    "event": "create_profile",
    "user_id":   10,
  }).Info("This is an info message.")

  log.WithFields(log.Fields{
    "event": "delete_profile",
    "user_id": 11,
  }).Warn("This is a warning message.")

  log.WithFields(log.Fields{
		"event" : "edit_profile",
    "user_id": 13,
		"package" : "main",
  }).Fatal("This is a critical message.")

}

Here I have specified the log file /tmp/go_logs/example.log. You can specify according to your need. We also specified the log format JSON.

3. Lets run the log.go

$ go run log.go

4. Check log file:

$ cat /tmp/go_logs/example.log

Output

{"event":"create_profile","level":"info","msg":"This is an info message.","time":"2020-06-06T22:51:30+05:30","user_id":10}
{"event":"delete_profile","level":"warning","msg":"This is a warning message.","time":"2020-06-06T22:51:30+05:30","user_id":11}
{"event":"edit_profile","level":"fatal","msg":"This is a critical message.","package":"main","time":"2020-06-06T22:51:30+05:30","user_id":13}

Here I have used the log format JSON. Every log will be written in JSON format on the newline. You can check more features about logrus here.

2. Shift All JSON logs on Elasticsearch

This part has no dependency on the above part. You can use any JSON log file irrespective of any language.

Before start I am assuming you have installed Elasticsearch, Filebeat & Kibana on your machine. If not then refer below link:

• Install Elasticsearch
• Install Filebeat
• Install Kibana

Note: In this part we are not going to use Logstash.

1. Start Elasticsearch Service

$ service elasticsearch start

It will run on port 9200. You can verify with the below command or just hit localhost:9200 on your browser.

ashish@ashish-laptop:~$ curl localhost:9200

{
  "name" : "753853fa62d1",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "JLlH0Z0pQqWjGEHO8MQgZQ",
  "version" : {
    "number" : "7.7.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
    "build_date" : "2020-05-28T16:30:01.040088Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

2. Start Kibana service

$ service kibana start

It will run on port 5601. You can verify by visiting localhost:5601 from your browser. You should see the kibana dashboard.

3. Edit filebeat.yml

Open filebeat.yml. Add below snippet in filebeat.inputs:

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /tmp/go_logs/*.log
  json.add_error_key: true

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

You can check on below link to know more about filebeat log input & JSON decoding

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json

4. Restart Filebeat

$ service filebeat restart

Verify If Data Indexed on Elasticsearch

Check Logs

$ tail -f /var/log/filebeat/filebeat

Check Indices:

$ curl localhost:9200/_cat/indices?v

Or you can simply visit localhost:9200/_cat/indices?v on your browser.

Output:

health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .apm-custom-link                 xp0mitnBQtijaZ9tEgan_g   1   0          0            0       208b           208b
green  open   .kibana_task_manager_1           7Q4mMTYxRhCB6sfnQ2ibmA   1   0          5            0       34kb           34kb
green  open   .apm-agent-configuration         3piA79spTbGWAVItYL3PlQ   1   0          0            0       208b           208b
yellow open   filebeat-7.7.1-2020.06.06-000001 nsFk7mOuTguIfaPSbeM3PA   1   1         19            0     74.9kb         74.9kb
green  open   .kibana_1                        LBmzoJspR8a8HAcs9WGr8g   1   0         54            0    171.6kb        171.6kb

As you can see a new index is created by filebeat with the name filebeat-7.7.1-2020.06.06-000001. In your case index name can be different but it will start with filebeat*.

Index lifecycle will be handled by filebeat. You don’t need to worry about that.

Check Documents

$ curl localhost:9200/filebeat-7.7.1-2020.06.06-000001/_search?pretty

Sample Output:

{
  "_index" : "filebeat-7.7.1-2020.06.06-000001",
  "_type" : "_doc",
  "_id" : "4qW3inIBVJJF9hMQm_yi",
  "_score" : 1.0,
  "_source" : {
    "@timestamp" : "2020-06-06T17:39:47.208Z",
    "log" : {
      "offset" : 0,
      "file" : {
        "path" : "/tmp/go_logs/ashish.log"
      }
    },
    "json" : {
      "msg" : "This is an info message.",
      "time" : "2020-06-06T23:09:26+05:30",
      "user_id" : 10,
      "event" : "create_profile",
      "level" : "info"
    },
    "input" : {
      "type" : "log"
    },
    "host" : {
      "name" : "ashish-laptop"
    },
    "agent" : {
      "version" : "7.7.1",
      "type" : "filebeat",
      "ephemeral_id" : "3f53f9c1-66a0-4e93-85fa-1532221c9670",
      "hostname" : "ashish-laptop",
      "id" : "72976284-d927-49d1-abcb-1d2a5be15176"
    },
    "ecs" : {
      "version" : "1.5.0"
    }
  }
}

Your data will be saved on the json key. Another is metadata which is added by filebeat. Filebeat provides lots of options, features & settings. You can use it according to your requirements.

Logs on Kibana

Visit on localhost:5601 from your browser. Create index pattern with the pattern filebeat*. Check here for more details.

Once you are done with defining the index pattern, Go to Discover section, Here you will see all your logs.

You can query on your logs. You can apply filters & create various types of dashboard to get better insights from your logs. You can find a wonderful explanation on the official site.

In the end

We have successfully shipped our logs on Elasticsearch. This is only a small use case of ELKB Stack. It provides much more than this. You can explore more on the internet.

Put your comments if you have any doubts.

In this blog, We are going to setup PHP and Nginx.

Why Docker?

I will not go too much deep, You can find more resources over the internet about the docker.

Docker makes the installation process very smooth and it gives your isolated environment as the container.

With the docker, There will be no more excuses like:

”It’s working on my machine.” :D

Because your docker environment remains the same across the platforms, So above excuses will going to turn with:

”It’s working on every machine” ;)

In Short, Docker makes tech person’s life easy like the developer, tester, etc.

My Machine configuration:

Docker Version: 18.09.7
Docker Compose Version: 1.24.1
RAM: 8GB
OS: Ubuntu 18.04

Make sure docker and docker-compose is installed on your machine.

Step 1: Create folders:

mkdir LEMP
cd LEMP
mkdir {public_html,nginx_conf}

public_html: It will contains your PHP code.
nginx_conf: It will contains your Nginx configuration file.

Step 2: Create Nginx conf file

Go to nginx_conf:

cd nginx_conf

Create file lemp-docker.conf (You can give any name according to your requirement) & Paste below configuration block.

server {
		listen 80;
		server_name _;
		root /public_html;

		location / {
				index index.php index.html;
		}

		location ~* \.php$ {
				fastcgi_pass    php:9000;
				fastcgi_index   index.php;
				include         fastcgi_params;
				fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
				fastcgi_param PATH_INFO $fastcgi_path_info;
		}
}

Save and Exit from folder.

Step 3: Create file docker-compose.yml

Create file docker-compose.yml in LEMP folder & paste the below lines:

version: '3'

services:
 web:
  image: nginx
  ports:
   - "80:80" 
  volumes:
   - ./public_html:/public_html
   - ./nginx_conf:/etc/nginx/conf.d
   - /tmp/nginx_logs:/var/log/nginx 
  networks:
   - nginx-php

 php:
  image: php:7.2-fpm
  volumes:
  - ./public_html:/public_html
  networks:
  - nginx-php

networks:
 nginx-php:

Here we have created the network with the name nginx-php. You can check more about the networks here.

You can set volumes path according to your requirement.

Step 4: Let’s Up the containers

Create container:

cd LEMP/
docker-compose up

Note: Make sure no other service is running on port 80.

Output:

You can hit the localhost on your browser.

Once you hit the localhost check nginx logs:

cat /tmp/nginx_logs/access.log 

172.25.0.1 - - [16/May/2020:12:51:04 +0000] "GET / HTTP/1.1" 404 556 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36" "-"

Now stop the containers by simply hitting Ctrl + c. Now run docker-compose in detached mode. It will run in the background.

docker-compose up -d

Lists containers

docker-compose ps

Step 5: Let’s Create test.php

Go to LEMP/public_html

cd LEMP/public_html

Create file test.php & paste below code:

<?php

echo phpinfo();

Step 6: Run test.php

Visit http://localhost/test.php.

Output:

At the End

We have successfully created Nginx and PHP Environment. In Part 2, We will check, How we can add MySQL to this environment.

Hugo version:

$ hugo version

Hugo Static Site Generator v0.69.0-4205844B linux/amd64 BuildDate: 2020-04-10T09:12:34Z

Step 1: Create Shortcode

Create gslides.html file

vim layouts/shortcodes/gslides.html

Step 2: Add below code:

<div id="Container"
 style="padding-bottom:56.25%; position:relative; display:block; width: 100%">
 <iframe id="googleSlideIframe"
  width="100%" height="100%"
  src="{{ .Get "src" }}"
  frameborder="0" allowfullscreen=""
  style="position:absolute; top:0; left: 0"></iframe>
</div>

Step 3: Use shortcode ‘gslides’ in your Blog/Post Markdown file

Simply place below snippet in your markdown file.

Replace src value with your Google slide URL.

{{< gslides src="https://docs.google.com/presentation/d/e/2PACX-1vQExSl-gRPoA9hC6qXuqrjwiQVHAanDieZN_5GpV2Lw9cuxjsVFEN_wkTThqpQwZ36vJz4zwmTvV7cC/embed?start=false&loop=false&delayms=3000" >}}

I have shown for google slides, Similarly, you can create or add any other document type like google sheets, doc, etc. OR you can place any iframe like this.

Successfully built custom #Linux system by referring https://t.co/lyrdKkfXo9 .

Listed my challenges here https://t.co/gU2SoAxly2@nixcraft#linux #lfs #linuxfromscratch pic.twitter.com/XlWchVHy26

— Ashish Tiwari 🇮🇳 (@_ashish_tiwari) April 27, 2020

I have started with www.linuxfromscratch.org, I am facing some challenges and problems which I am going to share with you in this blog. I will keep updating this blog as well as I move forward.

Before we get start

You need some basic linux command knowledge to use linuxfromscratch guide. I am going to build LFS on my local machine. If your machine has a different OS or different configuration, Then some solution will not work.

I am going to use the same chapter title as mentioned in linuxfromscratch. I will skip those sections where I haven’t faced any difficulties.

My Machine Configuration

OS: Ubuntu 14.04
RAM: 2GB
Disk: 500 GB

Going to build

LFS version: 9.1
Published: March 1st, 2020
LFS doc:http://www.linuxfromscratch.org/lfs/downloads/9.1/LFS-BOOK-9.1-NOCHUNKS.html

Challenges

2.2. Host System Requirements

Doc link: http://www.linuxfromscratch.org/lfs/downloads/9.1/LFS-BOOK-9.1-NOCHUNKS.html#ch-partitioning-hostreqs

Version check

cat > version-check.sh << "EOF"
#!/bin/bash
# Simple script to list version numbers of critical development tools
export LC_ALL=C
bash --version | head -n1 | cut -d" " -f2-4
MYSH=$(readlink -f /bin/sh)
echo "/bin/sh -> $MYSH"
echo $MYSH | grep -q bash || echo "ERROR: /bin/sh does not point to bash"
unset MYSH

echo -n "Binutils: "; ld --version | head -n1 | cut -d" " -f3-
bison --version | head -n1

if [ -h /usr/bin/yacc ]; then
  echo "/usr/bin/yacc -> `readlink -f /usr/bin/yacc`";
elif [ -x /usr/bin/yacc ]; then
  echo yacc is `/usr/bin/yacc --version | head -n1`
else
  echo "yacc not found" 
fi

bzip2 --version 2>&1 < /dev/null | head -n1 | cut -d" " -f1,6-
echo -n "Coreutils: "; chown --version | head -n1 | cut -d")" -f2
diff --version | head -n1
find --version | head -n1
gawk --version | head -n1

if [ -h /usr/bin/awk ]; then
  echo "/usr/bin/awk -> `readlink -f /usr/bin/awk`";
elif [ -x /usr/bin/awk ]; then
  echo awk is `/usr/bin/awk --version | head -n1`
else 
  echo "awk not found" 
fi

gcc --version | head -n1
g++ --version | head -n1
ldd --version | head -n1 | cut -d" " -f2-  # glibc version
grep --version | head -n1
gzip --version | head -n1
cat /proc/version
m4 --version | head -n1
make --version | head -n1
patch --version | head -n1
echo Perl `perl -V:version`
python3 --version
sed --version | head -n1
tar --version | head -n1
makeinfo --version | head -n1  # texinfo version
xz --version | head -n1

echo 'int main(){}' > dummy.c && g++ -o dummy dummy.c
if [ -x dummy ]
  then echo "g++ compilation OK";
  else echo "g++ compilation failed"; fi
rm -f dummy.c dummy
EOF


bash version-check.sh

Error 1:

/bin/sh does not point to bash

sudo ln -sf bash /bin/sh

Error 2:

version-check.sh: line 11: bison: command not found

sudo apt-get install bison

Error 3:

version-check.sh: line 25: gawk: command not found

sudo apt-get install gawk

Error 4:

version-check.sh: line 36: g++: command not found

sudo apt-get install g++

Error 5:

version-check.sh: line 48: makeinfo: command not found

sudo apt-get update -y
sudo apt-get install -y texinfo

Final success output:

$ bash version-check.sh

bash, version 4.3.11(1)-release
/bin/sh -> /bin/bash
Binutils: (GNU Binutils for Ubuntu) 2.24
bison (GNU Bison) 3.0.2
/usr/bin/yacc -> /usr/bin/bison.yacc
bzip2,  Version 1.0.6, 6-Sept-2010.
Coreutils:  8.21
diff (GNU diffutils) 3.3
find (GNU findutils) 4.4.2
GNU Awk 4.0.1
/usr/bin/awk -> /usr/bin/gawk
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4
g++ (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4
(Ubuntu EGLIBC 2.19-0ubuntu6.15) 2.19
grep (GNU grep) 2.16
gzip 1.6
Linux version 4.4.0-142-generic (buildd@lcy01-amd64-006) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4) ) #168~14.04.1-Ubuntu SMP Sat Jan 19 11:26:28 UTC 2019
m4 (GNU M4) 1.4.17
GNU Make 3.81
GNU patch 2.7.1
Perl version='5.18.2';
Python 3.4.3
sed (GNU sed) 4.2.2
tar (GNU tar) 1.27.1
makeinfo (GNU texinfo) 5.2
xz (XZ Utils) 5.1.0alpha
g++ compilation OK

2.5. Creating a File System on the Partition

Doc link: http://www.linuxfromscratch.org/lfs/downloads/9.1/LFS-BOOK-9.1-NOCHUNKS.html#ch-partitioning-creatingfilesystem

I had only one partition. I need to create new physical partition for LFS.

It is recommended to use Physical partition only not logical.

I have reinstall my OS with new partition which is long way. You can use GParted to make partitions. Remember you can create only 4 Physical partition. Give 1 physical partition dedicated to LFS. Below is my partition view.

$ lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 465.8G  0 disk 
|-sda1   8:1    0   476M  0 part /boot/efi
|-sda2   8:2    0   1.9G  0 part [SWAP]
|-sda3   8:3    0 190.8G  0 part /
`-sda4   8:4    0 272.7G  0 part /lfs

Challenge: My Partition wipe out after restart my system

After restart I again hit the same command lsblk. /lfs partition was not there.

In Ubuntu, You need to make entry in /etc/fstab in order to mount the partition on restart.

You have to find first volumeId or UUID to make entry in /etc/fstab. You can use blkid command to get all UUID:

$ blkid

/dev/sda1: UUID="1FBD-F51A" TYPE="vfat" 
/dev/sda2: UUID="9f41c453-3b5c-4537-81b0-2fb33f19ded8" TYPE="swap" 
/dev/sda3: UUID="1a16afba-cabb-40c7-b704-1129b7004ffb" TYPE="ext4" 
/dev/sda4: UUID="b3b2eb8c-2a43-405d-97da-af8b46c743e2" TYPE="ext4" 

Above my Partition is /dev/sda4 which is mounted on /lfs, So my UUID is b3b2eb8c-2a43-405d-97da-af8b46c743e2. Now I have added below line in /etc/fstab.

UUID=b3b2eb8c-2a43-405d-97da-af8b46c743e2 /lfs            ext4    rw,user,exec,errors=remount-ro              0       0

Here

• rw: Read, Write
• exec: Execute
• errors=remount-ro: It will mount as Read-Only, If you get any error on mounting.

Make sure you have added exec, Otherwise you cannot execute any binaries from chapter 5.4. Binutils-2.34 - Pass 1

5.4. Binutils-2.34 - Pass 1

Doc link: http://www.linuxfromscratch.org/lfs/downloads/9.1/LFS-BOOK-9.1-NOCHUNKS.html#ch-tools-binutils-pass1

Error 1:

bash: ../configure: No such file or directory

As I mentioned on above Section, Make sure you have added exec in your /etc/fstab entry OR Make sure your partition /lfs has Read, Write & Execute permission. Below /etc/fstab entry worked for me:

UUID=b3b2eb8c-2a43-405d-97da-af8b46c743e2 /lfs            ext4    rw,user,exec,errors=remount-ro              0       0

This setting can be different for different OS.

Error 2:

I had no idea where to create build folder.

Below steps worked for me:

$ cd $LFS/source
$ tar xvf binutils-2.34.tar.xz
$ cd binutils-2.34
$ mkdir -v build
$ cd build
$ ../configure --prefix=/tools          \
             --with-sysroot=$LFS        \
             --with-lib-path=/tools/lib \
             --target=$LFS_TGT          \
             --disable-nls              \
             --disable-werror

5.5. GCC-9.2.0 - Pass 1

cd $LFS/sources
tar -xvf gcc-9.2.0.tar.xz
cd gcc-9.2.0

After this you can continue with document flow.

tar -xf ../mpfr-4.0.2.tar.xz
mv -v mpfr-4.0.2 mpfr
tar -xf ../gmp-6.2.0.tar.xz
mv -v gmp-6.2.0 gmp
tar -xf ../mpc-1.1.0.tar.gz
mv -v mpc-1.1.0 mpc

Error 1:

On make command, I got below error:

error: ‘for’ loop initial declarations are only allowed in C99 mode

compute_powtab.c: In function 'mpn_compute_powtab_mul':
compute_powtab.c:142:3: error: 'for' loop initial declarations are only allowed in C99 mode
   for (long pi = start_idx; pi >= 0; pi--)
   ^
compute_powtab.c:142:3: note: use option -std=c99 or -std=gnu99 to compile your code
compute_powtab.c: In function 'mpn_compute_powtab_div':
compute_powtab.c:226:3: error: 'for' loop initial declarations are only allowed in C99 mode
   for (long pi = n_pows - 1; pi >= 0; pi--)
   ^
compute_powtab.c:274:13: error: redefinition of 'pi'
   for (long pi = n_pows; pi >= 0; pi--)
             ^
compute_powtab.c:226:13: note: previous definition of 'pi' was here
   for (long pi = n_pows - 1; pi >= 0; pi--)
             ^
compute_powtab.c:274:3: error: 'for' loop initial declarations are only allowed in C99 mode
   for (long pi = n_pows; pi >= 0; pi--)
   ^
compute_powtab.c: In function 'powtab_decide':
compute_powtab.c:296:3: error: 'for' loop initial declarations are only allowed in C99 mode
   for (size_t pn = (un + 1) >> 1; pn != 1; pn = (pn + 1) >> 1)
   ^
compute_powtab.c:304:10: error: redefinition of 'pn'
   size_t pn = un - 1;
          ^
compute_powtab.c:296:15: note: previous definition of 'pn' was here
   for (size_t pn = (un + 1) >> 1; pn != 1; pn = (pn + 1) >> 1)
               ^
compute_powtab.c:308:3: error: 'for' loop initial declarations are only allowed in C99 mode
   for (long i = n_pows - 2; i >= 0; i--)

As per (https://stackoverflow.com/a/29338269)[https://stackoverflow.com/a/29338269],

This happens because declaring variables inside a for loop wasn’t valid C until C99(which is the standard of C published in 1999).