Shipping Golang logs with ELKB stack

Goal of this blog In this blog, I am going to show you how easily we can write logs to the files in Golang. As well as we are going to store all logs on elasticsearch with EKB (Elasticsearch, Kibana, Beats). Why ELKB stack ? Logs are very important for debugging, reporting, insights etc. In today’s tech world, We uses multiple cloud servers, private servers etc. Which consist of lots of different applications, scripts, programs, daemons, services and they generate their logs too.

[Part 1] Setup LEMP environment with Docker - Setup Nginx and PHP

Hi guys, In this series, we are going to setup LEMP Stack (Linux, Nginx, MySQL, PHP). Mainly it is used by web developers. I am assuming you have a basic idea about Docker & How it works. In this blog, We are going to setup PHP and Nginx. Why Docker? I will not go too much deep, You can find more resources over the internet about the docker. Docker makes the installation process very smooth and it gives your isolated environment as the container.

Add Responsive Google Slides on Hugo

Steps to add Responsive google slides iframe with Hugo: Hugo version: $ hugo version Hugo Static Site Generator v0.69.0-4205844B linux/amd64 BuildDate: 2020-04-10T09:12:34Z Step 1: Create Shortcode Create gslides.html file vim layouts/shortcodes/gslides.html Step 2: Add below code: <div id="Container" style="padding-bottom:56.25%; position:relative; display:block; width: 100%"> <iframe id="googleSlideIframe" width="100%" height="100%" src="{{ .Get "src" }}" frameborder="0" allowfullscreen="" style="position:absolute; top:0; left: 0"></iframe> </div> Step 3: Use shortcode ‘gslides’ in your Blog/Post Markdown file Simply place below snippet in your markdown file.

Challenges in linuxfromscratch

I have started with www.linuxfromscratch.org, I am facing some challenges and problems which I am going to share with you in this blog. I will keep updating this blog as well as I move forward. Before we get start You need some basic linux command knowledge to use linuxfromscratch guide. I am going to build LFS on my local machine. If your machine has a different OS or different configuration, Then some solution will not work.

[Part 4] Setup Grafana With Prometheus

As you know Prometheus already having UI (localhost:9090). But it is not enough to give you better visualization on one screen. For better visualization and a graphical representation, we are going to use Grafana. What is Grafana? As grafana.com says ”Grafana is the open-source analytics and monitoring solution for every database.” This means Grafana is an independent tool for analytics and monitor which gives your various types of Graphs.

Install python3.6, pip3.6, pipenv on Ubuntu 14.04 LTS

Prerequisite OS: Ubuntu 14.04 LTS Processor: 64 Bit RAM: 2 GB 1. Install python3.6 From source Step 1.1: Compile $ wget https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz $ tar -xvf Python-3.6.3.tgz $ cd Python-3.6.3 $ sudo ./configure --enable-optimizations Step 1.2: Check $ python3.6 --version 2. Install pip3.6 Step 2.1: Download pip $ wget https://bootstrap.pypa.io/get-pip.py Step 2.2: Execute $ sudo python3.6 get-pip.py Step 2.3: If you Got below error Error zlib not available Traceback (most recent call last): File "get-pip.

Covid19 Data Source for India & Global

Hi Guys, I am trying to listing all data source & endpoints for COVID19 - India as well as Global. It can contains offical or unofficial APIs. Anyone is working on any COVID19 project for India, Can use these sources. APIs 1. Github: amodm/api-covid19-in (India) Repo: https://github.com/amodm/api-covid19-in APIs available: Statewise Data StateWise StateWise History Medica Hospital Stats Bed Stats Contacts HelpLines Contacts Patient Tracing History Sources The source is both types of official & Unofficial.

Golang basics & Handling 100k hourly webhooks with golang @MimePost

What this talk about? I am working on Golang for the last 1 year from the published date. I have shared some basics of Golang. Also, shared What are the pain points developers face when they migrate from any other language (Especially from web language like PHP) to Golang? I have explained the Webhook architecture of MimePost And how we sending 100k Request hourly( Though Benchmark proves we can scale up to 500k).

[SOLVED] Golang fatal error: concurrent map writes

The Problem: Suddenly got below errors which killed my daemon: fatal error: concurrent map writes goroutine 646 [running]: runtime.throw(0x75fd38, 0x15) /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc000315e60 sp=0xc000315e30 pc=0x42ecf2 runtime.mapdelete_fast64(0x6f0800, 0xc00008ad50, 0x2b3e) goroutine 1 [sleep]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:310 time.Sleep(0x12a05f200) /usr/local/go/src/runtime/time.go:105 +0x157 webhook/worker.Manager() goroutine 6 [IO wait]: internal/poll.runtime_pollWait(0x7fc308de6f08, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:184 +0x55 internal/poll.(*pollDesc).wait(0xc000110018, 0x72, 0x0, 0x0, 0x75b00b) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000110000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1f8 net.(*netFD).accept(0xc000110000, 0xc000050d50, 0xc000046700, 0x7fc308e426d0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.

How to scale with massive update queries in Elasticsearch?

Introduction What this talk is all about? We recently moved from MySQL to Elasticsearch where we got a direct 10x - 15x boost in our performance. We came up with unique use cases of heavy updates in Elasticsearch. That been challenging but yes currently Our Elaticsearch handling 200 million requests per day very efficiently. Our WRITE consist of the partial update, update with script conditions and of course simple indexing.

Elasticsearch Exceptions & Challenges

Below are some challenges & exceptions faced while setting up Elasticsearch. I just shared my experience and learning. Please correct me, If you guys feel somewhere i got wrong OR You can contribute if you have any experiences . Will keep update this gist.

How to reset 1-Click Installed WordPress on DigitalOcean?

The Requirement Need to install fresh wordpress with same version on wordpress droplet of digitalocean. The Problem My setup (wordpress droplet) was suddenly stop working. I started debugging. Debug Checked apache2 and mysql service: service mysql status service apache2 status Both services was active. Then I checked the apache2 processes with below command: $ ps -ef | grep apache2 | wc -l 151 Lots of apache child process has been forked.

[Part 3] How to write custom prometheus exporter?

Introduction In PART-1 and PART-2, We have seen how prometheus works and how to setup Prometheus and exporters. We have readymade exporters available on the internet. But sometime there is situation where you need to store your own custom metrics on prometheus. In such case you have to write your own exporters which will exporters the data into Prometheus. There is two way to exporting the data on prometheus: 1. Exporting to a Pushgateway Here we metrics are getting the push to prometheus server.

[Part 2] How to setup alertmanager and send alerts ?

Introduction In PART - 1, We have successfully setup Prometheus and exporters. In this part, we are going to setup alertmanager and will send our first alert. Alertmanager is software that is maintained by the prometheus and it is written in Go. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts.

[Part 1] How To Setup Prometheus And Exporters For Alerts And Monitoring?

As a developer, many times you would have worried, whether your services are up and running or not. Not only that, sometimes as an infrastructure guy you might be also worried about your server’s health too. What is the current RAM or disk utilization? or whether they are going to be fully occupied which in turn can completely bring the system down. These are just the basics and in fact there are tons of more such things which need to be monitored and fixed in everyday’s life.

What should be the value of max_gram and min_gram in Elasticsearch?

I was working on elasticsearch and the requirement was to implement like query “%text%” ( like mysql %like% ). We could use wildcard, regex or query string but those are slow. Hence i took decision to use ngram token filter for like query. It was quickly implemented on local and works exactly i want. The problem To know the actual behavior, I implemented the same on staging server. I found some problem while we start indexing on staging.

About me

Having 4+ Year experience in Software & Email Industry. I am Co-Founder at MimePost. Started journey with Software Engineer, Where i got opportunity to working with various stacks. I am from Mumbai, India. This is my new spot to share my thoughts and findings about programming and interesting tidbits of technology. Thanks. Ashish Tiwari