Monitoring is a very important aspect in the present Information Technology age. They keep us updated with the status of a running application along with other factors that can impact the performance of the application. We are dealing with so many different components and it is impossible to predict how they are going to impact the performance of an application. Monitoring helps us to keep an eye on the performance of these components and get notified before it actually impacts the performance of a running application. This way we can keep a check on any issue which can impact the business continuity.

Now the question arises that what to monitor and whether it is sufficient to get the complete idea of a system. Sometimes it is very difficult to know the root cause of any issue as it can be because of the code, database query, server issue, or because of any external factor. So it is very important to get data from all sources into the monitoring setup. Please refer to the below image to understand how full-stack monitoring can be configured:

Now take an example of a microservice-based application, here we can think of the following components which can impact the performance:

First of all, we want to know if all API's are running and they are fully functional.
In this case, we can use any tool (like Elastic Heartbeat) which can keep on checking the APIs and can provide that data to a central server where we can monitor them. This way we can have the current status of all running API's and we can also know when service was down and for how long.

Database server monitoring to keep a check on the database performance
Using database performance monitoring we can get details about user connections, DB transactions, requests per second, command rates, running threads, etc. These are very important information using which we can have an idea of how the database is performing under various scenarios. We can use Metricbeat with the database module to fetch specific database metrics.

Monitoring of application, server, and database logs to know if they are giving any error which can impact.
It is very important to read data from application, server, and database logs as they can provide us the details if there is any error. This has to be configured on each server where the application or databases are running. We need to collect these data to a central server on which we can configure the monitoring.  

Application performance monitoring to get details on database queries, response time for incoming requests, external HTTP requests, errors and exceptions, etc.
There are many tools like Elastic APM which can be configured with the application in order to get the internal application metrics. Using these tools we can get the details of database queries, response time for incoming requests, external HTTP requests, errors and exceptions, etc.

If you want to learn more about Elastic APM then please refer to the "Introduction to Elastic APM" blog. You may also like the blog on how to do "Log analysis with Elastic stack".

The infrastructure monitoring to check if it is working fine without any issue of processes, CPU, RAM, or disk space.
Infrastructure is very vital for any running application and the performance of any application depends on the infrastructure. If there is any bottleneck in the infrastructure, the application would create issues so it is very important to monitor the infrastructure. There are many tools like Metricbeats using which we can monitor the application infrastructure. They can be installed on the servers from where they collect the metrics and send them to a central server for monitoring.

Network monitoring to check the connectivity, error or packet loss, network latency, and bandwidth utilization, etc.
There are different network monitoring tools like t-shark, Packetbeat, etc using which we can capture the network packet details. Again these tools can be configured on individual servers from where they collect the metrics and send it to a central server for monitoring.

To monitor any external dependency on the application.
We also need to set up monitoring if there is any dependency on the external system. These systems can be any external APIs, or anything else, etc, we can plan their monitoring as well. Like in case of external APIs we can configure Heartbeat to check their availability etc.

There can be other components also which we can monitor but mainly these are the main components that we should monitor to get the complete monitoring data for any running application. These metrics can be pushed into Elasticsearch and using Kibana we can configure the dashboard. Once the full-stack monitoring system is configured, we can get all the details on a single dashboard using which we get all the details. Alerts can also be configured on different thresholds so that we can react in a proactive way on any issues.

This blog is on introduction to monitoring and it covers how important is the monitoring.  It helps us to figure out any issue in the system which is the most tedious task in debugging. I will provide details on the different monitoring components in future blogs. In case of any doubt please leave your comments. You can also follow me on Twitter: https://twitter.com/anu4udilse

Other Blogs on Elastic Stack:

How to create Elasticsearch Cluster
Bucket Aggregation in Elasticsearch
Metrics Aggregation in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch 
Wildcard and Boolean Search in Elasticsearch
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Log analysis with Elastic stack 
Elasticsearch Installation and Configuration on Ubuntu 14.04
Introduction to Elasticsearch
Configure Logstash to push MongoDB data into Elasticsearch
Load CSV Data into Elasticsearch

If you found this article interesting, then you can explore “Mastering Kibana 6.0”, “Kibana 7 Quick Start Guide”, “Learning Kibana 7”, and “Elasticsearch 7 Quick Start Guide” books to get more insight about Elastic Stack, how to perform data analysis, and how you can create dashboards for key performance indicators using Kibana.