Now let us talk about Elastic Stack and understand how it can solve the problem here. According to Elastic- Elastic Stack is a great tool to centralize logs from multiple sources, identify correlations, and perform the deep-data analysis. Elasticsearch is a search-and-analytics engine based on Apache Lucene that allows users to search and analyze large amounts of data in near real-time. Logstash can ingest and forward logs from anywhere to anywhere. Kibana is a dash-boarding tool with a user interface that allows us to query, visualize, and explore Elasticsearch data easily. I am not going to explain the installation process here but in the next article, I will try to cover them separately, and there I will try to explain the individual software starting from installation to implementation.

If you want to know the basics of Logstash then please refer to the "Introduction to Logstash" blog where I have explained the basics of Logstash.

The next step after installation is to set up a log pipeline into Elasticsearch for indexing and analysis using Kibana. There are various ways of forwarding data into Elasticsearch, but I am going to use Logstash. Logstash configuration files are written in JSON format under /etc/logstash/conf.d. The configuration consists of three sections: input, filter, and output. I am going to create a demo configuration file 'logs-apache.conf' for apache logs starting from the input section, the filter section is used to modify the input before sending it through the output section. For now, I am skipping this part to make it simple to understand the topic. Next is the output section:

 We have created the Logstash configuration file with the input and output sections. We need to start Logstash with the new configuration. I am using Ubuntu 17.04 here so run this command as per your Logstash setup in Operating System:

bin/logstash --path.settings /usr/share -f /etc/logstash/conf.d/logs-apache.conf

We can check the log data in Elasticsearch by accessing the created index through Logstash:

http://localhost:9200/logs_apache/_search

As we have the Apache logs in Elasticsearch, the next thing is to display it in Kibana. I will show this process through a series of images.

The next screen shows the index with fields and their data types. (I will explain the data types and other details on Elasticsearch in my next article.)

The discover tab of Kibana shows Apache log data with search capabilities.

Now try to search the keywords (as in the below image):

Now we have put everything in place let's play by accessing some local websites to push the Apache access logs. Logstash is already tailing this log, so these messages will be indexed into Elasticsearch and displayed in Kibana. Now play with the data in Kibana by analyzing it.

The Elastic Stack can be used with Beat to fetch files, network and system information, etc. It can be connected to an existing application to monitor the application performance as well as to create a great dashboard to monitor key performance indicators. We can also use it as a stand-alone system by pushing the data from any RDBMS or file-based data source. We can not only show or search data but can also perform analysis on top of that. In case of any confusion please do comment.

Other Blogs on Elastic Stack:
Introduction to Elasticsearch
Elasticsearch Installation and Configuration on Ubuntu 14.04
Log analysis with Elastic stack 
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Wildcard and Boolean Search in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch
Configure Logstash to push MongoDB data into Elasticsearch
Load CSV Data into Elasticsearch
Metrics Aggregation in Elasticsearch
Bucket Aggregation in Elasticsearch
How to create Elasticsearch Cluster