Introduction to Logstash
Logstash is a data collection pipeline of Elastic Stack which is a utility to fetch data from different sources and send it to multiple sources. We can transform the data before sending it to the output. We can explain Logstash in the following points:
- Data collection pipeline tool.
- Gathers data from different sources.
- A tool to collect, process, and forward events and log messages.
- Data collection, enrichment, and transformation pipeline.
- Configurable input and output plugins
e.g. log file, event log, socket, Syslog, Redis, CSV, DB
So as mentioned in the above points, Logstash can be configured to gather data, transform it as per the requirement like convert unstructured data into structured data, and send it to multiple sources like Elasticsearch or any file, etc.
Use Cases:
We can use Logstash for the following purposes:
- Analyze a large variety of structured/ unstructured data and events.
- Connect with various types of input sources and platforms.
- Transform the data and store it in different storage for analytics.
Logstash Pipeline:
A Logstash pipeline has three elements, input, filter, and output. Input and Output plugins are mandatory while the filter is an optional element. The input plugins consume data from a source, the filter plugins modify the data as you specify, and the output plugins write the data to a destination, please refer to the following diagram:
Input Plugins:
Input plugins can be configured for capturing data from different sources. Examples of Logstash input plugins are as follows:
- File
- Stdin
- Beats
- MongoDB
- Elasticsearch
- JDBC
- Redis
- Kafka
- Http
Filter Plugins:
Logstash filter plugins can be used to transform the input data if we want to remove a certain field or want to convert the unstructured data into structured data. Examples of Logstash filter plugins are as follows:
- GeoIP
- CSV
- JDBC
- Date
- Grok
- JSON
- XML
Output Plugins:
Logstash output plugins are there to send data from Logstash to single or multiple sources. Like if we want to send RDBMS data to Elasticsearch then we can configure the Logstash output plugin. Examples of Logstash output plugins are as follows:
- File
- Stdout
- MongoDB
- Elasticsearch
- Redis
- Kafka
- Http
- Nagios
- S3
Structure of Logstash Configuration:
A Logstash configuration file contains input, output, and/or filter plugin.
input {
………
}
filter {
……….
}
output {
……...
}
Test Logstash Pipeline:
To test the Logstash pipeline we can run the following command after installing the Logstash.
bin/logstash -e 'input { stdin { } } output { stdout {} }'
Execute the Logstash Pipeline:
To execute the Logstash configuration file, run the following command:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/crimes.conf
Configuring Logstash to get data:
Logstash is a data pipeline we can use to configure input to take data from multiple types of data sources, such as files, databases, CSV, or Kafka, and after taking the input, we can configure the output to send data on different sources, such as files, databases, Kafka, or Elasticsearch. Another important feature of Logstash filters, using which we can transform the input data before sending it to the output.
Fetch Apache log data:
For example, if we want to read the apache log data then it can be configured in the following way:
input
{
file
{
path => "/var/log/apache.log"
type => "apache-access"
start_position => "beginning"
}
}
filter
{
grok
{
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
}
output
{
elasticsearch
{
hosts => ["localhost:9200"]
}
}
This way we can configure Logstash to capture data from different sources and can send it to Elasticsearch or any other source destination.
In case of any query please leave a comment so that I can revert you back. After this introduction, you can use Logstash to push MySQL data into Elasticsearch, load CSV Data into Elasticsearch, send MongoDB data into Elasticsearch, or for Log analysis with Elastic stack.
Other Blogs on Elastic Stack:
Load CSV Data into Elasticsearch
Configure Logstash to push MongoDB data into Elasticsearch
How to create Elasticsearch Cluster
Bucket Aggregation in Elasticsearch
Metrics Aggregation in Elasticsearch
Configure Logstash to push MySQL data into Elasticsearch
Wildcard and Boolean Search in Elasticsearch
Elasticsearch Rest API
Basics of Data Search in Elasticsearch
Elasticsearch Rest API
Log analysis with Elastic stack
Elasticsearch Installation and Configuration on Ubuntu 14.04
Introduction to Elasticsearch
If you found this article interesting, then you can explore “Mastering Kibana 6.0”, “Kibana 7 Quick Start Guide”, “Learning Kibana 7”, and “Elasticsearch 7 Quick Start Guide”
books to get more insight about Elastic Stack, how to perform data
analysis, and how you can create dashboards for key performance
indicators using Kibana.
Comments (0)
Leave a comment
Related Blogs
Importing MongoDB data into Elasticsearch
Mar 9, 2019, 8:20:38 AM | Anurag Srivastava
Importing MySQL data into Elasticsearch
Feb 9, 2019, 12:06:18 PM | Anurag Srivastava
Snapshot and Restore Elasticsearch Indices
Sep 16, 2019, 5:55:06 AM | Anurag Srivastava
Log analysis with Elastic stack
Jan 31, 2018, 6:11:29 AM | Anurag Srivastava
Creating Elasticsearch Cluster
Apr 6, 2019, 8:41:41 PM | Anurag Srivastava
Introduction to Elasticsearch
Apr 14, 2018, 1:18:05 PM | Anurag Srivastava
Top Blogs
Configure SonarQube Scanner with Jenkins
Jun 21, 2018, 4:58:11 AM | Anurag Srivastava
Execute Commands on Remote Machines using sshpass
Jul 16, 2018, 5:00:02 PM | Anurag Srivastava
Importing MongoDB data into Elasticsearch
Mar 9, 2019, 8:20:38 AM | Anurag Srivastava
Importing MySQL data into Elasticsearch
Feb 9, 2019, 12:06:18 PM | Anurag Srivastava
Configure Jenkins for Automated Code Deployment
Jun 13, 2018, 3:44:01 PM | Anurag Srivastava
Deploying Angular code using Python script
Jun 26, 2018, 4:50:18 PM | Anurag Srivastava