Install ELK Stack Mac(Step By Step)

Elasticsearch, Kibana and Logstash together makes ELK stack and in this tutorial, we will be installing and setting up ELK stack. I have Mac Sierra and we will be setting up this stack locally for development purpose.On the official elastic website, we already have the .zip files available and instructions available to set it up but we will be using Homebrew for the installation as there are some advantages of installing it via Homebrew.

What is ELK Stack

ELK stands for Elasticsearch, Logstash, and Kibana. ElasticSearch is a JSON based search and analytics engine based on Lucene. Logstash is a server?side data processing pipeline that ingests data from multiple sources simultaneously,transforms it, and then sends it to a "stash" like Elasticsearch.Similarly, Kibana is a visualization tool for the data in elasticsearch with the help of beautiful charts and graphs.

We can also use Apache Kafka for server-side data processing as it is also open-source and provides a unified, high-throughput,low-latency platform for handling real-time data feeds.

Pre-Setup Checks

Mac provides a beautiful package manager called Homebrew that provides easy installation of software on MacOS. It is always recommended to install any software with brew command to avoid permission related issues and it also makes installation and un-installation fairly simple.Check if it already installed first using following command.

brew doctor

In case it is not installed use following command to install it first.

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

You can use following command to update it if it is already installed.

brew update && brew upgrade

Next pre-setup check is the java installation.ELK requires Java 8 to be installed.Check if Java is installed on your machine.

java -version

If it is not installed, then first try installing it as per instructions given here.

Installing ElasticSearch

We can install elasticsearch with following commands.

brew install elasticsearch && brew info elasticsearch
brew services start elasticsearch

if you don't want/need a background service you can just run: elasticsearch

Now let us get into understanding the folder structure as we require this for Elasticsearch customization.Following are the directory structure created after this installation.Here, macuser is your mac user id.

Data:    /usr/local/var/lib/elasticsearch/elasticsearch_macuser/
Logs:    /usr/local/var/log/elasticsearch/elasticsearch_macuser.log
Plugins: /usr/local/var/elasticsearch/plugins/
Config:  /usr/local/etc/elasticsearch/

I think we are clear about the log file location and data location of our elasticsearch.Also, inside the plugins directory, we can install any plugins such as XPack etc. For a full list of available plugins, you can check this link.

We are mainly interested in Config directory as we amy require to customize some default settings as per our need.We have jvm.options, elasticsearch.yml and log4j2.properties inside /usr/local/etc/elasticsearch/.

jvm.options - It has all the default JVM related configurations such as initial size of total heap space as 1g, maximum size of total heap space as 1g.

elasticsearch.yml -This file has elasticsearch specific configurations such as default cluster name,Path to directory where to store the data,Path to log files, host and port.By default, elasticsearch runs on port 9200 and if you want to change it some other port, then you can change it from this file.

log4j2.properties

Here, we are not making any changes in the default configurations and hence elasti search should be accessible to us at localhost:9200 We have elasticsearch version of 6.2.2 and lucene version of 7.2.1

Installing Logstash

We can install Logstash with following commands.

brew install logstash
brew services start logstash

if you don't want/need a background service you can just run: logstash.Following is the configuration files location..

Config : /usr/local/Cellar/logstash/6.2.2/libexec/config
Scripts : /usr/local/Cellar/logstash/6.2.2/libexec/bin

Similar to elasticsearch, we have all the files such as jvm.options, logstash.yml, pipelines.yml etc inside /usr/local/Cellar/logstash/6.2.2/libexec/config.

Now let us pull some data in logstash.Either we can pull data from database or any log file.Here, we will be pulling data from mysql database.

Importing Logstash Data From Database

Now, we will be importing data from mysql database in logstash and logstash will push these data to elasticsearch.I have mysql instance running on my local machine which has following schema details:

CREATE Database testdb;
CREATE TABLE Users
(
   users_id bigint PRIMARY KEY NOT NULL,
   attempts int,
   createdtimestamp timestamp,
   enabled int,
   modifiedtimestamp timestamp,
   username varchar(255),
   salary varchar(20)
)
;
CREATE UNIQUE INDEX PRIMARY ON Users(users_id)
;

Let us create our users.conf file at location /usr/local/Cellar/logstash/6.2.2/bin/users.conf

## Configuration details for Devglan test ##
input {
	jdbc {
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:mysql://localhost:3306/testdb"
        jdbc_user => "root"
        jdbc_password => "root"
        jdbc_driver_library => "/Users/macuser/Documents/work/soft/drivers/com.mysql.jdbc_5.1.5.jar"
        jdbc_driver_class => "Java::com.mysql.jdbc.Driver"
		tags => [ "devglan" ]
		statement => "SELECT * FROM USERS "
		#schedule => "*/3 * * * *"
		#schedule => "0 12-13 * * *"
	   }
}
output {
	if "devglan" in [tags]{
	stdout { codec => json_lines }
    elasticsearch {
		action => "index"
        index => "devglan"
		document_type => "devglan"
		document_id => "%{id}"
		hosts => "127.0.0.1"
	}}
}

Now, we can start our logstash with following command.

logstash -f users.conf

Installing Kibana

brew install kibana

Once, Kibana is installed, we have config file - kibana.yml located inside /usr/local/Cellar/kibana/6.2.2/config. This file has configurations such as server port, server host, configurations related to elasticsearch and many more.

By default, Kibana runs on port 5601 and it assumes elasticsearch running on port 9200. These are the default configurtions and since we have not made any chnages in the default configurations of elasticsearch and logstash, we are good to go.Now, let us start kibana with following command.

brew services start kibana

Now you can hit http://localhost:5601/ to see kibana dashboard up and running.

Now, under management section you can see one index pattern as devglan that we created in our user.conf file.

Conclusion

In this tutorial, we learned about setting up ELK stack on mac and configuring logstash to fetch data from database and creating index in Kibana. There are many things to explore in ELK stack which we will be exploring in next article.