Create Elastic Search cluster + Kibana + Logstash using Azure

in Blog

Elastic Search in Azure

  1. Search for Elastic Search in Azure Marketplace. You will find this template: My helpful screenshot

  2. Add user name and credentials. My helpful screenshot

  3. Creare your Vnet and subnet. If you want to add some web api after, you will need to add your web app to the same Vnet. My helpful screenshot

  4. Create 1 client node My helpful screenshot

  5. By the way here you don’t need to add user name. For example, for Kibana you will use “elastic” as a user name to login as a superuser. You need to enter only your passwords.

My helpful screenshot

  1. You want to install Kibana for visualization. You will receive an error if you enable jump box. .

My helpful screenshot

  1. Click ‘ok’ for everything else after that step and wait when your resources will be deployed. When Elastic Search successfully deployed on Azure you will see something like this.

My helpful screenshot To see deployment status click on the bell – upper right corner. My helpful screenshot

  1. Search in your Resource Group for Kibana virtual machine. Go to an VM overview. Copy DNS or public IP address if you prefer, after - check networking tab. My helpful screenshot You can see here what ports are open. Add Kibana dns/ip + port - 5601 in the browser and sign in to Kibana. My helpful screenshot During provisioning elastic search cluster you added 2 passwords. To login and to see Kibana dashboard you can use you can use username ‘Elastic’ and your password to login as a superuser, or username ‘kibana’ and your Kibana password. You Kibana dashboard must look similar to this: My helpful screenshot

  2. By default in Azure Elastic Search Marketplace offer/ ARM template you can’t find Logstash. Why do we need Logstash? By using LogStash we can create data pipelines. Lets create Ubuntu VM for Logstash. Go to Azure portal. Click on +. Choose ‘Compute’ -> Ubuntu Server.
    My helpful screenshot
  3. Add name and password for your VM. Choose the same Resource Group and region. Later we would add this VM to the same Vnet. My helpful screenshot
  4. Make sure that your VM is in the same Virtual Network and we have public ip address (better if it is static). Also choose http, https, ssh as public inbounds ports. You can delete them later. Click ok and click create button after. And wait for deployment. My helpful screenshot
  5. Install bash for Windows to ssh to you Logstash VM https://www.windowscentral.com/how-install-bash-shell-command-line-windows-10 Find your new Ubuntu VM that you’ve just created. My helpful screenshot Click on connect and ssh and copy command to you bash window, after enter your password. My helpful screenshot
  6. Run script below to create a new instalation file.
sudo nano ./install.sh

Paste those lines to your file

# installing java 8 (9, 10, ... not supported)
sudo apt install -y openjdk-8-jdk
# installing logstash
# Download and install the Public Signing Key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# You may need to install the apt-transport-https package on Debian before proceeding:
sudo apt-get install -y apt-transport-https
# Save the repository definition to /etc/apt/sources.list.d/elastic-6.x.list:
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
# installing logstash package
sudo apt-get update && sudo apt-get install -y logstash

# Running Logstash as a service
sudo systemctl start logstash.service

Press ‘CTRL+X’ to exit and ‘Y’ It will save your changes automagically. Let’s run that file

sh ./install.sh

It will take a while to install all packages. More details you can find here: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

https://linuxconfig.org/how-to-install-java-on-ubuntu-18-04-bionic-beaver-linux

My helpful screenshot

  1. Let’s create a simple data pipeline. Check what is your current directory
    pwd

    You should be here

    /home/username

    Let’s change it

    cd /usr/share/logstash

    sudo bin/logstash -e ‘input {stdin {} } output {stdout { } }’ (if you forget to write sudo you may receive an error that logshash data must be a writable directory) Using this command you can write something in console and receive the same message to console. Maybe you will need to restart your logstash.servise like

    sudo systemctl stop logstash.service
sudo systemctl start logstash.service

You should see te message that ‘Pipeline started successfully’ After several seconds you will receive an output into the console with a timestamp.

Thanks

When something is important enough, you do it even if the odds are not in your favor.

© 2018. All rights reserved.