Tag: ELK

How to do a basic install of an ELK stack on Ubuntu for log stashing?

Posted on by

Install and configure elasticsearch

Use a machine with at least 4GiB of RAM. This guide will install Elasticsearch, Logstash and Kibana all on the same machine, so this is suited only for a small-scale setup.

# apt-get install apt-transport-https software-properties-common wget
# add-apt-repository ppa:webupd8team/java
# apt-get update
# apt-get install oracle-java8-installer
# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
# apt-get update
# apt-get install elasticsearch
# systemctl enable elasticsearch
# systemctl start elasticsearch

Then edit `/etc/elasticsearch/elasticsearch.yml` and do the following:

  • Set `node.name` to something descriptive
  • Set `cluster.name` to something unique to avoid issues with auto-discovery
  • Set `network.host` to “localhost”
  • Change `path.data` and `path.logs` if necessary

Edit `/etc/elasticsearch/jvm.options` and set the heap size according to your machine RAM, for example (to set it to 2GiB):

-Xms2g
-Xmx2g

A setting of 1GiB is probably the minimum you should use.

Make sure you disable swapping.

Install and configure Kibana

Run this: `$ sudo apt-get install kibana`. Then edit `/etc/kibana/kibana.yml` and set `server.host` to “localhost”.

Then: `# systemctl enable kibana && systemctl start kibana`

Create a DNS entry for your kibana host and an SSL certificate.

Install nginx as reverse proxy with authentication:

# apt-get install nginx
# rm /etc/nginx/sites-enabled/default
# echo "admin:$(openssl passwd -apr1 PASSWORD)" | tee -a /etc/nginx/kibana.htpasswd

Then edit a nginx site file, eg: `/etc/nginx/sites-available/kibana` and make it look like this:

server {
    listen 80 default_server;
    server_name _;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 default_server ssl http2;
    server_name _;

    ssl_certificate /etc/letsencrypt/live/YOURDOMAIN/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/YOURDOMAIN/privkey.pem;

    auth_basic "My Kibana";
    auth_basic_user_file /etc/nginx/kibana.htpasswd;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Activate the site like so:

# ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana
# nginx -t
# systemctl reload nginx

Install and configure logstash

Just run: `$ sudo apt-get install logstash`

Edit `/etc/logstash/jvm.options` and set the heap size according to your machine RAM, for example (to set it to 2GiB):

-Xms2g
-Xmx2g

A setting of 1GiB is probably the minimum you should use.

Add a file `/etc/logstash/conf.d/GIVE_ME_A_NAME.conf` and edit it so it looks like this:

input {
    beats {
        port => "5044"
    }
}

filter {
    if [fields][log_type] == "apache-access" {
        grok {
            match => { "message" => "%{IPORHOST:vhost}:%{NUMBER:vhost_port} %{COMBINEDAPACHELOG}" }
        }
        geoip {
            source => "clientip"
        }
    } else if [fields][log_type] == "apache-error" {
       grok {
           match => { "message" => "%{IPORHOST:vhost} \[%{TIMESTAMP_ISO8601:timestamp}\] \[%{DATA:module}:%{LOGLEVEL}\] \[pid: %{POSINT:pid}:tid %{DATA:tid}\] \[OS error: %{DATA:oserror}\] \[client %{DATA:clientip}\] %{GREEDYDATA:error_message}" }
       }
       geoip {
           source => "clientip"
       }
    }
}

output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "%{[fields][log_type]}-%{+YYYYMMdd}"
    }
}

Then run: `# systemctl restart logstash`

NB: The grok patterns assume the following log formats for apache:

ErrorLogFormat “%-v [%{cu}t] [%-m:%l] [pid: %-P:tid %-T] [OS error: %-E] [client %-a] %M” LogFormat “%v:%p %h %l %u %t \”%r\” %>s %O \”%{Referer}i\” \”%{User-Agent}i\”” vhost_combined

NB: Use this to test your grok patterns.

Install and configure filebeat

Filebeat goes on the machine where your service is running.

To install filebeat:

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
# apt-get install apt-transport-https
# echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | 
tee -a /etc/apt/sources.list.d/elastic-6.x.list
# apt-get update
# apt-get install filebeat

Then edit `/etc/filebeat/filebeat.yml` so it looks like this:

filebeat.inputs:

- type: log
  paths:
   - /var/log/apache2/access.log
  fields:
   log_type: apache-access

- type: log
  paths:
   - /var/log/apache2/error.log
  fields:
   log_type: apache-error

output.logstash:
 hosts: ["ELK_HOSTNAME_OR_IP:5044"]

#output.console:
# pretty: true

Then run `# systemctl restart filebeat`