How to properly use certbot with Nginx on Ubuntu?

Install and configure Nginx:

$ sudo apt update
$ sudo apt install nginx
$ sudo rm /etc/nginx/sites-enabled/default
$ sudo vi /etc/nginx/sites-available/MYSITE  # See below
$ sudo ln -s /etc/nginx/sites-available/MYSITE /etc/nginx/sites-enabled
$ sudo systemctl restart nginx

The /etc/nginx/sites-available/MYSITE config file should look like so:

server {
  listen 80;
  server_name MYSITE.COM WWW.MYSITE.COM;
}

server {
  listen 443 ssl;
  server_name MYSITE.COM WWW.MYSITE.COM;
  server_tokens off;

  # If you use nginx as reverse-proxy
  location / {
    proxy_pass http://localhost:8080;
    proxy_set_header Host $host;
  }
}

Install certbot:

$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt update
$ sudo apt install certbot python-certbot-nginx

Create the certificate:

$ sudo certbot --nginx --agree-tos --redirect --uir --hsts --staple-ocsp --email ME@MYSITE.COM -d MYSITE.COM -d WWW.MYSITE.COM

You could potentially use the --must-staple option to enable the OCSP “must staple” extension, but it doesn’t work well with Firefox (to fix in your Firefox browser: go to about:config, and set security.ssl.enable_ocsp_must_staple to false).

The arguments are:

  • --nginx: Use Nginx for authentication and install certificate for Nginx
  • --agree-tos: Agree with Let’s Encrypt terms of service
  • --redirect: Add Nginx config to redirect HTTP to HTTPS
  • --uir: Add Content-Security-Policy: upgrade-insecure-requests to HTTP responses
  • --hsts: Add Strict-Transport-Security header to HTTP responses
  • --staple-ocsp: Enable OSCP stapling (allow browser to skip verification of whether the SSL certificate has been revoked or not)
  • --email: Email address to use to register with Let’s Encrypt, and potentially for recovery

How to install an old version of PHP using geerlingguy.php on Ubuntu target

Your playbook should look like so:

---
- hosts: XYZ
  become: yes

  pre_tasks:
    - name: Add repository for old PHP
      apt_repository:
        repo: ppa:ondrej/php
        update_cache: yes
        state: present

  roles:
    - role: geerlingguy.apache
      ...

    - role: geerlingguy.php
      php_webserver_daemon: apache2
      php_default_version_debian: "7.1"
      php_version: "7.1"
      php_packages:
        - php7.1
        - php7.1-cli
        - etc...

How to install and kick off minikube on Ubuntu?

Install kubectl:

$ sudo apt-get update
$ sudo apt-get install apt-transport-https
$ curl -sSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-get install kubectl

Install virtualbox:

$ sudo apt-get update
$ sudo apt-get install virtualbox

Install minikube (replace v0.30.0 with the latest version available for you):

$ wget https://github.com/kubernetes/minikube/releases/download/v0.30.0/minikube-linux-amd64
$ mv minikube-linux-amd64 minikube
$ chmod +x minikube
$ sudo cp minikube /usr/local/bin/
$ rm minikube

Create the VM and start the cluster (this will create the file ~/.kube/config):

$ minikube start  # use `minikube start --help` for more options
$ kubectl cluster-info

To get the IP address of the cluster:

$ minikube ip
$ sudo vi /etc/hosts  # if you want to assign names to this IP address

To show the Kubernetes dashboard:

$ minikube dashboard

To stop the cluster (shuts down the VM and preserve all states):


$ minikube stop

To restart the VM and cluster:

$ minikube start

To delete the cluster (and the VM):

$ minikube delete
$ rm ~/.kube/config

How to do a basic install of Prometheus on Ubuntu?

Machine requirement

I will assume your machine is Ubuntu >= 16.04.

Please make sure only ports 80 and 443 are open.

I assume there are two DNS entries to your machine, such as prometheus.blablabla.com and grafana.blablabla.com (replace as appropriate throughout the tutorial).

Install Prometheus

Follow the steps (you can replace version 2.4.3 with whatever latest version is available for you):

$ sudo useradd --no-create-home --shell /bin/false prometheus
$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xf prometheus-2.4.3.linux-amd64.tar.gz
$ cd prometheus-2.4.3.linux-amd64
$ sudo cp prometheus promtool /usr/local/bin
$ sudo mkdir /etc/prometheus /var/lib/prometheus
$ sudo cp -r prometheus.yml consoles console_libraries /etc/prometheus
$ sudo chown prometheus.prometheus /var/lib/prometheus

Create the file /etc/systemd/system/prometheus.service and put the following content in it:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries
Restart=always

[Install]
WantedBy=multi-user.target

Then run:

$ sudo systemctl daemon-reload
$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus

Install Grafana

To install Grafana:

$ echo "deb https://packagecloud.io/grafana/stable/debian/ stretch main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
$ curl -sSL https://packagecloud.io/gpg.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install grafana
$ sudo systemctl daemon-reload
$ sudo systemctl enable grafana-server
$ sudo systemctl start grafana-server

Do not try to configure Grafana yet! We need to do all the SSL stuff first.

Reverse-proxy and SSL certificates

Install certbot

Follow the steps:

$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot python-certbot-nginx

Reverse-proxy for Prometheus

Prometheus web interface has no authentication mechanism, so we will add one with nginx.

To install nginx as a reverse-proxy for SSL termination and authentication for Prometheus (replace USERNAME and PASSWORD with a username and password of your choice):

$ sudo apt-get install nginx
$ sudo rm /etc/nginx/sites-enabled/default
$ echo "USERNAME:$(openssl passwd -apr1 PASSWORD)" | tee -a /etc/nginx/prometheus.htpasswd

Then edit the file /etc/nginx/sites-available/prometheus to look like this (replace prometheus.blablabla.com with your real domain name; NB: more SSL config will be added by certbot):

server {
  listen 80;
  server_name prometheus.blablabla.com;
}

server {
  listen 443 ssl;
  server_name prometheus.blablabla.com;

  location / {
    proxy_pass http://localhost:9090;
    proxy_set_header Host $host;
    auth_basic "My Prometheus";
    auth_basic_user_file /etc/nginx/prometheus.htpasswd;
  }
}

Reload nginx:

$ sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Create an SSL certificate for prometheus.blablabla.com:

$ sudo certbot --nginx -d prometheus.blablabla.com

Certbot will start an interactive session. Answer as follows:

  • When asked for your email address, enter it
  • Agree to the terms of service
  • Agree or not to the marketing, that has no incidence on your SSL certificate
  • When asked to redirect all HTTP traffic to HTTPS, choose to redirect

Reverse-proxy for Grafana

Edit the file /etc/nginx/sites-available/grafana to look like this (NB: more SSL config will be added by certbot):

server {
  listen 80;
  server_name grafana.blablabla.com;
}

server {
  listen 443 ssl;
  server_name grafana.blablabla.com;

  location / {
    proxy_pass http://localhost:3000;
    proxy_set_header Host $host;
  }
}

Reload nginx:

$ sudo ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Create an SSL certificate for grafana.blablabla.com:

$ sudo certbot --nginx -d grafana.blablabla.com

Answer the interactive prompts in the same manner as previously.

Configure Grafana

Open your favourite browser to http://grafana.blablabla.com. The Grafana login page should come up. Enter admin for both the username and password; the next page should ask you for a more secure password.

Then click on the 2nd icon “Add data source” and enter the following details:

  • Name: Whatever you like
  • Type: Prometheus
  • URL: http://localhost:9090
  • Leave the rest untouched
  • Click “Save & Test”
  • You should see a message at the bottom of the page “Data source is working”

Install and configure Alertmanager

Follow the steps (you can replace version 0.15.2 with whatever latest version is available for you):

$ sudo useradd --no-create-home --shell /bin/false alertmanager
$ wget //github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz
$ tar xf alertmanager-0.15.2.linux-amd64.tar.gz
$ cd alertmanager-0.15.2.linux-amd64
$ sudo cp alertmanager amtool /usr/local/bin
$ sudo mkdir -p /etc/alertmanager/templates /var/lib/alertmanager
$ sudo chown alertmanager.alertmanager /var/lib/alertmanager

Create the file /etc/alertmanager/alertmanager.yml so it looks like this (substitute the correct values for your setup):

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@prometheus.blablabla.com'
  smtp_auth_username: ''
  smtp_auth_password: ''
  smtp_require_tls: false

templates:
  - /etc/alertmanager/templates/*.tmpl

route:
  repeat_interval: 1h
  receiver: MYTEAM

receivers:
  - name: MYTEAM
    email_configs:
    - to: MYTEAM@MYORG.com
    slack_configs:
    - api_url: https://hooks.slack.com/services/XXXXXX/XXXXXX/XXXXXX
      send_resolved: true

Create the file /etc/systemd/system/altermanager.service and put the following content in it:

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
        --config.file /etc/alertmanager/alertmanager.yml \
        --storage.path /var/lib/alertmanager
Restart=always

[Install]
WantedBy=multi-user.target

Then run:

$ sudo systemctl daemon-reload
$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus

You need to tell prometheus that an alertmanager service is available. To do so, edit the /etc/prometheus/prometheus.yml file and add the following at the end:

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - localhost:9093

Then:

$ sudo systemctl restart prometheus

How to install Docker on Ubuntu

Follow the steps:

$ sudo apt-get install curl software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce
$ sudo usermod -aG docker $(id -un)  # optional, allows you to run docker without `sudo`

How to enable and analyse the slow query log on MySQL?

Set the following variables:

slow_query_log = 1 (means true)
long_query_time = 0 (means 0 seconds)
log_queries_not_using_indexes = 1 (means true)
log_output = FILE (default is TABLE)

NB: Setting `long_query_time` to 0 will instruct MySQL to log all queries to the slow query log. Some queries might not be that slow, but if repeated often enough will have a significant impact on the performance of your MySQL database. Setting `long_query_time` to 0 will allow you to catch that kind of scenario as well.

Let MySQL run for a while, then download pt-query-digest, and run it like so:

$ pt-query-digest slow-query.log > slow-query.log.out

Look at the `slow-query.log.out` file to find out which queries are slowing down your MySQL database.

How to do a basic install of an ELK stack on Ubuntu for log stashing?

Install and configure elasticsearch

Use a machine with at least 4GiB of RAM. This guide will install Elasticsearch, Logstash and Kibana all on the same machine, so this is suited only for a small-scale setup.

# apt-get install apt-transport-https software-properties-common wget
# add-apt-repository ppa:webupd8team/java
# apt-get update
# apt-get install oracle-java8-installer
# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
# apt-get update
# apt-get install elasticsearch
# systemctl enable elasticsearch
# systemctl start elasticsearch

Then edit `/etc/elasticsearch/elasticsearch.yml` and do the following:

  • Set `node.name` to something descriptive
  • Set `cluster.name` to something unique to avoid issues with auto-discovery
  • Set `network.host` to “localhost”
  • Change `path.data` and `path.logs` if necessary

Edit `/etc/elasticsearch/jvm.options` and set the heap size according to your machine RAM, for example (to set it to 2GiB):

-Xms2g
-Xmx2g

A setting of 1GiB is probably the minimum you should use.

Make sure you disable swapping.

Install and configure Kibana

Run this: `$ sudo apt-get install kibana`. Then edit `/etc/kibana/kibana.yml` and set `server.host` to “localhost”.

Then: `# systemctl enable kibana && systemctl start kibana`

Create a DNS entry for your kibana host and an SSL certificate.

Install nginx as reverse proxy with authentication:

# apt-get install nginx
# rm /etc/nginx/sites-enabled/default
# echo "admin:$(openssl passwd -apr1 PASSWORD)" | tee -a /etc/nginx/kibana.htpasswd

Then edit a nginx site file, eg: `/etc/nginx/sites-available/kibana` and make it look like this:

server {
    listen 80 default_server;
    server_name _;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 default_server ssl http2;
    server_name _;

    ssl_certificate /etc/letsencrypt/live/YOURDOMAIN/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/YOURDOMAIN/privkey.pem;

    auth_basic "My Kibana";
    auth_basic_user_file /etc/nginx/kibana.htpasswd;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Activate the site like so:

# ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana
# nginx -t
# systemctl reload nginx

Install and configure logstash

Just run: `$ sudo apt-get install logstash`

Edit `/etc/logstash/jvm.options` and set the heap size according to your machine RAM, for example (to set it to 2GiB):

-Xms2g
-Xmx2g

A setting of 1GiB is probably the minimum you should use.

Add a file `/etc/logstash/conf.d/GIVE_ME_A_NAME.conf` and edit it so it looks like this:

input {
    beats {
        port => "5044"
    }
}

filter {
    if [fields][log_type] == "apache-access" {
        grok {
            match => { "message" => "%{IPORHOST:vhost}:%{NUMBER:vhost_port} %{COMBINEDAPACHELOG}" }
        }
        geoip {
            source => "clientip"
        }
    } else if [fields][log_type] == "apache-error" {
       grok {
           match => { "message" => "%{IPORHOST:vhost} \[%{TIMESTAMP_ISO8601:timestamp}\] \[%{DATA:module}:%{LOGLEVEL}\] \[pid: %{POSINT:pid}:tid %{DATA:tid}\] \[OS error: %{DATA:oserror}\] \[client %{DATA:clientip}\] %{GREEDYDATA:error_message}" }
       }
       geoip {
           source => "clientip"
       }
    }
}

output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "%{[fields][log_type]}-%{+YYYYMMdd}"
    }
}

Then run: `# systemctl restart logstash`

NB: The grok patterns assume the following log formats for apache:

ErrorLogFormat “%-v [%{cu}t] [%-m:%l] [pid: %-P:tid %-T] [OS error: %-E] [client %-a] %M” LogFormat “%v:%p %h %l %u %t \”%r\” %>s %O \”%{Referer}i\” \”%{User-Agent}i\”” vhost_combined

NB: Use this to test your grok patterns.

Install and configure filebeat

Filebeat goes on the machine where your service is running.

To install filebeat:

# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
# apt-get install apt-transport-https
# echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | 
tee -a /etc/apt/sources.list.d/elastic-6.x.list
# apt-get update
# apt-get install filebeat

Then edit `/etc/filebeat/filebeat.yml` so it looks like this:

filebeat.inputs:

- type: log
  paths:
   - /var/log/apache2/access.log
  fields:
   log_type: apache-access

- type: log
  paths:
   - /var/log/apache2/error.log
  fields:
   log_type: apache-error

output.logstash:
 hosts: ["ELK_HOSTNAME_OR_IP:5044"]

#output.console:
# pretty: true

Then run `# systemctl restart filebeat`

How to install the dns-route53 plugin for certbot on Ubuntu?

There are no instruction on how to install the dns route53 plugin for certbot. Here is how to do it for Ubuntu.

To install certbot:


$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot

To install the dns-route53 plugin:

# apt-get install python3-pip
$ sudo pip3 install certbot-dns-route53

You can then create a new certificate with something like that:


$ sudo certbot certonly --dns-route53 -d YOUR-DOMAIN.com --deploy-hook 'systemctl reload apache2'