How to properly use certbot with Nginx on Ubuntu?

Posted on December 18, 2018 by fabricetriboix

Install and configure Nginx:

$ sudo apt update
$ sudo apt install nginx
$ sudo rm /etc/nginx/sites-enabled/default
$ sudo vi /etc/nginx/sites-available/MYSITE  # See below
$ sudo ln -s /etc/nginx/sites-available/MYSITE /etc/nginx/sites-enabled
$ sudo systemctl restart nginx

The /etc/nginx/sites-available/MYSITE config file should look like so:

server {
  listen 80;
  server_name MYSITE.COM WWW.MYSITE.COM;
}

server {
  listen 443 ssl;
  server_name MYSITE.COM WWW.MYSITE.COM;
  server_tokens off;

  # If you use nginx as reverse-proxy
  location / {
    proxy_pass http://localhost:8080;
    proxy_set_header Host $host;
  }
}

Install certbot:

$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt update
$ sudo apt install certbot python-certbot-nginx

Create the certificate:

$ sudo certbot --nginx --agree-tos --redirect --uir --hsts --staple-ocsp --email ME@MYSITE.COM -d MYSITE.COM -d WWW.MYSITE.COM

You could potentially use the --must-staple option to enable the OCSP “must staple” extension, but it doesn’t work well with Firefox (to fix in your Firefox browser: go to about:config, and set security.ssl.enable_ocsp_must_staple to false).

The arguments are:

--nginx: Use Nginx for authentication and install certificate for Nginx
--agree-tos: Agree with Let’s Encrypt terms of service
--redirect: Add Nginx config to redirect HTTP to HTTPS
--uir: Add Content-Security-Policy: upgrade-insecure-requests to HTTP responses
--hsts: Add Strict-Transport-Security header to HTTP responses
--staple-ocsp: Enable OSCP stapling (allow browser to skip verification of whether the SSL certificate has been revoked or not)
--email: Email address to use to register with Let’s Encrypt, and potentially for recovery

How to do a basic install of Prometheus on Ubuntu?

Posted on October 9, 2018 by fabricetriboix

Machine requirement

I will assume your machine is Ubuntu >= 16.04.

Please make sure only ports 80 and 443 are open.

I assume there are two DNS entries to your machine, such as prometheus.blablabla.com and grafana.blablabla.com (replace as appropriate throughout the tutorial).

Install Prometheus

Follow the steps (you can replace version 2.4.3 with whatever latest version is available for you):

$ sudo useradd --no-create-home --shell /bin/false prometheus
$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xf prometheus-2.4.3.linux-amd64.tar.gz
$ cd prometheus-2.4.3.linux-amd64
$ sudo cp prometheus promtool /usr/local/bin
$ sudo mkdir /etc/prometheus /var/lib/prometheus
$ sudo cp -r prometheus.yml consoles console_libraries /etc/prometheus
$ sudo chown prometheus.prometheus /var/lib/prometheus

Create the file /etc/systemd/system/prometheus.service and put the following content in it:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries
Restart=always

[Install]
WantedBy=multi-user.target

Then run:

$ sudo systemctl daemon-reload
$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus

Install Grafana

To install Grafana:

$ echo "deb https://packagecloud.io/grafana/stable/debian/ stretch main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
$ curl -sSL https://packagecloud.io/gpg.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install grafana
$ sudo systemctl daemon-reload
$ sudo systemctl enable grafana-server
$ sudo systemctl start grafana-server

Do not try to configure Grafana yet! We need to do all the SSL stuff first.

Reverse-proxy and SSL certificates

Install certbot

Follow the steps:

$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot python-certbot-nginx

Reverse-proxy for Prometheus

Prometheus web interface has no authentication mechanism, so we will add one with nginx.

To install nginx as a reverse-proxy for SSL termination and authentication for Prometheus (replace USERNAME and PASSWORD with a username and password of your choice):

$ sudo apt-get install nginx
$ sudo rm /etc/nginx/sites-enabled/default
$ echo "USERNAME:$(openssl passwd -apr1 PASSWORD)" | tee -a /etc/nginx/prometheus.htpasswd

Then edit the file /etc/nginx/sites-available/prometheus to look like this (replace prometheus.blablabla.com with your real domain name; NB: more SSL config will be added by certbot):

server {
  listen 80;
  server_name prometheus.blablabla.com;
}

server {
  listen 443 ssl;
  server_name prometheus.blablabla.com;

  location / {
    proxy_pass http://localhost:9090;
    proxy_set_header Host $host;
    auth_basic "My Prometheus";
    auth_basic_user_file /etc/nginx/prometheus.htpasswd;
  }
}

Reload nginx:

$ sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Create an SSL certificate for prometheus.blablabla.com:

$ sudo certbot --nginx -d prometheus.blablabla.com

Certbot will start an interactive session. Answer as follows:

When asked for your email address, enter it
Agree to the terms of service
Agree or not to the marketing, that has no incidence on your SSL certificate
When asked to redirect all HTTP traffic to HTTPS, choose to redirect

Reverse-proxy for Grafana

Edit the file /etc/nginx/sites-available/grafana to look like this (NB: more SSL config will be added by certbot):

server {
  listen 80;
  server_name grafana.blablabla.com;
}

server {
  listen 443 ssl;
  server_name grafana.blablabla.com;

  location / {
    proxy_pass http://localhost:3000;
    proxy_set_header Host $host;
  }
}

Reload nginx:

$ sudo ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Create an SSL certificate for grafana.blablabla.com:

$ sudo certbot --nginx -d grafana.blablabla.com

Answer the interactive prompts in the same manner as previously.

Configure Grafana

Open your favourite browser to http://grafana.blablabla.com. The Grafana login page should come up. Enter admin for both the username and password; the next page should ask you for a more secure password.

Then click on the 2nd icon “Add data source” and enter the following details:

Name: Whatever you like
Type: Prometheus
URL: http://localhost:9090
Leave the rest untouched
Click “Save & Test”
You should see a message at the bottom of the page “Data source is working”

Install and configure Alertmanager

Follow the steps (you can replace version 0.15.2 with whatever latest version is available for you):

$ sudo useradd --no-create-home --shell /bin/false alertmanager
$ wget //github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz
$ tar xf alertmanager-0.15.2.linux-amd64.tar.gz
$ cd alertmanager-0.15.2.linux-amd64
$ sudo cp alertmanager amtool /usr/local/bin
$ sudo mkdir -p /etc/alertmanager/templates /var/lib/alertmanager
$ sudo chown alertmanager.alertmanager /var/lib/alertmanager

Create the file /etc/alertmanager/alertmanager.yml so it looks like this (substitute the correct values for your setup):

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@prometheus.blablabla.com'
  smtp_auth_username: ''
  smtp_auth_password: ''
  smtp_require_tls: false

templates:
  - /etc/alertmanager/templates/*.tmpl

route:
  repeat_interval: 1h
  receiver: MYTEAM

receivers:
  - name: MYTEAM
    email_configs:
    - to: MYTEAM@MYORG.com
    slack_configs:
    - api_url: https://hooks.slack.com/services/XXXXXX/XXXXXX/XXXXXX
      send_resolved: true

Create the file /etc/systemd/system/altermanager.service and put the following content in it:

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
        --config.file /etc/alertmanager/alertmanager.yml \
        --storage.path /var/lib/alertmanager
Restart=always

[Install]
WantedBy=multi-user.target

Then run:

$ sudo systemctl daemon-reload
$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus

You need to tell prometheus that an alertmanager service is available. To do so, edit the /etc/prometheus/prometheus.yml file and add the following at the end:

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - localhost:9093

Then:

$ sudo systemctl restart prometheus

How to do a basic install of an ELK stack on Ubuntu for log stashing?

Posted on September 5, 2018 by fabricetriboix

Install and configure elasticsearch

Use a machine with at least 4GiB of RAM. This guide will install Elasticsearch, Logstash and Kibana all on the same machine, so this is suited only for a small-scale setup.


# apt-get install apt-transport-https software-properties-common wget
# add-apt-repository ppa:webupd8team/java
# apt-get update
# apt-get install oracle-java8-installer
# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
# echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
# apt-get update
# apt-get install elasticsearch
# systemctl enable elasticsearch
# systemctl start elasticsearch

Then edit `/etc/elasticsearch/elasticsearch.yml` and do the following:

Set `node.name` to something descriptive
Set `cluster.name` to something unique to avoid issues with auto-discovery
Set `network.host` to “localhost”
Change `path.data` and `path.logs` if necessary

Edit `/etc/elasticsearch/jvm.options` and set the heap size according to your machine RAM, for example (to set it to 2GiB):


-Xms2g
-Xmx2g

A setting of 1GiB is probably the minimum you should use.

Make sure you disable swapping.

Install and configure Kibana

Run this: `$ sudo apt-get install kibana`. Then edit `/etc/kibana/kibana.yml` and set `server.host` to “localhost”.

Then: `# systemctl enable kibana && systemctl start kibana`

Create a DNS entry for your kibana host and an SSL certificate.

Install nginx as reverse proxy with authentication:


# apt-get install nginx
# rm /etc/nginx/sites-enabled/default
# echo "admin:$(openssl passwd -apr1 PASSWORD)" | tee -a /etc/nginx/kibana.htpasswd

Then edit a nginx site file, eg: `/etc/nginx/sites-available/kibana` and make it look like this:


server {
    listen 80 default_server;
    server_name _;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 default_server ssl http2;
    server_name _;

    ssl_certificate /etc/letsencrypt/live/YOURDOMAIN/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/YOURDOMAIN/privkey.pem;

    auth_basic "My Kibana";
    auth_basic_user_file /etc/nginx/kibana.htpasswd;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Activate the site like so:


# ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana
# nginx -t
# systemctl reload nginx

Install and configure logstash

Just run: `$ sudo apt-get install logstash`

Edit `/etc/logstash/jvm.options` and set the heap size according to your machine RAM, for example (to set it to 2GiB):


-Xms2g
-Xmx2g

A setting of 1GiB is probably the minimum you should use.

Add a file `/etc/logstash/conf.d/GIVE_ME_A_NAME.conf` and edit it so it looks like this:


input {
    beats {
        port => "5044"
    }
}

filter {
    if [fields][log_type] == "apache-access" {
        grok {
            match => { "message" => "%{IPORHOST:vhost}:%{NUMBER:vhost_port} %{COMBINEDAPACHELOG}" }
        }
        geoip {
            source => "clientip"
        }
    } else if [fields][log_type] == "apache-error" {
       grok {
           match => { "message" => "%{IPORHOST:vhost} \[%{TIMESTAMP_ISO8601:timestamp}\] \[%{DATA:module}:%{LOGLEVEL}\] \[pid: %{POSINT:pid}:tid %{DATA:tid}\] \[OS error: %{DATA:oserror}\] \[client %{DATA:clientip}\] %{GREEDYDATA:error_message}" }
       }
       geoip {
           source => "clientip"
       }
    }
}

output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "%{[fields][log_type]}-%{+YYYYMMdd}"
    }
}

Then run: `# systemctl restart logstash`

NB: The grok patterns assume the following log formats for apache:

ErrorLogFormat “%-v [%{cu}t] [%-m:%l] [pid: %-P:tid %-T] [OS error: %-E] [client %-a] %M” LogFormat “%v:%p %h %l %u %t \”%r\” %>s %O \”%{Referer}i\” \”%{User-Agent}i\”” vhost_combined

NB: Use this to test your grok patterns.

Install and configure filebeat

Filebeat goes on the machine where your service is running.

To install filebeat:


# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
# apt-get install apt-transport-https
# echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | 
tee -a /etc/apt/sources.list.d/elastic-6.x.list
# apt-get update
# apt-get install filebeat

Then edit `/etc/filebeat/filebeat.yml` so it looks like this:


filebeat.inputs:

- type: log
  paths:
   - /var/log/apache2/access.log
  fields:
   log_type: apache-access

- type: log
  paths:
   - /var/log/apache2/error.log
  fields:
   log_type: apache-error

output.logstash:
 hosts: ["ELK_HOSTNAME_OR_IP:5044"]

#output.console:
# pretty: true

Then run `# systemctl restart filebeat`

How to install the dns-route53 plugin for certbot on Ubuntu?

Posted on August 24, 2018 by fabricetriboix

There are no instruction on how to install the dns route53 plugin for certbot. Here is how to do it for Ubuntu.

To install certbot:


$ sudo apt-get update
$ sudo apt-get install software-properties-common
$ sudo apt-add-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot

To install the dns-route53 plugin:

# apt-get install python3-pip
$ sudo pip3 install certbot-dns-route53

You can then create a new certificate with something like that:


$ sudo certbot certonly --dns-route53 -d YOUR-DOMAIN.com --deploy-hook 'systemctl reload apache2'

How to create a self-signed SSL certificate?

Posted on August 23, 2018 by fabricetriboix

Do this:

$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365

How to generate a fingerprint of an ssh key?

Posted on August 18, 2018 by fabricetriboix

Run the following:

$ ssh-keygen -l -f /PATH/TO/PUBLIC/KEY

Or for the old MD5 fingerprint:

$ ssh-keygen -l -E md5 -f /PATH/TO/PUBLIC/KEY

Are you puzzled about why ssh does not use your ssh-agent?

Posted on August 18, 2018 by fabricetriboix

It could be that you are using the IdentityFile combined with the IdentitiesOnly option. When IdentitiesOnly is set to yes, ssh will not try to use the ssh agent, but only the key that you specified with IdentityFile. Consequently, you have to enter the passphrase every time.

Instead, use the CertificateFile option (and specify the corresponding public key). This way, ssh will use the ssh agent properly.

Please note that in that case, you will need to load the private key in the ssh-agent first, eg: $ ssh-add /PATH/TO/PRIVATE/KEY. If you don’t, ssh will not find the private key in the ssh agent, and has no real way to know what is the private key to use otherwise.

How to get the SSH public key from the private key?

Posted on August 18, 2018 by fabricetriboix

Run the following:

$ ssh-keygen -y -f /PATH/TO/PRIVATE/KEY

How to display the content of an SSL certificate?

Posted on June 19, 2018 by fabricetriboix

Run this:

$ openssl x509 -in SSL-CERTIFICATE.PEM -text

How to use swaks to send a test email?

Posted on June 7, 2018 by fabricetriboix

It can be downloaded here

./swaks --auth \
        --server smtp.mailgun.org \
        --au USERNAME@YOUR_DOMAIN_NAME \
        --ap YOUR_PASSWORD \
        --from USERNAME@YOUR_DOMAIN_NAME \
        --to bar@example.com \
        --h-Subject: "This is a test email" \
        --body "Hello, World!"

DevOps Fu

Tips and Tricks for DevOps Engineers

Category: Sysadmin

How to properly use certbot with Nginx on Ubuntu?

How to do a basic install of Prometheus on Ubuntu?

Machine requirement

Install Prometheus

Install Grafana

Reverse-proxy and SSL certificates

Install certbot

Reverse-proxy for Prometheus

Reverse-proxy for Grafana

Configure Grafana

Install and configure Alertmanager

How to do a basic install of an ELK stack on Ubuntu for log stashing?

Install and configure elasticsearch

Install and configure Kibana

Install and configure logstash

Install and configure filebeat

How to install the dns-route53 plugin for certbot on Ubuntu?

How to create a self-signed SSL certificate?

How to generate a fingerprint of an ssh key?

Are you puzzled about why ssh does not use your ssh-agent?

How to get the SSH public key from the private key?

How to display the content of an SSL certificate?

How to use swaks to send a test email?