How to install and configure AWS CloudWatch agent on Ubuntu?

I will assume that you want to install the CloudWatch agent on an EC2 instance. I took the example of sending Apache logs to CloudWatch Logs.

Ssh to your EC2 instance running Ubuntu 18.04 and running the following:

$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
$ sudo dpkg -i amazon-cloudwatch-agent.deb

NB: Do not modify any installed files, especially the ones in /opt/aws/amazon-cloudwatch-agent/etc.

You can run the configuration wizard /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard, or edit the configuration file manually which gives you more control. The documentation is available here.

For example:

$ sudo vi /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
$ cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

For example:

{
  "agent": {
    "metrics_collection_interval": 60,  // Default metrics collection interval in seconds
    "logfile": "/var/log/amazon-cloudwatch-agent.log",  // Where should the CloudWatch Agent writes its logs
    "debug": false  // Should CloudWatch Agent run in debug mode
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/apache2/access.log",  // File where logs are written that should be forwarded to CloudWatch Logs
            "log_group_name": "apache-access",  // Log group to save logs to
            "log_stream_name": "{instance_id}"  // Log stream to save logs to
          },
          {
            "file_path": "/var/log/apache2/error.log",
            "log_group_name": "apache-error",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  },
  "metrics": {
    "append_dimensions": {  // Dimensions to add to all metrics
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "InstanceId": "${aws:InstanceId}"
    },
    "aggregation_dimensions": [["AutoScalingGroupName"]],  // Dimensions on which metrics are to be aggregated
    "metrics_collected": {
      "collectd": {  // Collect metrics through collectd; you must install collectd for this to work
        "metrics_aggregation_interval": 60
      },
      "statsd": {  // CloudAgent will act as a statsd server to allow collection of additional metrics
        "metrics_aggregation_interval": 60,
        "metrics_collection_interval": 10,
        "service_address": ":8125"
      },
      "cpu": {
        "measurement": [
          "cpu_usage_idle",
          "cpu_usage_iowait",
          "cpu_usage_user",
          "cpu_usage_system"
        ],
        "totalcpu": true  // Set to `false` to show each CPU, `true` to aggregate all CPUs
      },
      "disk": {
        "measurement": [
          "used_percent",
          "inodes_free"
        ],
        "resources": [
          "*"
        ]
      },
      "diskio": {
        "measurement": [
          "io_time",
          "write_bytes",
          "read_bytes",
          "writes",
          "reads"
        ],
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ]
      },
      "netstat": {
        "measurement": [
          "tcp_established",
          "tcp_time_wait"
        ]
      },
      "swap": {
        "measurement": [
          "swap_used_percent"
        ]
      }
    }
  }
}

To start the CloudWatch agent, run the following:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

Please note this will convert the JSON config file into a TOML file which is actually used by the CloudWatch Agent; the original JSON file will be deleted.

The CloudWatch agent is integrated with systemd. It will start automatically after a reboot, and you can restart it like so:

$ sudo systemctl restart amazon-cloudwatch-agent

Restarting the agent using systemd will use the existing TOML file and will ignore any new JSON file. If you create a new configuration JSON file, you will need to use the amazon-cloudwatch-agent-ctl command.

How to properly use certbot with Nginx on Ubuntu?

Install and configure Nginx:

$ sudo apt update
$ sudo apt install nginx
$ sudo rm /etc/nginx/sites-enabled/default
$ sudo vi /etc/nginx/sites-available/MYSITE  # See below
$ sudo ln -s /etc/nginx/sites-available/MYSITE /etc/nginx/sites-enabled
$ sudo systemctl restart nginx

The /etc/nginx/sites-available/MYSITE config file should look like so:

server {
  listen 80;
  server_name MYSITE.COM WWW.MYSITE.COM;
}

server {
  listen 443 ssl;
  server_name MYSITE.COM WWW.MYSITE.COM;
  server_tokens off;

  # If you use nginx as reverse-proxy
  location / {
    proxy_pass http://localhost:8080;
    proxy_set_header Host $host;
  }
}

Install certbot:

$ sudo apt update
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt update
$ sudo apt install certbot python-certbot-nginx

Create the certificate:

$ sudo certbot --nginx --agree-tos --redirect --uir --hsts --staple-ocsp --email ME@MYSITE.COM -d MYSITE.COM -d WWW.MYSITE.COM

You could potentially use the --must-staple option to enable the OCSP “must staple” extension, but it doesn’t work well with Firefox (to fix in your Firefox browser: go to about:config, and set security.ssl.enable_ocsp_must_staple to false).

The arguments are:

  • --nginx: Use Nginx for authentication and install certificate for Nginx
  • --agree-tos: Agree with Let’s Encrypt terms of service
  • --redirect: Add Nginx config to redirect HTTP to HTTPS
  • --uir: Add Content-Security-Policy: upgrade-insecure-requests to HTTP responses
  • --hsts: Add Strict-Transport-Security header to HTTP responses
  • --staple-ocsp: Enable OSCP stapling (allow browser to skip verification of whether the SSL certificate has been revoked or not)
  • --email: Email address to use to register with Let’s Encrypt, and potentially for recovery

How to install an old version of PHP using geerlingguy.php on Ubuntu target

Your playbook should look like so:

---
- hosts: XYZ
  become: yes

  pre_tasks:
    - name: Add repository for old PHP
      apt_repository:
        repo: ppa:ondrej/php
        update_cache: yes
        state: present

  roles:
    - role: geerlingguy.apache
      ...

    - role: geerlingguy.php
      php_webserver_daemon: apache2
      php_default_version_debian: "7.1"
      php_version: "7.1"
      php_packages:
        - php7.1
        - php7.1-cli
        - etc...

How to install and kick off minikube on Ubuntu?

Install kubectl:

$ sudo apt-get update
$ sudo apt-get install apt-transport-https
$ curl -sSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-get install kubectl

Install virtualbox:

$ sudo apt-get update
$ sudo apt-get install virtualbox

Install minikube (replace v0.30.0 with the latest version available for you):

$ wget https://github.com/kubernetes/minikube/releases/download/v0.30.0/minikube-linux-amd64
$ mv minikube-linux-amd64 minikube
$ chmod +x minikube
$ sudo cp minikube /usr/local/bin/
$ rm minikube

Create the VM and start the cluster (this will create the file ~/.kube/config):

$ minikube start  # use `minikube start --help` for more options
$ kubectl cluster-info

To get the IP address of the cluster:

$ minikube ip
$ sudo vi /etc/hosts  # if you want to assign names to this IP address

To show the Kubernetes dashboard:

$ minikube dashboard

To stop the cluster (shuts down the VM and preserve all states):


$ minikube stop

To restart the VM and cluster:

$ minikube start

To delete the cluster (and the VM):

$ minikube delete
$ rm ~/.kube/config

How to do a basic install of Prometheus on Ubuntu?

Machine requirement

I will assume your machine is Ubuntu >= 16.04.

Please make sure only ports 80 and 443 are open.

I assume there are two DNS entries to your machine, such as prometheus.blablabla.com and grafana.blablabla.com (replace as appropriate throughout the tutorial).

Install Prometheus

Follow the steps (you can replace version 2.4.3 with whatever latest version is available for you):

$ sudo useradd --no-create-home --shell /bin/false prometheus
$ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
$ tar xf prometheus-2.4.3.linux-amd64.tar.gz
$ cd prometheus-2.4.3.linux-amd64
$ sudo cp prometheus promtool /usr/local/bin
$ sudo mkdir /etc/prometheus /var/lib/prometheus
$ sudo cp -r prometheus.yml consoles console_libraries /etc/prometheus
$ sudo chown prometheus.prometheus /var/lib/prometheus

Create the file /etc/systemd/system/prometheus.service and put the following content in it:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries
Restart=always

[Install]
WantedBy=multi-user.target

Then run:

$ sudo systemctl daemon-reload
$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus

Install Grafana

To install Grafana:

$ echo "deb https://packagecloud.io/grafana/stable/debian/ stretch main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
$ curl -sSL https://packagecloud.io/gpg.key | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install grafana
$ sudo systemctl daemon-reload
$ sudo systemctl enable grafana-server
$ sudo systemctl start grafana-server

Do not try to configure Grafana yet! We need to do all the SSL stuff first.

Reverse-proxy and SSL certificates

Install certbot

Follow the steps:

$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot python-certbot-nginx

Reverse-proxy for Prometheus

Prometheus web interface has no authentication mechanism, so we will add one with nginx.

To install nginx as a reverse-proxy for SSL termination and authentication for Prometheus (replace USERNAME and PASSWORD with a username and password of your choice):

$ sudo apt-get install nginx
$ sudo rm /etc/nginx/sites-enabled/default
$ echo "USERNAME:$(openssl passwd -apr1 PASSWORD)" | tee -a /etc/nginx/prometheus.htpasswd

Then edit the file /etc/nginx/sites-available/prometheus to look like this (replace prometheus.blablabla.com with your real domain name; NB: more SSL config will be added by certbot):

server {
  listen 80;
  server_name prometheus.blablabla.com;
}

server {
  listen 443 ssl;
  server_name prometheus.blablabla.com;

  location / {
    proxy_pass http://localhost:9090;
    proxy_set_header Host $host;
    auth_basic "My Prometheus";
    auth_basic_user_file /etc/nginx/prometheus.htpasswd;
  }
}

Reload nginx:

$ sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Create an SSL certificate for prometheus.blablabla.com:

$ sudo certbot --nginx -d prometheus.blablabla.com

Certbot will start an interactive session. Answer as follows:

  • When asked for your email address, enter it
  • Agree to the terms of service
  • Agree or not to the marketing, that has no incidence on your SSL certificate
  • When asked to redirect all HTTP traffic to HTTPS, choose to redirect

Reverse-proxy for Grafana

Edit the file /etc/nginx/sites-available/grafana to look like this (NB: more SSL config will be added by certbot):

server {
  listen 80;
  server_name grafana.blablabla.com;
}

server {
  listen 443 ssl;
  server_name grafana.blablabla.com;

  location / {
    proxy_pass http://localhost:3000;
    proxy_set_header Host $host;
  }
}

Reload nginx:

$ sudo ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/
$ sudo systemctl reload nginx

Create an SSL certificate for grafana.blablabla.com:

$ sudo certbot --nginx -d grafana.blablabla.com

Answer the interactive prompts in the same manner as previously.

Configure Grafana

Open your favourite browser to http://grafana.blablabla.com. The Grafana login page should come up. Enter admin for both the username and password; the next page should ask you for a more secure password.

Then click on the 2nd icon “Add data source” and enter the following details:

  • Name: Whatever you like
  • Type: Prometheus
  • URL: http://localhost:9090
  • Leave the rest untouched
  • Click “Save & Test”
  • You should see a message at the bottom of the page “Data source is working”

Install and configure Alertmanager

Follow the steps (you can replace version 0.15.2 with whatever latest version is available for you):

$ sudo useradd --no-create-home --shell /bin/false alertmanager
$ wget //github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz
$ tar xf alertmanager-0.15.2.linux-amd64.tar.gz
$ cd alertmanager-0.15.2.linux-amd64
$ sudo cp alertmanager amtool /usr/local/bin
$ sudo mkdir -p /etc/alertmanager/templates /var/lib/alertmanager
$ sudo chown alertmanager.alertmanager /var/lib/alertmanager

Create the file /etc/alertmanager/alertmanager.yml so it looks like this (substitute the correct values for your setup):

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'alertmanager@prometheus.blablabla.com'
  smtp_auth_username: ''
  smtp_auth_password: ''
  smtp_require_tls: false

templates:
  - /etc/alertmanager/templates/*.tmpl

route:
  repeat_interval: 1h
  receiver: MYTEAM

receivers:
  - name: MYTEAM
    email_configs:
    - to: MYTEAM@MYORG.com
    slack_configs:
    - api_url: https://hooks.slack.com/services/XXXXXX/XXXXXX/XXXXXX
      send_resolved: true

Create the file /etc/systemd/system/altermanager.service and put the following content in it:

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
        --config.file /etc/alertmanager/alertmanager.yml \
        --storage.path /var/lib/alertmanager
Restart=always

[Install]
WantedBy=multi-user.target

Then run:

$ sudo systemctl daemon-reload
$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus

You need to tell prometheus that an alertmanager service is available. To do so, edit the /etc/prometheus/prometheus.yml file and add the following at the end:

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - localhost:9093

Then:

$ sudo systemctl restart prometheus

How to install Docker on Ubuntu

Follow the steps:

$ sudo apt-get install curl software-properties-common
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce
$ sudo usermod -aG docker $(id -un)  # optional, allows you to run docker without `sudo`

How to enable and analyse the slow query log on MySQL?

Set the following variables:

slow_query_log = 1 (means true)
long_query_time = 0 (means 0 seconds)
log_queries_not_using_indexes = 1 (means true)
log_output = FILE (default is TABLE)

NB: Setting `long_query_time` to 0 will instruct MySQL to log all queries to the slow query log. Some queries might not be that slow, but if repeated often enough will have a significant impact on the performance of your MySQL database. Setting `long_query_time` to 0 will allow you to catch that kind of scenario as well.

Let MySQL run for a while, then download pt-query-digest, and run it like so:

$ pt-query-digest slow-query.log > slow-query.log.out

Look at the `slow-query.log.out` file to find out which queries are slowing down your MySQL database.