Machine requirement
I will assume your machine is Ubuntu >= 16.04.
Please make sure only ports 80 and 443 are open.
I assume there are two DNS entries to your machine, such as prometheus.blablabla.com
and grafana.blablabla.com
(replace as appropriate throughout the tutorial).
Install Prometheus
Follow the steps (you can replace version 2.4.3 with whatever latest version is available for you):
$ sudo useradd --no-create-home --shell /bin/false prometheus $ wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz $ tar xf prometheus-2.4.3.linux-amd64.tar.gz $ cd prometheus-2.4.3.linux-amd64 $ sudo cp prometheus promtool /usr/local/bin $ sudo mkdir /etc/prometheus /var/lib/prometheus $ sudo cp -r prometheus.yml consoles console_libraries /etc/prometheus $ sudo chown prometheus.prometheus /var/lib/prometheus
Create the file /etc/systemd/system/prometheus.service
and put the following content in it:
[Unit] Description=Prometheus Wants=network-online.target After=network-online.target [Service] User=prometheus Group=prometheus Type=simple ExecStart=/usr/local/bin/prometheus \ --config.file /etc/prometheus/prometheus.yml \ --storage.tsdb.path /var/lib/prometheus \ --web.console.templates /etc/prometheus/consoles \ --web.console.libraries /etc/prometheus/console_libraries Restart=always [Install] WantedBy=multi-user.target
Then run:
$ sudo systemctl daemon-reload $ sudo systemctl enable prometheus $ sudo systemctl start prometheus
Install Grafana
To install Grafana:
$ echo "deb https://packagecloud.io/grafana/stable/debian/ stretch main" | sudo tee -a /etc/apt/sources.list.d/grafana.list $ curl -sSL https://packagecloud.io/gpg.key | sudo apt-key add - $ sudo apt update $ sudo apt install grafana $ sudo systemctl daemon-reload $ sudo systemctl enable grafana-server $ sudo systemctl start grafana-server
Do not try to configure Grafana yet! We need to do all the SSL stuff first.
Reverse-proxy and SSL certificates
Install certbot
Follow the steps:
$ sudo apt install software-properties-common $ sudo add-apt-repository ppa:certbot/certbot $ sudo apt update $ sudo apt install certbot python-certbot-nginx
Reverse-proxy for Prometheus
Prometheus web interface has no authentication mechanism, so we will add one with nginx.
To install nginx as a reverse-proxy for SSL termination and authentication for Prometheus (replace USERNAME and PASSWORD with a username and password of your choice):
$ sudo apt install nginx $ sudo rm /etc/nginx/sites-enabled/default $ echo "USERNAME:$(openssl passwd -apr1 PASSWORD)" | tee -a /etc/nginx/prometheus.htpasswd
Then edit the file /etc/nginx/sites-available/prometheus
to look like this (replace prometheus.blablabla.com
with your real domain name; NB: don’t worry if the SSL config seems to lack some stuff, things will be added by certbot later):
server { listen 80; server_name prometheus.blablabla.com; } server { listen 443 ssl; server_name prometheus.blablabla.com; location / { proxy_pass http://localhost:9090; proxy_set_header Host $host; auth_basic "My Prometheus"; auth_basic_user_file /etc/nginx/prometheus.htpasswd; } }
Reload nginx:
$ sudo ln -s /etc/nginx/sites-available/prometheus /etc/nginx/sites-enabled/ $ sudo systemctl reload nginx
Create an SSL certificate for prometheus.blablabla.com
:
$ sudo certbot --nginx -d prometheus.blablabla.com
Certbot will start an interactive session. Answer as follows:
- When asked for your email address, enter it
- Agree to the terms of service
- Agree or not to the marketing, that has no incidence on your SSL certificate
- When asked to redirect all HTTP traffic to HTTPS, choose to redirect
Reverse-proxy for Grafana
Edit the file /etc/nginx/sites-available/grafana
to look like this (same remarks as before):
server { listen 80; server_name grafana.blablabla.com; } server { listen 443 ssl; server_name grafana.blablabla.com; location / { proxy_pass http://localhost:3000; proxy_set_header Host $host; } }
Reload nginx:
$ sudo ln -s /etc/nginx/sites-available/grafana /etc/nginx/sites-enabled/ $ sudo systemctl reload nginx
Create an SSL certificate for grafana.blablabla.com
:
$ sudo certbot --nginx -d grafana.blablabla.com
Answer the interactive prompts in the same manner as previously.
Configure Grafana
Open your favourite browser to http://grafana.blablabla.com
. The Grafana login page should come up. Enter admin
for both the username and password; the next page should ask you for a more secure password.
Then click on the 2nd icon “Add data source” and enter the following details:
- Name: Whatever you like
- Type: Prometheus
- URL: http://localhost:9090 (reminder: Prometheus is running on the same VM as Grafana)
- Leave the rest untouched
- Click “Save & Test”
- You should see a message at the bottom of the page “Data source is working”
Install and configure Alertmanager
Follow the steps (you can replace version 0.15.2 with whatever latest version is available for you):
$ sudo useradd --no-create-home --shell /bin/false alertmanager $ wget //github.com/prometheus/alertmanager/releases/download/v0.15.2/alertmanager-0.15.2.linux-amd64.tar.gz $ tar xf alertmanager-0.15.2.linux-amd64.tar.gz $ cd alertmanager-0.15.2.linux-amd64 $ sudo cp alertmanager amtool /usr/local/bin $ sudo mkdir -p /etc/alertmanager/templates /var/lib/alertmanager $ sudo chown alertmanager.alertmanager /var/lib/alertmanager
Create the file /etc/alertmanager/alertmanager.yml
so it looks like this (substitute the correct values for your setup):
global: smtp_smarthost: 'localhost:25' smtp_from: 'alertmanager@prometheus.blablabla.com' smtp_auth_username: '' smtp_auth_password: '' smtp_require_tls: false templates: - /etc/alertmanager/templates/*.tmpl route: repeat_interval: 1h receiver: MYTEAM receivers: - name: MYTEAM email_configs: - to: MYTEAM@MYORG.com slack_configs: - api_url: https://hooks.slack.com/services/XXXXXX/XXXXXX/XXXXXX send_resolved: true
Create the file /etc/systemd/system/altermanager.service
and put the following content in it:
[Unit] Description=Alertmanager Wants=network-online.target After=network-online.target [Service] User=alertmanager Group=alertmanager Type=simple ExecStart=/usr/local/bin/alertmanager \ --config.file /etc/alertmanager/alertmanager.yml \ --storage.path /var/lib/alertmanager Restart=always [Install] WantedBy=multi-user.target
Then run:
$ sudo systemctl daemon-reload $ sudo systemctl enable prometheus $ sudo systemctl start prometheus
You need to tell prometheus that an alertmanager service is available. To do so, edit the /etc/prometheus/prometheus.yml
file and add the following at the end:
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost: 9093
Then:
$ sudo systemctl restart prometheus
I work as a freelancer, so if you don’t want to do that kind of things yourself or don’t have the time, just drop me a line to hire me.