How to install and configure AWS CloudWatch agent on Ubuntu?

I will assume that you want to install the CloudWatch agent on an EC2 instance. I took the example of sending Apache logs to CloudWatch Logs.

Ssh to your EC2 instance running Ubuntu 18.04 and running the following:

$ wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
$ sudo dpkg -i amazon-cloudwatch-agent.deb

NB: Do not modify any installed files, especially the ones in /opt/aws/amazon-cloudwatch-agent/etc.

You can run the configuration wizard /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard, or edit the configuration file manually which gives you more control. The documentation is available here.

For example:

$ sudo vi /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
$ cat /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

For example:

{
  "agent": {
    "metrics_collection_interval": 60,  // Default metrics collection interval in seconds
    "logfile": "/var/log/amazon-cloudwatch-agent.log",  // Where should the CloudWatch Agent writes its logs
    "debug": false  // Should CloudWatch Agent run in debug mode
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/apache2/access.log",  // File where logs are written that should be forwarded to CloudWatch Logs
            "log_group_name": "apache-access",  // Log group to save logs to
            "log_stream_name": "{instance_id}"  // Log stream to save logs to
          },
          {
            "file_path": "/var/log/apache2/error.log",
            "log_group_name": "apache-error",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  },
  "metrics": {
    "append_dimensions": {  // Dimensions to add to all metrics
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "InstanceId": "${aws:InstanceId}"
    },
    "aggregation_dimensions": [["AutoScalingGroupName"]],  // Dimensions on which metrics are to be aggregated
    "metrics_collected": {
      "collectd": {  // Collect metrics through collectd; you must install collectd for this to work
        "metrics_aggregation_interval": 60
      },
      "statsd": {  // CloudAgent will act as a statsd server to allow collection of additional metrics
        "metrics_aggregation_interval": 60,
        "metrics_collection_interval": 10,
        "service_address": ":8125"
      },
      "cpu": {
        "measurement": [
          "cpu_usage_idle",
          "cpu_usage_iowait",
          "cpu_usage_user",
          "cpu_usage_system"
        ],
        "totalcpu": true  // Set to `false` to show each CPU, `true` to aggregate all CPUs
      },
      "disk": {
        "measurement": [
          "used_percent",
          "inodes_free"
        ],
        "resources": [
          "*"
        ]
      },
      "diskio": {
        "measurement": [
          "io_time",
          "write_bytes",
          "read_bytes",
          "writes",
          "reads"
        ],
        "resources": [
          "*"
        ]
      },
      "mem": {
        "measurement": [
          "mem_used_percent"
        ]
      },
      "netstat": {
        "measurement": [
          "tcp_established",
          "tcp_time_wait"
        ]
      },
      "swap": {
        "measurement": [
          "swap_used_percent"
        ]
      }
    }
  }
}

To start the CloudWatch agent, run the following:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s

Please note this will convert the JSON config file into a TOML file which is actually used by the CloudWatch Agent; the original JSON file will be deleted.

The CloudWatch agent is integrated with systemd. It will start automatically after a reboot, and you can restart it like so:

$ sudo systemctl restart amazon-cloudwatch-agent

Restarting the agent using systemd will use the existing TOML file and will ignore any new JSON file. If you create a new configuration JSON file, you will need to use the amazon-cloudwatch-agent-ctl command.