autozane Jenkins

Automating AWS CloudWatch Logs on Ubuntu

The AWS CloudWatch Logs service acts like a Logstash agent on your EC2 instances. It can be configured to capture log entires and send them to CloudWatch. There are a lot of different customization options with AWS CloudWatch Logs, such as how to format log entries, log group names, etc. In this post we will automate the installation of AWS CloudWatch Logs on an Ubuntu instance using PackerIO. The example service that we will capture logs for will be an Aptly API server. PackerIO will automate the installation and configuration of the service for us, and Terraform will be used to configure the IAM Role and Instance Profile we will need to be able to interact with CloudWatch and Log services on AWS. This is a working deployment strategy for Aptly server here. We will just focus on the awslogs agent install and required Terraform.

Let's start with the PackerIO configurations required to make this work.

Under the provisioner section lets download and install the awslogs agent.

  "provisioners": [
    {
      "type"   : "shell",
      "inline" : [
        "sudo apt-get update",
        "sudo apt-get -y install curl python software-properties-common xz-utils bzip2 gnupg wget graphviz",
        "sudo wget -O /tmp/awslogs-agent-setup.py http://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py",
        "sudo chmod 775 /tmp/awslogs-agent-setup.py",
        "sudo mkdir -p /var/awslogs/etc/"
      ]
    },

Next, let's call out an awslogs.conf file that is local to the PackerIO configuration that we can use with the PackerIO file provisioner.

    {
      "type"        : "file",
      "source"      : "awslogs/awslogs.conf.aptly",
      "destination" : "/var/awslogs/etc/awslogs.conf.packer"
    },

The final setup on the PackerIO side is to run a privileged Shell provisioner to perform a no-prompt install of awslogs and call use our awslogs.conf file that will pull in the logs of our Aptly API service.

    {
      "type"   : "shell",
      "inline" : [
        "sudo python /tmp/awslogs-agent-setup.py --region=us-west-2 --non-interactive -c /var/awslogs/etc/awslogs.conf.packer",
        "mv /tmp/aptly.service /etc/systemd/system/aptly.service",
        "mv /tmp/aptly.conf /etc/aptly.conf",
        "echo 'deb http://repo.aptly.info/ squeeze main' > /etc/apt/sources.list.d/aptly.list",
        "apt-key adv --keyserver keys.gnupg.net --recv-keys 2A194991",
        "apt-get update",
        "apt-get install aptly -y --force-yes",
        "touch /var/log/aptly-api.log",
        "sudo systemctl daemon-reload",
        "sudo systemctl enable awslogs",
        "sudo systemctl enable aptly"
      ],
      "execute_command" : "echo 'packer' | {{ .Vars }} sudo -E -S sh '{{ .Path }}'"
    }

The above shell provisioner configues awslogs to run on boot, and also our example Aptly API service which we will be forwarding logs for.

Let's run PackerIO from our Jenkins CI and save this AMI.

Now that we have a PackerIO built AMI we can promote some Terraform that will use the latest AMI based upon a name search. In our PackerIO configuration we saved the AMI with a name "aptly-{timestamp}". In Terraform we can call out the latest based upon this AMI. We are leveraging the new AMI data source here.

data "aws_ami" "aptly" {
  most_recent = true

  filter {
    name   = "name"
    values = ["aptly-*"]
  }

  owners = ["self"]
}

Our Terraform will apply the appropriate IAM Instance Profile to grant awslogs agent access to write logs into CloudWatch and to create log groups.

resource "aws_iam_instance_profile" "aptly_instance_profile" {
  name  = "aptly-instance-profile"
  roles = ["${aws_iam_role.aptly_role.name}"]
}

resource "aws_iam_role" "aptly_role" {
  name = "aptly-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_policy" "CloudWatchAccess" {
  name        = "CloudWatchAccess-aptly"
  description = "CloudWatch Access"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
               "cloudwatch:DeleteAlarms",
               "cloudwatch:DescribeAlarmHistory",
               "cloudwatch:DescribeAlarms",
               "cloudwatch:DescribeAlarmsForMetric",
               "cloudwatch:DisableAlarmActions",
               "cloudwatch:EnableAlarmActions",
               "cloudwatch:GetMetricData",
               "cloudwatch:GetMetricStatistics",
               "cloudwatch:ListMetrics",
               "cloudwatch:PutMetricAlarm",
               "cloudwatch:PutMetricData",
               "cloudwatch:SetAlarmState",
               "logs:CreateLogGroup",
               "logs:CreateLogStream",
               "logs:GetLogEvents",
               "logs:PutLogEvents",
               "logs:DescribeLogGroups",
               "logs:DescribeLogStreams",
               "logs:PutRetentionPolicy"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
EOF
}

resource "aws_iam_policy_attachment" "attach_cloudwatch" {
  name       = "aptly-iam-attachment"
  policy_arn = "${aws_iam_policy.CloudWatchAccess.arn}"
  roles      = ["${aws_iam_role.aptly_role.name}"]
}

Let's apply promote this get an EC2 instance running using Terraform and then validate that logs are coming into CloudWatch for the Aptly API server.

Great! With that instance running, let's confirm that we have logs coming into AWS CloudWatch now.

In this post we were able to dial in automated installation of AWS CloudWatch Logs (awslogs agent) with PackerIO and then used Terraform to create the required IAM Instance Profile that is attached to the instance. This Instance Profile gave the ec2 instance running our example service (Aptly API) and awslogs agent access to CloudWatch to create a Log Group and start sending logs to it.

Supporting PackerIO and Terraform Code Examples:
https://github.com/sepulworld/terraform-examples/tree/master/autozane_awslogs_aptly

Managing Terraform Versions on Jenkins

When leveraging Terraform to code your infrastructure you will notice that the release cadence for new versions of Terraform is fast. In this post I will show you how to manage upgrading versions of Terraform on a per Jenkins Job basis. This will allow you to run multiple versions of Terraform on your Jenkins system and gives you the flexibility to control when to upgrade a given Terraform state to a newer version of Terraform.

We will be leveraging this great open source wrapper: tfenv. Thank you kamatam41 for sharing this.

To get this installed on a Jenkins system I have provide a small Chef snippet for this:

This will setup a couple versions of Terraform for our job to work with. Here is the example Jenkins Terraform job. We will leverage some Terraform code I put on Github that will create an AWS S3 bucket, a S3 bucket policy to attach to the S3 bucket, and an AWS VPC.

I want to leverage a new Terraform AWS resource provider new to 0.7.3 called aws_s3_bucket_policy. We add it to our Terraform code example, seen here.

Let's first try to 'plan' this Terraform code using 0.7.2. With this version we should expect to see a failure to plan since the aws_s3_bucket_policy resource isn't available in this version. We can control the version we want to use for Terraform with the hidden file .terraform-version in our configuration directory where we run Terraform.

Note we see the failure:

aws_s3_bucket_policy.autozane_s3_policy: Provider doesn't support resource: aws_s3_bucket_policy

Ok, now let's commit a change to our .terraform-version file and call out the usage of 0.7.3 for this state. We should see now see a successful 'plan' that will include the creation of the requested AWS resources (S3 bucket, S3 Bucket Policy, VPC).

Success! We were able to leverage tfenv inside of our Jenkins CI environment to upgrade to a newer version of Terraform. This upgrade didn't affect any other Terraform jobs on this Jenkins system that were perhaps using other Terraform versions.

Example Terraform and Jenkins config

tfenv on Github

Chef to install tfenv on Jenkins