autozane Ruby

Managing Terraform Versions on Jenkins

When leveraging Terraform to code your infrastructure you will notice that the release cadence for new versions of Terraform is fast. In this post I will show you how to manage upgrading versions of Terraform on a per Jenkins Job basis. This will allow you to run multiple versions of Terraform on your Jenkins system and gives you the flexibility to control when to upgrade a given Terraform state to a newer version of Terraform.

We will be leveraging this great open source wrapper: tfenv. Thank you kamatam41 for sharing this.

To get this installed on a Jenkins system I have provide a small Chef snippet for this:

This will setup a couple versions of Terraform for our job to work with. Here is the example Jenkins Terraform job. We will leverage some Terraform code I put on Github that will create an AWS S3 bucket, a S3 bucket policy


Leave a message at the beep, I mean in the queue


Let's say you don't have an AWS Direct Connect or a VPN connection from your AWS account to your on-premise datacenter. But you have some processed data that needs to be sent back to your on-premise Hadoop cluster upon completion. This requires you to somehow initiate a process (in this example a Hadoop Distcp) from your on-premise. How do we know when the processed data is ready? How do we know when to start the data copy process? Just leave a message!

Here is an example of a small AWS SQS consumer application that works with Hadoop and AWS s3 to copy processed data from s3 to your local Hadoop cluster.

In this example scenario, we have Amazon EMR processing data that is outputted to s3. The last step in the EMR workflow is to post a message to an AWS SQS queue. The body of this message contains the