Over the last few months, I have spent a lot of time working on AWS. I often need to spin up EC2 instances, databases, or other assets for testing. Doing this by hand can become burdensome. You need to click through the AWS CLI and keep track of everything you have created. This sounds like a perfect use case for infrastructure as code. Enter Pulumi!
Motivation
If you are familiar with python, the learning curve for Pulumi is relatively low. I quickly learned how to spin up and destroy infrastructure in a programmatic way. One topic that I always struggled with was configuration and template files. Pulumi has no obvious built-in way to create template files that contain the dynamic values generated from Pulumi. For example, I may want to pass the IP address of my EC2 instance into a .env file.
After many different experiments, I have finally landed on a pattern that allows me to write templates using Jinja2. This approach will enable me to:
- Define and render templates using Jinja2.
- Automatically update the templates by hashing the template files.
TL/DR
You can find all of the code on GitHub: https://github.com/SamEdwardes/personal-blog/tree/main/blog/2022-07-14-pulumi-with-jinja-templates. You can download the code as a zip file using this link from DownGit.
Run the code below to spin up the infrastructure for yourself!
# Set your AWS Environment Variables so that Pulumi can access AWS
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
# Create a new virtual environment
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
pip install -r requirements.txt
# Create a key pair
python create_keypair.py
chmod 400 key.pem
# Create a new pulumi stack
pulumi stack init dev
# Spin up the infrastructure
pulumi up
# SSH into the EC2 instance and verify that the .env file has been set
ssh -i key.pem -o StrictHostKeyChecking=no ubuntu@$(pulumi stack output server_public_dns)
cat .env
Setup
To spin up AWS infrastructure, Pulumi needs to be able to log into your account. You can do this through AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and environment variables.
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
Next, create a virtual environment, and install the dependencies:
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
pip install -r requirements.txt
Lets see what is inside the requirements.txt:
pulumi>=3.0.0,<4.0.0
pulumi-aws>=5.0.0,<6.0.0
pulumi-command
Jinja2
pycryptodome
There are several packages we installed in addition to the minimum requirements from Pulumi:
- pulumi-command: Used to execute commands on the remote EC2 server.
- Jinja2: Used to generate dynamic templates.
- pycryptodome: Used to hash our template files so that Pulumi automatically.
Next, we need to create a new public and private key that we will use to SSH into our EC2 instance. I made a Python script that can quickly generate a unique public/private key pair. Create a unique pair by running:
python create_keypair.py
chmod 400 key.pem
Expand to see create_keypair.py
import os
from Crypto.PublicKey import RSA
def main():
"""Create a new keypair."""
key = RSA.generate(2048)
private_key = key.exportKey("PEM")
public_key = key.publickey().exportKey("OpenSSH")
with open("key.pem", "w") as f:
f.write(private_key.decode())
with open("key.pub", "w") as f:
f.write(public_key.decode())
return public_key, private_key
if __name__ == '__main__':
main()
Lastly, create a new pulumi stack. I will name my stack dev, but you can call it whatever you like.
pulumi stack init dev
pulumi up
You are now ready to spin up your infrastructure. Just run:
pulumi up
__main__.py deep dive
Let us take a closer look at main.py and see what is happening.
Expand to see __main.py__
"""An AWS Python Pulumi program"""
import hashlib
from pathlib import Path
import jinja2
import pulumi
from pulumi_aws import ec2
from pulumi_command import remote
# ------------------------------------------------------------------------------
# Helper functions
# ------------------------------------------------------------------------------
def create_template(path: str) -> jinja2.Template:
with open(path, 'r') as f:
template = jinja2.Template(f.read())
return template
def hash_file(path: str) -> pulumi.Output:
with open(path, mode="r") as f:
text = f.read()
hash_str = hashlib.sha224(bytes(text, encoding='utf-8')).hexdigest()
return pulumi.Output.concat(hash_str)
# ------------------------------------------------------------------------------
# Infrastructure
# ------------------------------------------------------------------------------
def main():
# Customize these with your own tags!
tags = {
"rs:environment": "development",
"rs:owner": "name@email.com",
"rs:project": "solutions",
}
key_pair = ec2.KeyPair(
"ec2 key pair",
key_name=f"keypair-for-pulumi",
public_key=Path("key.pub").read_text(),
tags=tags | {"Name": "keypair-for-pulumi"},
)
# Make security groups
security_group = ec2.SecurityGroup(
"security group",
description="Security group for my blog post",
ingress=[{"protocol": "TCP", "from_port": 22, "to_port": 22, 'cidr_blocks': ['0.0.0.0/0'], "description": "SSH"}],
egress=[{"protocol": "All", "from_port": -1, "to_port": -1, 'cidr_blocks': ['0.0.0.0/0'], "description": "Allow all outbound traffic"}],
tags=tags
)
# Create a new ec2 instance
server = ec2.Instance(
"EC2 instance",
instance_type="t3.medium",
vpc_security_group_ids=[security_group.id],
ami="ami-0fb653ca2d3203ac1", # Ubuntu Server 20.04 LTS (HVM), SSD Volume Type
tags=tags,
key_name=key_pair.key_name
)
pulumi.export('server_public_dns', server.public_dns)
# Create a connection that will be used to SSH into the ec2 instance
connection = remote.ConnectionArgs(
host=server.public_dns,
user="ubuntu",
private_key=Path("key.pem").read_text()
)
# Render a template on the ec2 instance
local_file_path = "templates/template.env"
remote_file_path = "~/.env"
command_render_template = remote.Command(
"copy .env",
create=pulumi.Output.concat(
'echo "',
pulumi.Output.all(
public_ip=server.public_ip,
availability_zone=server.availability_zone,
cpu_core_count=server.cpu_core_count
).apply(
lambda args: create_template(local_file_path).render(
ip_address=args['public_ip'],
availability_zone=args['availability_zone'],
cpu_core_count=args['cpu_core_count']
)
),
f'" > {remote_file_path}'
),
connection=connection,
opts=pulumi.ResourceOptions(depends_on=[server]),
triggers=[hash_file(local_file_path)]
)
main()
Below we will walk through some of the critical parts of the code.
create_template(path: str)
import jinja2
def create_template(path: str) -> jinja2.Template:
with open(path, 'r') as f:
template = jinja2.Template(f.read())
return template
This is a helper function so we can quickly create a jinja2.Template object. We wrap this logic in a function so we can easily call it later inside of pulumi_command.remote.Command.
hash_file(path: str)
import hashlib
def hash_file(path: str) -> pulumi.Output:
with open(path, mode="r") as f:
text = f.read()
hash_str = hashlib.sha224(bytes(text, encoding='utf-8')).hexdigest()
return pulumi.Output.concat(hash_str)
This function will create a unique hash of our template files. Note that we return a pulumi.Output object.
command_render_template
This code chunk is really the core of what we are doing:
# Render a template on the ec2 instance
local_file_path = "templates/template.env"
remote_file_path = "~/.env"
command_render_template = remote.Command(
"copy .env",
create=pulumi.Output.concat(
'echo "',
pulumi.Output.all(
public_ip=server.public_ip,
availability_zone=server.availability_zone,
cpu_core_count=server.cpu_core_count
).apply(
lambda args: create_template(local_file_path).render(
ip_address=args['public_ip'],
availability_zone=args['availability_zone'],
cpu_core_count=args['cpu_core_count']
)
),
f'" > {remote_file_path}'
),
connection=connection,
opts=pulumi.ResourceOptions(depends_on=[server]),
triggers=[hash_file(local_file_path)]
)
The basic approach is to run an echo command on the remote server that writes our rendered template to a file.
echo "TEMPLATE CONTENTS" > .env
The tricky part is getting our rendered template into "TEMPLATE CONTENTS". To do this, we need to use pulumi.Output.concat. This function allows you to use the output of other pulumi.Output objects. Notice that we pass in all the values we want to access in our template.
pulumi.Output.all(
public_ip=server.public_ip,
availability_zone=server.availability_zone,
cpu_core_count=server.cpu_core_count
)
Then, we can use pulumi.Output.concat().apply to pass these values into another function. Here, we will create a jinja2.Template object with our create_template function and dynamically render the values.
pulumi.Output.all(
public_ip=server.public_ip,
availability_zone=server.availability_zone,
cpu_core_count=server.cpu_core_count
).apply(
lambda args: create_template(local_file_path).render(
ip_address=args['public_ip'],
availability_zone=args['availability_zone'],
cpu_core_count=args['cpu_core_count']
)
)
Note that the keyword arguments inside create_template(local_file_path).render should match the values in your template.
IP_ADDRESS={{ip_address}}
CPU_CORE_COUNT={{cpu_core_count}}
AVAILABILITY_ZONE={{availability_zone}}
See the results
Now that you have run pulumi up and created a dynamically rendered template let us check out the results. SSH into your EC2 instance:
ssh -i key.pem -o StrictHostKeyChecking=no ubuntu@$(pulumi stack output server_public_dns)
Once inside your EC2 instance, inspect the template that we generated.
cat .env
IP_ADDRESS=3.21.35.72
CPU_CORE_COUNT=1
AVAILABILITY_ZONE=us-east-2c
Wrap up
With jinja2 and Pulumi we are now able to turn this:
IP_ADDRESS={{ip_address}}
CPU_CORE_COUNT={{cpu_core_count}}
AVAILABILITY_ZONE={{availability_zone}}
Into this!
IP_ADDRESS=3.21.35.72
CPU_CORE_COUNT=1
AVAILABILITY_ZONE=us-east-2c