Using Jinja2 Templates with Pulumi

July 14, 2022

Over the last few months, I have spent a lot of time working on AWS. I often need to spin up EC2 instances, databases, or other assets for testing. Doing this by hand can become burdensome. You need to click through the AWS CLI and keep track of everything you have created. This sounds like a perfect use case for infrastructure as code. Enter Pulumi!

Motivation

If you are familiar with python, the learning curve for Pulumi is relatively low. I quickly learned how to spin up and destroy infrastructure in a programmatic way. One topic that I always struggled with was configuration and template files. Pulumi has no obvious built-in way to create template files that contain the dynamic values generated from Pulumi. For example, I may want to pass the IP address of my EC2 instance into a .env file.

After many different experiments, I have finally landed on a pattern that allows me to write templates using Jinja2. This approach will enable me to:

  • Define and render templates using Jinja2.
  • Automatically update the templates by hashing the template files.

TL/DR

You can find all of the code on GitHub: https://github.com/SamEdwardes/personal-blog/tree/main/blog/2022-07-14-pulumi-with-jinja-templates. You can download the code as a zip file using this link from DownGit.

Run the code below to spin up the infrastructure for yourself!

Terminal window
# Set your AWS Environment Variables so that Pulumi can access AWS
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>


# Create a new virtual environment
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
pip install -r requirements.txt


# Create a key pair
python create_keypair.py
chmod 400 key.pem


# Create a new pulumi stack
pulumi stack init dev


# Spin up the infrastructure
pulumi up


# SSH into the EC2 instance and verify that the .env file has been set
ssh -i key.pem -o StrictHostKeyChecking=no ubuntu@$(pulumi stack output server_public_dns)
cat .env

Setup

To spin up AWS infrastructure, Pulumi needs to be able to log into your account. You can do this through AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and environment variables.

Terminal window
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>

Next, create a virtual environment, and install the dependencies:

Terminal window
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
pip install -r requirements.txt

Lets see what is inside the requirements.txt:

Terminal window
pulumi>=3.0.0,<4.0.0
pulumi-aws>=5.0.0,<6.0.0
pulumi-command
Jinja2
pycryptodome

There are several packages we installed in addition to the minimum requirements from Pulumi:

  • pulumi-command: Used to execute commands on the remote EC2 server.
  • Jinja2: Used to generate dynamic templates.
  • pycryptodome: Used to hash our template files so that Pulumi automatically.

Next, we need to create a new public and private key that we will use to SSH into our EC2 instance. I made a Python script that can quickly generate a unique public/private key pair. Create a unique pair by running:

Terminal window
python create_keypair.py
chmod 400 key.pem
Expand to see create_keypair.py
Terminal window
import os


from Crypto.PublicKey import RSA




def main():
    """Create a new keypair."""
    key = RSA.generate(2048)
    private_key = key.exportKey("PEM")
    public_key = key.publickey().exportKey("OpenSSH")


    with open("key.pem", "w") as f:
        f.write(private_key.decode())


    with open("key.pub", "w") as f:
        f.write(public_key.decode())


    return public_key, private_key




if __name__ == '__main__':
    main()

Lastly, create a new pulumi stack. I will name my stack dev, but you can call it whatever you like.

Terminal window
pulumi stack init dev

pulumi up

You are now ready to spin up your infrastructure. Just run:

Terminal window
pulumi up

_main_.py deep dive

Let us take a closer look at main.py and see what is happening.

Expand to see __main__.py
Terminal window
"""An AWS Python Pulumi program"""


import hashlib
from pathlib import Path


import jinja2
import pulumi
from pulumi_aws import ec2
from pulumi_command import remote


# ------------------------------------------------------------------------------
# Helper functions
# ------------------------------------------------------------------------------


def create_template(path: str) -> jinja2.Template:
    with open(path, 'r') as f:
        template = jinja2.Template(f.read())
    return template




def hash_file(path: str) -> pulumi.Output:
    with open(path, mode="r") as f:
        text = f.read()
    hash_str = hashlib.sha224(bytes(text, encoding='utf-8')).hexdigest()
    return pulumi.Output.concat(hash_str)




# ------------------------------------------------------------------------------
# Infrastructure
# ------------------------------------------------------------------------------


def main():
    # Customize these with your own tags!
    tags = {
        "rs:environment": "development",
        "rs:owner": "name@email.com",
        "rs:project": "solutions",
    }


    key_pair = ec2.KeyPair(
        "ec2 key pair",
        key_name=f"keypair-for-pulumi",
        public_key=Path("key.pub").read_text(),
        tags=tags | {"Name": "keypair-for-pulumi"},
    )


    # Make security groups
    security_group = ec2.SecurityGroup(
        "security group",
        description="Security group for my blog post",
        ingress=[{"protocol": "TCP", "from_port": 22, "to_port": 22, 'cidr_blocks': ['0.0.0.0/0'], "description": "SSH"}],
        egress=[{"protocol": "All", "from_port": -1, "to_port": -1, 'cidr_blocks': ['0.0.0.0/0'], "description": "Allow all outbound traffic"}],
        tags=tags
    )


    # Create a new ec2 instance
    server = ec2.Instance(
        "EC2 instance",
        instance_type="t3.medium",
        vpc_security_group_ids=[security_group.id],
        ami="ami-0fb653ca2d3203ac1",  # Ubuntu Server 20.04 LTS (HVM), SSD Volume Type
        tags=tags,
        key_name=key_pair.key_name
    )


    pulumi.export('server_public_dns', server.public_dns)


    # Create a connection that will be used to SSH into the ec2 instance
    connection = remote.ConnectionArgs(
        host=server.public_dns,
        user="ubuntu",
        private_key=Path("key.pem").read_text()
    )


    # Render a template on the ec2 instance
    local_file_path = "templates/template.env"
    remote_file_path = "~/.env"


    command_render_template = remote.Command(
        "copy .env",
        create=pulumi.Output.concat(
            'echo "',
            pulumi.Output.all(
                public_ip=server.public_ip,
                availability_zone=server.availability_zone,
                cpu_core_count=server.cpu_core_count
            ).apply(
                lambda args: create_template(local_file_path).render(
                    ip_address=args['public_ip'],
                    availability_zone=args['availability_zone'],
                    cpu_core_count=args['cpu_core_count']
                )
            ),
            f'" > {remote_file_path}'
        ),
        connection=connection,
        opts=pulumi.ResourceOptions(depends_on=[server]),
        triggers=[hash_file(local_file_path)]
    )




main()

Below we will walk through some of the critical parts of the code.

create_template(path: str)

import jinja2


def create_template(path: str) -> jinja2.Template:
    with open(path, 'r') as f:
        template = jinja2.Template(f.read())
    return template

This is a helper function so we can quickly create a jinja2.Template object. We wrap this logic in a function so we can easily call it later inside of pulumi_command.remote.Command.

hash_file(path: str)

import hashlib


def hash_file(path: str) -> pulumi.Output:
    with open(path, mode="r") as f:
        text = f.read()
    hash_str = hashlib.sha224(bytes(text, encoding='utf-8')).hexdigest()
    return pulumi.Output.concat(hash_str)

This function will create a unique hash of our template files. Note that we return a pulumi.Output object.

command_render_template

This code chunk is really the core of what we are doing:

# Render a template on the ec2 instance
local_file_path = "templates/template.env"
remote_file_path = "~/.env"


command_render_template = remote.Command(
    "copy .env",
    create=pulumi.Output.concat(
        'echo "',
        pulumi.Output.all(
            public_ip=server.public_ip,
            availability_zone=server.availability_zone,
            cpu_core_count=server.cpu_core_count
        ).apply(
            lambda args: create_template(local_file_path).render(
                ip_address=args['public_ip'],
                availability_zone=args['availability_zone'],
                cpu_core_count=args['cpu_core_count']
            )
        ),
        f'" > {remote_file_path}'
    ),
    connection=connection,
    opts=pulumi.ResourceOptions(depends_on=[server]),
    triggers=[hash_file(local_file_path)]
)

The basic approach is to run an echo command on the remote server that writes our rendered template to a file.

Terminal window
echo "TEMPLATE CONTENTS" > .env

The tricky part is getting our rendered template into "TEMPLATE CONTENTS". To do this, we need to use pulumi.Output.concat. This function allows you to use the output of other pulumi.Output objects. Notice that we pass in all the values we want to access in our template.

pulumi.Output.all(
    public_ip=server.public_ip,
    availability_zone=server.availability_zone,
    cpu_core_count=server.cpu_core_count
)

Then, we can use pulumi.Output.concat().apply to pass these values into another function. Here, we will create a jinja2.Template object with our create_template function and dynamically render the values.

pulumi.Output.all(
    public_ip=server.public_ip,
    availability_zone=server.availability_zone,
    cpu_core_count=server.cpu_core_count
).apply(
    lambda args: create_template(local_file_path).render(
        ip_address=args['public_ip'],
        availability_zone=args['availability_zone'],
        cpu_core_count=args['cpu_core_count']
    )
)

Note that the keyword arguments inside create_template(local_file_path).render should match the values in your template.

Terminal window
IP_ADDRESS={{ip_address}}
CPU_CORE_COUNT={{cpu_core_count}}
AVAILABILITY_ZONE={{availability_zone}}

See the results

Now that you have run pulumi up and created a dynamically rendered template let us check out the results. SSH into your EC2 instance:

Terminal window
ssh -i key.pem -o StrictHostKeyChecking=no ubuntu@$(pulumi stack output server_public_dns)

Once inside your EC2 instance, inspect the template that we generated.

Terminal window
cat .env
IP_ADDRESS=3.21.35.72
CPU_CORE_COUNT=1
AVAILABILITY_ZONE=us-east-2c

Wrap up

With jinja2 and Pulumi we are now able to turn this:

Terminal window
IP_ADDRESS={{ip_address}}
CPU_CORE_COUNT={{cpu_core_count}}
AVAILABILITY_ZONE={{availability_zone}}

Into this!

Terminal window
IP_ADDRESS=3.21.35.72
CPU_CORE_COUNT=1
AVAILABILITY_ZONE=us-east-2c
Previous Post
How to learn Python for Data Science
Next Post
The Best Way to Install Jupyter Lab (it's Pipx)