Skip to main content

· 4 min read

uv is "an extremely fast Python package installer and resolver, written in Rust" from https://astral.sh. Recently, I have started using uv in my day-to-day Python workflows. After a few weeks of usage I am sold! uv will be my go-to package manager for Python projects moving forward. It is MUCH faster than pip and I really like the new workflow it provides me with uv pip compile and uv pip sync.

TL/DR​

I am now using uv instead of pip for most of my projects. My workflow looks like this:

alias uvinit='uv venv && source .venv/bin/activate'
alias uvsync='uv pip compile requirements.in --quiet --output-file requirements.txt && uv pip sync requirements.txt'

uvinit
echo "pandas" > requirements.in
uvsync

· 2 min read

I usually create a requirements.txt when I start a new Python project. I will do something like this:

# Create a new project
mkdir new-project
cd new-project

# Create requirements.txt
tee -a requirements.txt <<EOF
pandas
pyarrow
EOF

# Create a virtual environment and install dependencies
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
python -m pip install -r requirements.txt

This lets me get a new project up and running quickly. I have documented which packages I am using, which is good, but I have not documented which version of the package I am using. Since it is a new project, I want to use the latest version of every package. I could search PyPI for each package and find the latest version, but that takes a lot of time. Instead, I only document the package name in my initial requirements.txt. When I am ready to pin the versions, I use this bash script to check the version I have installed for each dependency listed in requirements.txt:

python -m pip freeze | grep -E $(cat requirements.txt | sed ':a;N;$!ba;s/\n/==|/g')

The result will be something like this:

pandas==2.1.4
pyarrow==14.0.2

· 2 min read

At the start of the month I wrote a blog post about how to use 1Password for managing all of your secrets on the command line: How to use 1Password for Secrets in ~/.bashrc or ~/.zshrc. After 1 month of using this approach in the wild, I have a few updates.

My primary problem was that it was very annoying to authenticate every time I opened a new terminal session. But... liked the idea of using 1Password to store all of my secrets, and not copying and pasting secrets into my startup scripts.

After some experimentation, I found a compromise that works for me.

export DOTFILES_DIR="${HOME}/.dotfiles"

# If zhs/secrets-out.zsh does not exist, create it.
secrets_out_path="${DOTFILES_DIR}/zsh/secrets-out.zsh"

if [ ! -f "$secrets_out_path" ]; then
    echo "Creating ${secrets_out_path}..."
    op --account "my.1password.com" inject --in-file "${DOTFILES_DIR}/zsh/secrets-in.zsh"  --out-file "${DOTFILES_DIR}/zsh/secrets-out.zsh"
fi

# Check to see that if after removing everything to the right of `=` in
# zsh/secrets-in.zsh and zsh/secrets-out.zsh, the files are the same. If they
# are the same do nothing. If the are different create an updated version of
# zsh/secrets-out.zsh.
secrets_in_no_values=$(cat "${DOTFILES_DIR}/zsh/secrets-in.zsh" | sed 's/=.*//' | base64)
secrets_out_no_values=$(cat "${DOTFILES_DIR}/zsh/secrets-out.zsh" | sed 's/=.*//' | base64)

if [ ! "$secrets_in_no_values" = "$secrets_out_no_values" ]; then
    echo "Secrets have changed... Updating ${secrets_out_path}"
    rm "${DOTFILES_DIR}/zsh/secrets-out.zsh"
    op --account "my.1password.com" inject --in-file "${DOTFILES_DIR}/zsh/secrets-in.zsh"  --out-file "${DOTFILES_DIR}/zsh/secrets-out.zsh"
fi

· 2 min read
Update

A new blog post has been written about this topic! Check out How to use 1Password for Secrets in ~/.bashrc or ~/.zshrc (UPDATE).

1Password is a password manager. Over the years, I have tried LastPass, BitWarden, and 1Password. Out of the three, 1Password has been my favourite. The Mac app, browser extension, and IOS app are well-polished. One of my favourite parts about 1Password is the ability to access passwords using the 1Password CLI.

Using op inject for secrets​

I recently discovered a pattern to use the 1Password CLI to store all of my secrets in my dotfiles:

# ~/.zshrc
op  inject --in-file "${HOME}/.dotfiles/secrets.zsh" | while read -r line; do
  eval "$line"
done
# ~/.dotfiles/secrets.zsh
export NOTION_API_KEY="op://private/notion.so/api-token"
export TEST_PYPI_TOKEN="op://private/test.pypi.org/token"

· 2 min read

Reticulate is an R library that lets you execute Python code from within R. Recently, I attempted to use reticulate to access numpy from Python. Numpy was installed, but I kept getting the following message:

> reticulate::py_config()
python:         /usr/home/sam.edwardes/rstudio-demos/applications/shiny-for-r-with-reticulate/.venv/bin/python3
libpython:      /opt/python/3.10.11/lib/libpython3.10.so
pythonhome:     /usr/home/sam.edwardes/rstudio-demos/applications/shiny-for-r-with-reticulate/.venv:/usr/home/sam.edwardes/rstudio-demos/applications/shiny-for-r-with-reticulate/.venv
version:        3.10.11 (main, Jun  4 2023, 22:34:21) [GCC 11.3.0]
numpy:           [NOT FOUND]

· One min read

VS Code has a built in feature that allows you to send code directly from the editor to the terminal. By default, it is not assigned to any shortcut. You can assign it to a shortcut by adding the following to your keybindings.json file.

  • Open the command pallet using cmd + shift + p.
  • Type Preferences: Open Keyboard Shortcuts (JSON)
  • Then edit the keybindings.json file:
[
    // Send selected bash code to terminal
    {
        "key": "shift+enter",
        "command": "workbench.action.terminal.runSelectedText",
        "when": "editorTextFocus && !findInputFocussed && !replaceInputFocussed && editorLangId == 'shellscript'"
    }
]

The when key ensures that this shortcut is only active when you are working on a Shell Script file. Here is an example of the shortcut in action.

Gif of the sending shell script from the editor to the terminal.

· One min read

I always forget these commands... so here is my list of helpful commands to get information about your server.

  • Get your external IP address.
curl ifconfig.io
  • Get your internal IP address. For example, this can be helpful if you are working with several EC2 instances that need to communicate with each other. You can use this instead of the external IP address for node to node communication. This has the benefit of being static even if you stop and start your EC2 instance.
hostname -i
  • Get information about your operating system.
lsb_release -a
  • Get information about your current operating system when in a Docker image.
cat  /etc/os-release

· 2 min read

Over the past year I have dedicated a lot of time to learning Kubernetes. Here are the resources I have found most helpful!

Kubernetes Tutorial for Beginners (FULL COURSE in 4 Hours)

I would recommend starting with this video. After taking this four hour crash course you will have a basic understanding of how Kubernetes works, and you will have a Kubernetes cluster running on your computer you can use for further learning.

Kubernetes Fundamentals (LFS258)

This course is very comprehensive and will prepare you for the CKA Exam. I found this course to be my favourite resource. There are only two downsides:

  • It is expensive. The course alone costs $299 USD. The course and the CKA Exam cost $595 USD. (As of 2023-04-15). However, if you can afford it, I think putting a little bit of skin in the game is a good motivator to learn and take the course seriously.
  • The course also takes some time to complete. The course took me about 3 months to complete spending a few hours each weekend.

Kubernetes: Up and Running

This book is a good reference and will be nice resource to have in the future. However, it is also a bit pricey.

· One min read

One common task in Kubernetes is to exec into a pod to run commands. The typical way to do this is as follows:

POD_NAME="test-pod-1234asdfas"
kubectl exec -it pod/$POD_NAME -- /bin/bash

This pattern typically works fine. However, if you are frequently creating and deleting new pods, the name will constantly change. This means you will have to retype the command with the new pod name instead of just using the up arrow to select a recent command. A more consistent way is to use the deployment name instead:

DEPLOYMENT_NAME="test-pod"
kubectl exec -it deployment/$DEPLOYMENT_NAME -- /bin/bash

· One min read

I often want to create new users in Linux for testing. Creating new users without interaction can be challenging to automate because the passwd command provides no way for you to pass in a plain text password. It will prompt you for a password which is OK for interactive sessions but not suitable for automation (e.g. creating new users in Pulumi).

NEW_USER_NAME=sam
useradd --create-home --home-dir /home/$NEW_USER_NAME -s /bin/bash $NEW_USER_NAME
passwd $NEW_USER_NAME
# New password: 
# Retype new password: 
# passwd: password updated successfully

The solution I have found is to pipe the password into the passwd command like this:

NEW_USER_NAME=sam
NEW_USER_PASSWORD=password
useradd --create-home --home-dir /home/$NEW_USER_NAME -s /bin/bash $NEW_USER_NAME
echo -e "${NEW_USER_PASSWORD}\n${NEW_USER_PASSWORD}" | passwd $NEW_USER_NAME

This trick allows you to create new users and set their passwords without interaction!

· 5 min read

Over the last few months, I have spent a lot of time working on AWS. I often need to spin up EC2 instances, databases, or other assets for testing. Doing this by hand can become burdensome. You need to click through the AWS CLI and keep track of everything you have created. This sounds like a perfect use case for infrastructure as code. Enter Pulumi!

· 18 min read

I have recently been exploring how to use FastAPI to build web apps. Check out this recent blog post How to create a FastAPI Web App with MongoDB and Beanie. One area I have found difficult is configuring authentication for web apps in FastAPI. The FastAPI security docs are really good, however they focus only on securing API endpoints, and not how to implement security for a web app. This blog post will demonstrate how to stand up a fully functional FastAPI web app with authentication.

· 17 min read

I always like to experiment with the hottest new frameworks and libraries in Python. A few technologies that I have found interesting lately are:

  • FastAPI - A framework for building APIs based on pydantic.
  • MongoDB - A NoSQL database.
  • Beanie - An "object document mapper" (ODM) that allows you to model your MongoDB using python.

This blog post provides a working example of a webapp that uses all three technologies 🎉!

· 4 min read

As a data scientist one of the most common questions I get from colleagues and clients is how to get started on learning R. There is a plethora of great options out there today. Some of which are paid, and some of which are free.

· 5 min read

One of the funnest (and most frustrating) parts of data science is the vast array of tools available to us. It can be overwhelming where to start. Every now and then I like to completely wipe my computer clean, and then reinstall everything from scratch. This helps clean up my computer, and make sure everything is running smoothly.

· 9 min read

When a UFC match ends with a knock-out or submission there is never any doubt who the better fighter was. But only 54% of fights end with a knock-out or submission. The other 45% of fights go to the judges score cards.

· 10 min read

With a few spare days between finishing work and starting a masters program at UBC I was keen to experience some adventure before going back to school. Since moving to Vancouver 1 year ago, I had been imagining some of the cycle touring trips I may do. BC's West coast holds so many potential options.

· 16 min read

During my first year of university my friend Cory and I began to plan a cycling trip across Canada. I don’t exactly remember where the idea came from, but it quickly developed from a dream to something that could actually happen. We decided to raise money for the B.C. Cancer Foundation, as both of our families have been effected by cancer. We raised over $15,000 from friends, families, and local businesses. The experiences of fundraising, and the completion of the actual cycling trip have been two of the most rewarding experiences of my life.

· 6 min read

While studying abroad in Istanbul I took my semester break to travel to Portugal and cycle from Lisbon to Faro. The goal of the trip was to really enjoy the nature. I wanted it to be slow paced and relaxing. I wanted to take many stops, and enjoy all the sights and sounds. When I cycled across Canada there was no stopping to smell the roses. We were really determined to get across the country in 50 days, so we missed a lot of cool sights along the way for the sake of time.

· 12 min read

After 4 years of school at Queen's University I was at last done. I wrote my final exam, packed my things, and the next morning departed on a bike ride from Kingston to Toronto with my friend Sam I have known since our first year together at Queen's. The trip was more or less a spur of the moment adventure. We planned the idea several weeks before leaving, and only came up with a rough estimate of where we would be each day.

· 6 min read

The summer of 2013 I and my friends Cory Babiak, Taylor Jenkins, and Jose Pauig ran across PEI. Jose followed us on a bike for support; Cory Taylor and myself all ran. We raised over $7000 for Project Jumpstart, a charity that helps Canadian Kids who cannot afford to play sports get in the game. It was great raising money for Project Jumpstart, as it is something we all felt very strong about. The run itself was over 6 marathons in six days. It ended up being an extremely challenging task, much harder on the body than cycling. By the end we were running and walking the trail, and moving at a snails pace, but we did finish in our goal of 280 km in six days!