#051 - Simplify your Python Project Setup | Cookiecutter

Automate Repetitive Tasks and Standardize Your Workflow with Custom Templates

Nov 19, 2024

Today’s article is about a very handy Python library I use quite frequently called Cookiecutter.

One of the most significant barriers to Python adoption, often cited by skeptics and beginners alike, is the perceived complexity of project initialization, managing virtual environments and handling dependencies – which can lead to frustration and abandonment before the real work even begins.

No more. This article intends to address this issue.

Cookiecutter offers a streamlined solution to standardize, simplify, and bring consistency to your project setups, effectively addressing these common pain points.

Why Use Cookiecutter?

Cookiecutter is a command-line utility that creates projects from templates. It’s widely used across various domains, from data science to web development, but it can be tailored for any kind of project. If you find yourself performing frequent repetitive tasks, Cookiecutter is a great way to streamline your workflow. It’s very flexible and lightweight.

Below, I provide a script for generating a simple Jupyter project. We'll walk through it step-by-step to understand how it works and how you can customize it for your needs. At first glance, I know it’s a mouthful but it’s worth investigating because the principles broadly apply to so many different workflows.

Just scan the code below and then bear with me as we walk through it.

https://github.com/joreilly86/create_notebook_project

What This Script Does

This script automates the creation of a Jupyter notebook-based Python project with the following features:

Creates a Template Structure: Sets up directories for data (data/) and source code (src/), along with a blank Jupyter notebook (notebook_01.ipynb) and a README.md file.
Initializes the Project Environment: Uses uv (a tool for managing Python virtual environments) to set up a virtual environment and installs ipykernel for Jupyter notebook integration.

Setting Up the Template Structure

Defining a Project Template

First, the script defines a template structure using a Python dictionary. This is part of the cookiecutter process. The structure includes folders for storing data and source code, a blank Jupyter notebook, and a README file:

TEMPLATE_STRUCTURE: Dict[str, Union[str, dict]] = {
    "cookiecutter.json": json.dumps(
        {
            "project_name": "default_project_name",
            "author_name": "Your Name",
            "description": "A Jupyter notebook-based project.",
        },
        indent=4,
    ),
    "{{cookiecutter.project_name}}": {
        "data": {},  # Data directory
        "src": {},   # Source code directory
        "notebook_01.ipynb": json.dumps(
            {"cells": [], "metadata": {}, "nbformat": 4, "nbformat_minor": 5}, indent=4
        ),  # Blank Jupyter notebook file
        "README.md": (
            "# {{cookiecutter.project_name}}\\\\n\\\\n"
            "{{cookiecutter.description}}\\\\n\\\\n"
            "## Author\\\\n"
            "{{cookiecutter.author_name}}"
        ),
    },
}

Explanation:

cookiecutter.json: This configuration file contains default values that can be customized when generating a new project. It includes placeholders for the project name, author name, and project description.
Project Directory: The directory named after the project ({{cookiecutter.project_name}}) contains:
- data/: For datasets, PDFs, CSVs, images, text, etc.
- src/: For source code files and modules.
- notebook_01.ipynb: A completely blank Jupyter notebook ready for your content.
- README.md: A README file populated with your project name, description, and author.

Automating Template Creation

The create_template_structure function recursively traverses the TEMPLATE_STRUCTURE dictionary, creating directories and files as specified:

def create_template_structure(
    base_path: Path, structure: Dict[str, Union[str, dict]]
) -> None:
    """
    Recursively creates the project structure based on the template dictionary.
    """
    for name, content in structure.items():
        current_path = base_path / name
        if isinstance(content, dict):
            current_path.mkdir(parents=True, exist_ok=True)
            create_template_structure(current_path, content)
        else:
            current_path.parent.mkdir(parents=True, exist_ok=True)
            current_path.write_text(content, encoding="utf-8")

Key Points:

Recursion: The function handles nested directories by calling itself when it encounters a dictionary.
File Creation: Files are created with the specified content. For example, the notebook_01.ipynb is a blank Jupyter notebook.

Running Commands with Subprocess

To handle system-level commands, such as initializing the project or installing dependencies, the script uses the subprocess.run() method. This automates setup tasks like creating virtual environments or installing libraries.

def run_command(command: list, cwd: Path = None) -> None:
    """
    Runs shell commands using subprocess, with error handling.
    """
    logging.info(f"Running command: {' '.join(command)}")
    try:
        subprocess.run(
            command, check=True, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
        )
    except subprocess.CalledProcessError as e:
        logging.error(
            f"Error running command {' '.join(command)}: {e.stderr.decode().strip()}"
        )
        sys.exit(1)

Functionality:

Logging: Logs the command being run for transparency.
Error Handling: If a command fails, the script logs the error and exits gracefully.

Using `uv` for Environment Management

The script utilizes uv, a tool for managing Python virtual environments and dependencies. uv has become my preferred choice, replacing Poetry for setting up my virtual environments. I’ve written about uv before, it’s excellent, check that article here:

#044 - Python Environments, Again | uv: A Guide to Python Package Management

James O'Reilly

Oct 8

Read full story

def initialize_project_environment(project_path: Path) -> None:
    """
    Initializes a Python project environment using `uv` and installs `ipykernel`.
    """
    if not shutil.which("uv"):
        logging.error("'uv' is not installed or not in the system PATH.")
        sys.exit(1)

    run_command(["uv", "init"], cwd=project_path)
    run_command(["uv", "add", "ipykernel"], cwd=project_path)

Steps:

Check Installation: Verifies that uv is installed and accessible.
Initialize Environment: Runs uv init to set up the virtual environment.
Add ipykernel: Installs ipykernel, allowing Jupyter notebooks to operate within this environment.

Putting It All Together

The main() function orchestrates the entire process:

def main():
    """
    Main function to handle the creation of the project template and initialize the project.
    """
    args = parse_arguments()

    # Define the base path where all templates are stored
    templates_base_dir = Path.cwd() / "cookiecutter-project-templates"
    templates_base_dir.mkdir(parents=True, exist_ok=True)

    # Prepare the template directory
    template_path = prepare_template_directory(templates_base_dir, args.template_folder)

    # Use Cookiecutter to create a new project with the provided context
    try:
        project_dir = Path(
            cookiecutter(
                str(template_path),
                no_input=True,
                extra_context={
                    "project_name": args.name,
                    "author_name": args.author,
                    "description": args.description,
                },
            )
        )
        logging.info(f"Project created at: {project_dir}")

        # Initialize the project environment
        initialize_project_environment(project_dir)
        logging.info("Project environment initialized successfully.")

    except Exception as e:
        logging.error(f"An error occurred: {e}")
        sys.exit(1)

Workflow:

Parse Arguments: Handles command-line arguments for project customization.
Define Template Directory: Sets up the cookiecutter-project-templates directory in the current working directory. This centralizes all your project templates, making them easy to manage and locate.
Prepare Template Directory: Ensures that the specified template folder exists within cookiecutter-project-templates. If it doesn't, the script creates it using the defined TEMPLATE_STRUCTURE.
Generate Project: Uses Cookiecutter with the provided context (project_name, author_name, description) to generate a new project based on the template.
Initialize Environment: Sets up the virtual environment and installs necessary dependencies using uv.

Clarification:

cookiecutter-project-templates Directory: This directory is created in the current working directory from which you run the script. All your project templates will reside here, allowing you to organize multiple templates (e.g., jupyter-basic, jupyter-ml, etc.) in one centralized location. This is handy. You can create templates for specific software outputs or report generation etc. Again, I’ll talk about this in the future.

Usage Instructions

To use the script, follow these steps:

Ensure Dependencies Are Installed
- Python: Make sure you have Python installed.
- Cookiecutter: Install Cookiecutter if you haven't already
- UV: Ensure that the uv tool is installed. See https://docs.astral.sh/uv/

pip install cookiecutter uv

Save the Script
- Save the script as create_notebook_project.py in your desired directory. I use a directory called /dev for all my code based projects. When I’m ready I move them to Git-lab or our company servers.
Run the Script
- Open your terminal or command prompt.
- Navigate to the directory containing create_notebook_project.py.
- Execute the script with the required arguments. The n/--name argument is mandatory, while others are optional:

python create_notebook_project.py -n "bridge_abutment_results-01" -a "James O'Reilly" -d "Exploring outputs from SAP2000 FE model."

- - Parameters:
    - n, -name: (Required) Name of the new project.
    - a, -author: (Optional) Author's name. Defaults to "Your Name".
    - d, -description: (Optional) Project description. Defaults to "A Jupyter notebook-based project.".
    - t, -template-folder: (Optional) Template folder name to use. Defaults to "jupyter-basic".
Interactive Prompt for Project Name
- If you omit the n/--name argument, the script will prompt you to enter the project name interactively. This is usually how I run it, nice and simple.

python create_notebook_project.py

Enter the project name: MyJupyterProject
[INFO] Creating new template at: /path/to/cookiecutter-project-templates/jupyter-basic
[INFO] Running command: uv init
[INFO] Running command: uv add ipykernel
[INFO] Project created at: /path/to/MyJupyterProject
[INFO] Project environment initialized successfully.

Directory Structure After Project Creation

After running the script, your project directory (e.g., name it MyJupyterProject) will have the following structure:

MyJupyterProject/
├── data/
├── src/
├── .gitignore
├── .python-version
├── .venv
├── hello.py
├── notebook_01.ipynb
├── pyproject.toml
├── README.md
└── uv.lock

data/: Directory for storing project data files
src/: Directory for source code files
.gitignore: Specifies intentionally untracked files to ignore
.python-version: Specifies the Python version for the project
.venv: uv creates your virtual environment with ipykernel preinstalled.
hello.py: Example Python script
notebook_01.ipynb: Jupyter notebook for analysis and experimentation
pyproject.toml: Project configuration file (dependencies, build settings, etc.)
README.md: Project documentation and overview (you are here!)
uv.lock: Lock file for uv package manager, specifying exact versions of dependencies

Why Use This Setup?

By using Cookiecutter with tools like uv, you automate the entire project setup process. This is particularly useful for those who work on similar types of projects repeatedly.

Benefits:

Consistency: All your projects will follow the same structure, making it easier for teams to collaborate. I’m working on this as part of a larger standardization effort within our company.
Efficiency: You can get a new project up and running in seconds, with all the necessary directories and files in place.
Environment Management: uv makes it easy to manage dependencies and virtual environments, so you don’t have to worry about conflicting libraries.
Customization: Easily extend the script and template to include additional features or dependencies as your projects evolve. I have templates for plotting, calculations and software specific applications.

Conclusion

Cookiecutter is a valuable tool in my workflow, significantly reducing the friction in project initiation, especially with Jupyter which is probably my most frequent use case.

Often I need to spin something up quickly to test an idea or a concept and this has been a great way to do it. I just run…

python create_notebook_project.py

While the script may appear complex at first glance, its utility is straightforward: one command, and a fully-structured Jupyter project is ready for use. There are many ways to skin this cat but this is how I do it, it’s a simple command line operation to set up a new project.

The primary benefit is less administrative overhead, meaning more time for engineering.

For those interested in customizing templates, future posts will cover the process.

Feel free to further customize the script or the template structure to better fit your workflow. Whether you're standardizing setups for personal projects or implementing it across a team, this tool is extremely handy.

Try it out and let me know how you’re using it.

See you in the next one!

James 🌊