python_package_template

Feature	Option
OS	Linux/WSL 2
IDE	VS Code
Python	>=3.9
Env Management	docker
Testing	pytest
Build system	setuptools
Documentation	sphinx
CI	CircleCI

Provides a minimal template for a Python package repository.

Table of Contents

• Getting Started
• Usage
• Workflow
  • Setup
  • Test
  • Build
  • Document
  • Deploy
• License
• Authors

Other Python package templates

• Python Packaging Authority sample repository
• github/ionelmc/cookiecutter-pylibrary
• github/audreyfeldroy/cookiecutter-pypackage

Getting Started

Start by creating a fork of this repository and then cloning a local copy onto your machine.

This package uses VS Code Developer Containers to handle the virtual environment and developer workflow.

• If you are using Windows or Mac, you will need to install Docker Desktop
• If you are using Windows Subsystem for Linux you must enable the WSL Integration in Docker Desktop

Next, install the Remote Development Extension for VS Code.

Next time you open VS Code this extension will prompt you to reopen VS Code inside the developer container. IF you make changes to the Dockerfile or .devcontainer.json this operation can take some time as Docker will need to rebuild the image.

Once you have access to the code locally you are almost ready to start developing your Python package. However, to avoid any awkard name and/or path errors we suggest making the following changes:

• Rename the repository directory - Change the name of the repository directory where you have cloned the local copy into to something more descriptive and representative of your project. e.g. my_awesome_python_project not, python_package_template
• Change the name of the package directory - Change the name of the top-level Python package directory which is currently at src/python_package_template. Best practice is to name this directory after your package name, e.g. src/my_awesome_python_project.
• Change all instance of python_package_template In order for CI to run correctly you now need to change all mentions of python_package_template and replace this with your chosen package name. This includes modifying metadata (e.g. in setup.py and setup.cfg) and making sure all paths are correctly defined.
  • If you are using VS Code you can use the search functionality with Ctrl + Shift + F and search for "python_package_template" to get all hits in the directory and subdirectories.
  • TODO - Add shell command to automate this

A Note on Directory Structure

The directory structure for Python projects seems to be a contentious issue.

The blog posts by ionel and Jean-Paul Calderone outline the two main approaches that are prevalent in the Python community.

Usage

This repository provides the basic setup to enable a Setup > Test > Build > Document > Deploy workflow.

These operations are supported via a Makefile. To view available Make recipes you can type make in the command prompt and view the documentation.

Setup

make install

Test

make test

Build

make build-dist

Document

make build-docs

Deploy

# Test jobs locally using the CircleCI CLI
circleci local execute -c .circleci/config.yml --job setup-env

Workflow

The following sections are representative of an example developer workflow that uses WSL and VS Code as the IDE.

This is by no means prescriptive and is for guidance only.

Setup

This package uses venv to manage dependencies.

See make create-env and make install-python-deps to view the details.

Developer Tools

This repository uses the following developer tools:

• Docker - Running services in isolated environments
• VS Code Developer Containers - Writing and testing code in isolated environments

• pre-commit: pre-commit is provided as a developer dependency. To install pre-commit use pre-commit install and then run on all files using pre-commit run --all-files
• CircleCI: Continuous Integration is provided by CircleCI. The CircleCI CLI can be used to debug CI jobs locally.
• pylint: Linting is provided by pylint
• mypy: Static type checking is provided by mypy
• bandit: Code security is provided by bandit
• safety: safety is used to scan dependencies for vulnerabilities.

Example settings are provided in the .devcontainer.json.

Further details can be found in DEVELOPMENT.md

Test

Testing is provided by pytest and test files are defined in the tests directory.

Test discovery supports tests written using Behaviour Driven Development syntax, e.g. def should_pass_this_really_simply_test(arg_1, arg_2, expected)

See make test and make test-coverage to view the details and the pytest docs for more information.

Build

Packaging is provided by setuptools.

To build the source (.tar.gz) and distribution (.whl) archives

make build-dist

This will create the source (.tar.gz) and distribution (.whl) archives in the dist directory.

The package can be installed from these archives using

# Install from the source archive
python3 -m pip install path_to_the_source_archive.tar.gz

# Install from the distribution archive
python3 -m pip install path_to_the_distribution_archive.whl

The setuptools documentation provides details on the differences between the source and distribution archives.

Document

Documentation is provided by Sphinx.

To create the documentation for the Python package automatically we use the sphinx-autodoc extension

This allows Sphinx to automatically traverse the contents of the python package found at src/python_package_template and construct the ReStructuredText .rst file from the docstrings.

This repository uses napoleon extension to convert docstrings into ReStructuredText format. We have favoured the Google Style for writing docstrings over the numpy style but this is purely a matter of personal taste.

The HTML version of the documentation can be built using

make build-docs

This will create several html files in the docs/build directory. To view the docs in a web browser simply open the docs/build/index.html file.

Deploy

In this repository, "deploy" is a term used to loosely describe the act of hosting the code remotely on GitHub, running Continuous Integration (CI), monitoring code health and versioning the code.

For more information of CI/CD checkout the CI/CD Checklist and associated sources.

Continuous Integration

This repository uses CircleCI to provide continuous integration.

The following jobs are defined in the CircleCI config file:

• setup-env: Sets up the virtual environment for the CI jobs
• pre-commit: Runs pre-commit on all files in the repository
• linting: Runs linters on the code files
• security: Run DevSecOps tools on the code files and checks dependencies for vulnerabilities.
• unit-tests: Run the test framework
• build-dist: Builds the distribution files and stores the artifacts
• build-docs: Builds the documentation and stores the artifacts

We use the fan-out/fan-in workflow strategy to maximise concurrency.

The CI jobs setup-env and pre-commit can be tested locally using the the CircleCI CLI (requires Docker). The other jobs use caching to persist the dependency data between jobs, which is currently unsupported by the CircleCI CLI.

The following resources will assist first time installation for Docker and CircleCI CLI:

• Docker
  • Install Docker Engine on Ubuntu
  • Docker Desktop WSL 2 backend
• CircleCI CLI
  • Getting Started with the CircleCI CLI
  • Using the CircleCI Local CLI

Code Metrics

Code health is monitored using CodeCov and the settings can be found in the CodeCov configuration file.

Versioning

This repository uses versioneer for versioning and adheres to the Semantic Versioning style.

See setup.cfg, setup.py and versioneer.py for examples of how versioneer is incorporated into the Python package.

Specific instructions on how to set up versioneer can be found in INSTALL.md on the versioneer GitHub.

Security Warning

Using versioneer introduces some low severity security vulnerabilities in the codebase as reported by bandit, namely B101, B404 and B603.

In order for security checks to pass in CI we have explicitly ignored the warnings from bandit for the files versioneer.py and src/python_package_template/_version.py using the # nosec annotation. We have also updated the .bandit config file to ignore assert statements in these files.

We favour a targetted approach over a blanket skipping of these tests so that we can still expose security vulnerabilities related to these errors downstream during development. This decision is effectively a decision to trust the developers of versioneer that they are not acting maliciously. If you disagree with these changes feel free to revert the changes made in these files.

GitHub Actions

TODO - Read up on this for automatic versioning

License

MIT

Authors

• Christopher Szczyglowski

Inspired by the python_pack_and_doc repository by @chryswoods.