Creating a Shared Python Module

Overview

This guide will cover how to configure a Python module that can be shared within the Python pipelines. This enables sharing classes, functions, and variables between the pipelines, reducing redundant code and speeding up the development process.

Folder and pom.xml Setup

This section will cover structuring the folders and pom.xml files that are needed for the Python module to build successfully.

  1. Create a new folder named python-shared in the <project-name> directory. This folder will contain the Python modules that will be used in your project.

  2. In the <project-name>/pom.xml file, under the <modules> tag, add:

<module>python-shared<module>
The <module> tag for python-shared should be added before the <module> tag for <project-name>-pipelines to ensure that the Python modules are generated before being utilized in the pipelines.
As of Habushu version 2.6.0, maven build dependency is established through the ordering of the <module> tags, which could potentially cause errors when building in parallel. This will be addressed in future Habushu versions.

  1. Create a pom.xml in the <project-name>/python-shared directory and include project and parent info and set the <packaging> tag to pom. The pom.xml should resemble:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>[groupId]</groupId>
    <artifactId>[artifactId]</artifactId>
    <version>[version]</version>
  </parent>

  <artifactId>python-shared</artifactId>

  <name>[name]</name>

  <packaging>pom</packaging>
</project>
Items in the form of [value] need to be replaced with the required info of the corresponding tag.

  1. You are now ready to create the Poetry folder structure! In the terminal, change directories into the python-shared folder and run the following command to create the Poetry structure and files:

poetry new <module-name> --src

  1. In <project-name>/python-shared/pom.xml file, add the <modules> tag for the new module that was created:

<modules>
  <module>[module-name]<module>
<modules>

  1. Under <project-name>/python-shared/<module-name> directory, create a pom.xml. Use the previous pom.xml template for the <project-name>/python-shared/pom.xml, setting the packaging to habushu and artifactId to python-shared.

  2. Add the habushu plugin to <project-name>/python-shared/<module-name>/pom.xml in the <project> tag:

<build>
  <plugins>
    <plugin>
      <groupId>org.technologybrewery.habushu</groupId>
      <artifactId>habushu-maven-plugin</artifactId>
      <configuration>
          <!-- if using a private pypi repo: -->
          <pypiRepoUrl>${pypi.repo.url}</pypiRepoUrl>
      </configuration>
    </plugin>
  </plugins>
</build>

  1. Within the <project-name>/python-shared/<module-name>/pyproject.toml, change the Python version to match a pipeline’s Python version and include the black, behave, and nose dependencies. The <project-name>/python-shared/<module-name>/pyproject.toml file should resemble:

[tool.poetry]
name = "module-name"
version = "0.1.0"
description = ""
authors = ["John Doe <doe_john@bah.com>"]
readme = "README.md"
packages = [{ include = "module_name", from = "src" }]

[tool.poetry.dependencies]
python = "#.#.#"

[tool.poetry.group.dev.dependencies]
black = "#.#.#"
behave = "#.#.#"
nose = "#.#.#"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
The Python version and black, behave, and nose versions should match the corresponding versions listed in the target Python pipelines' pyproject.toml files.

Pipeline Setup

This section will cover the steps for incorporating the Python module into a pipeline.

  1. To add the Python module to the pipeline, in the terminal, navigate to the <project-name>-pipelines/<pipeline-name>/<step-name> and run in terminal:

poetry add ../../../python-shared/<module-name>/ --group python-shared --editable
If the pipeline is a Python data-flow pipeline, then navigate to <project-name>-pipelines/<step-name> and run the command: 
poetry add ../../python-shared/<module-name>/ --group python-shared --editable

  1. In the <project-name>/<pipeline-name>/<step-name>/pom.xml, add to the habushu-maven-plugin <configuration> tag:

<withGroups>
  <withGroup>python-shared</withGroup>
</withGroups>

Dockerfile Setup

This section will cover the steps for incorporating the Python module into the Docker image.

  1. Under the <project-name>-docker/<step-name>-docker/pom.xml, add the following <execution> tag to move the wheel and requirements.txt files from python-shared/<module-name>/dist folder to target folder:

<build>
  <plugins>
    <plugin>
      <artifactId>maven-resources-plugin</artifactId>
      <executions>
        ...
        <execution>
          <id>copy-[module-name]</id>
          <phase>prepare-package</phase>
          <goals>
            <goal>copy-resources</goal>
          </goals>
          <configuration>
            <outputDirectory>${project.build.directory}/[module-name]</outputDirectory>
              <resources>
                <resource>
                <directory>${project.parent.parent.basedir}/python-shared/[module-name]</directory>
                <filtering>false</filtering>
                <includes>
                  <include>dist/requirements.txt</include>
                  <include>dist/*-${version.habushu.dist.artifact}-*.whl</include>
                </includes>
              </resource>
            </resources>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

  1. In the <project-name>-docker/<step-name>-docker/src/main/resources/Dockerfile, add a COPY command to copy the requirements.txt file to the Docker image and a RUN command to install the dependencies:

COPY target/<module-name>/dist/requirements.txt /installation/<module-name>/
RUN python3 -m pip install -r /installation/<module-name>/requirements.txt

  1. In the <project-name>-docker/<step-name>-docker/src/main/resources/Dockerfile, add a COPY command to copy the wheel file to the Docker image and a RUN command to install the Python module dependency:

COPY target/<module-name>/dist /modules/<module-name>
RUN cd /modules/<module-name>; for x in *.whl; do python3 -m pip install $x --no-cache-dir --no-deps; done