Adding a Pipeline

Once you have generated your new project, it is time to add a pipeline. Pipelines are the core of most projects and are responsible for major data delivery and machine learning tasks. The following content walks through the process of standing up a pipeline at a very high level.

Process

Step 1: Creating the pipeline model file

aiSSEMBLE™ uses Model Driven Architecture (MDA) to accelerate development. Pipeline models are JSON files used to drive the generation of multiple aspects in your project - including the pipeline code module and deployment modules.

  1. Create a new JSON file in your project’s pipeline-model directory.

    • Sample path: test-project/test-project-pipeline-models/src/main/resources/pipelines

  2. Create the model pipeline data within the newly added JSON file. For detailed options, see the pipeline metamodel documentation.

    1. Note: Ensure the name of the JSON file matches the "name" of the pipeline.

Example: Shown below is the SimpleDataDeliveryExample pipeline.

View SimpleDataDeliveryExample.json 
{
   "name":"SimpleDataDeliveryExample",
   "package":"com.boozallen.aissemble.documentation",
   "type":{
      "name":"data-flow",
      "implementation":"data-delivery-spark"
   },
   "steps":[
      {
         "name":"IngestData",
         "type":"synchronous",
         "dataProfiling":{
            "enabled":false
         }
      }
   ]
}

Step 2: Generating the pipeline code

After creating your model pipeline, execute the build to trigger the creation of the Maven modules that accompany it.

  1. Run the maven build to execute the MDA generation engine.

    1. ./mvnw clean install

  2. The MDA generator will take several build iterations to fully generate your project, and requires that you modify certain files to enable this generation. These Manual Actions are meant to guide you through that process, and will only need to be performed after changes to your pipeline model(s).

***********************************************************************
*** MANUAL ACTION NEEDED!                                           ***
***********************************************************************

  1. Re-run the build and address all manual actions until they have been resolved.