Create custom flows

Flows are designed to be easily usable, and once they have been created, managing them on a project is a simple task. Nevertheless, the creation of a new flow is relatively involved task. It involves several steps

  • Defining the flow.

    The means writing the actual code that performs the automation, as well as the formal definition of the interface to the flow so that other flows, and UI tools can interact with it.

  • Deciding where the flow will live.

    Will it be simply a private part of one project? Will it be in a shared repository on WVS? Will it be at some other location accessible by URL.

  • Registering the Flow.

    You can choose to register the flow in the WVS flow registry so that UI tools are aware of its existence (if desired).

  • Testing the flow on a live project.

    Since flows run in the cloud and operate on data comitted to the repo, any testing strategy has to take place in the cloud and not at the desktop. There are best practices on how to test

The flow definition

A flow definition consists of two files that have an identical name that differs only in the extension.

  • The flow definition file. (json)
  • The flow code. (yml)

The flow code file can include other code files, so technicall the flow definition could comprise many files, but at the very least there are always two files that have the same base name and differ only in the extension. For example: UnrealShippingBuild.json and UnrealShippingBuild.yml.

The definition file

The definition file is a json file that describes the attributes of the flow, as well as the input and outputs.

Example - Flow.json:

{
    "version" : 4,
    "name": "UE5 Build",
    "category": "build",
    "requireEngine": true,
    "engineName": "ue",
    "validEngineVersions" : [],
    "validEngineTargetPlatforms" : ["windows"],
    "jobName": ".build_ue_win",
    "inputs" :
    [
        {
            "inputName" : "PROJECT_DIR",
            "inputType" : "variable",
            "required" : true
        },
        {
            "inputName" : "PROJECT_FILE",
            "inputType" : "variable",
            "required" : true
        },
        {
            "inputName" : "TARGET",
            "inputType" : "variable",
            "required" : true
        } 
    ],
    "outputs" :
    [
        {
        "outputName" : "OUTPUT_BUILD_ARTIFACT",
        "outputType" : "artifact"
        }
    ]
}

Parameters:

Attribute Type Required Description
version integer yes The file format version for this file. Should be 4
name string yes The display name that will be used for this flow in menus/reports.
category string yes Defines the category of the flow. This determines where and how the flow will be presented in any UI, and how it will be treated by the flow scheduler. Currently valid values are: “build”, “deploy”, “other
jobName string yes Defines the entry point in the corresponding .yml file that defines the flow. The .yml file can contain multiple low level CI jobs and this field specifies which one should be invoked to execute the flow.
requireEngine bool yes Specifies whether this is a flow that uses some capability of a game engine and thus requires the engine being used by the project be present on the system that will be executing the flow. This can only be set to true if you are using one of the WVS-supported engines.
engineName bool yes * Required only if requireEngine is true. Specifies the game engine required by this flow. Currently valid values are “ue” and “unity”.
validEngineVersions []string yes * Used only if requireEngine is true. Defines the list of engine versions that are supported by this flow. Can be empty. If the array is empty, then all engine version are assumed to be supported.
ValidEngineTargetPlatforms []string yes * Used only if requireEngine is true. Defines the list of target platforms that are supported by this flow. Can be empty. If the array is empty, then all target platforms are assumed to be supported.
inputs []FlowInput yes * Can be empty. Defines the list of input variables that this flow requires. The flow system will make sure that these inputs are defined before the flow is invoked. See below for definition of the FlowInput structure
outputs []FlowOutput yes * Can be empty. Defines the list of output variables that this flow defines. When the flow is executed, the system guarantees that these output variables are passed to succeeding flows. See below for definition of the FlowOutput structure

FlowInput parameters:

Attribute Type Required Description
inputName string yes The variable name that must be defined.
inputType string yes The type of value that the variable represents. Currently valid values are “variable”, and “artifact”. A value of “variable” indicates that the variable simply holds a value. A value of “artifact” indicates that there is a resource on disk (file or directory) and that the value of the variable is a path reference that can use to access that resource.
required bool yes Indicates whether this input is required by the flow.
default string no Can only be specified for inputs that have required defined to fals. This field defines the value that the optional input will have if it is not defined by the caller. If an input is not required, and does not have the default field defined, the value will be an empty string.

FlowOutput parameter:

Attribute Type Required Description
outputName string yes The variable name that is exported to subsequent flows.
outputType string yes The type of value that the variable represents. Currently valid values are “variable”, and “artifact”. A value of “variable” indicates that the variable simply holds a value. A value of “artifact” indicates that there is a resource on disk (file or directory) and that the value of the variable is a path reference that subsequent flows can use to access that resource.

The code

The code that implements the flow lives in a .yml file that has the same root name as the corresponding json file described above.

There are no restrictions on what the file contain. By definition, besides system-predefined variables, the only other input variables it should depend on are those defined in the corresponding json file. That way the system can make sure that all inputs are satisfied before the flow is called.

There must be a job with the name identified in the jobName field in the corresponding flow definition file. In the case of our example “.build_ue_win”.

The syntax of the yml is derived from gitlab CI syntax.

Example - Flow.yml:

.build_ue_win:
  variables:
    # Output variables (defined in json)
    OUTPUT_BUILD_ARTIFACT: StagedBuild

  stage: build_dev

  script:
    - 'cd $uebatpath'
    - '& .\runuat -nop4 -verbose BuildCookRun -project="$PROJECT_FILE" -Target="$TARGET" -targetplatform="$FLOW_ENGINE_TARGET_PLATFORM_X" -cook -clean -unattended -stage -stagingdirectory="$PROJECT_DIR/$OUTPUT_BUILD_ARTIFACT" -build -CookAll -package'
    - Add-Content -Path $CI_PROJECT_DIR/build.env -Value "OUTPUT_BUILD_ARTIFACT=$OUTPUT_BUILD_ARTIFACT"
       
  artifacts:
      name: "$CI_JOB_NAME-$CI_COMMIT_REF_NAME"
      paths:
        - $PROJECT_DIR/$OUTPUT_BUILD_ARTIFACT
      reports:
        dotenv: build.env

Predefined VARIABLES

The following variables are defined by the system and should not be overridden.

Variable Value
WVS_EXEC_TAG The name of the tag that should be used to select a runner for the flow. The system used this tag to initiate the runner for the root job in the flow, but for more complex flows it may be necessary for the contents of the flow to know that tag in order to start a child job. If no tag was specified to start the flow, this variable will contain an empty string.
WVS_EXEC_IMAGE The name of the image that was used to run the flow, for those flows that run in a container. Not all flows are required to run in containers; some my run on provisioned machines and in such cases the value of this variable may be blank.

The following variables are defined by the system based on the flow definition.

Variable Value
FLOW_CATEGORY This variable will always be set to a valid value. It categorizes the flow into one of the currently recognized categories. The value of this variable will always be equal to the category field in the flow definition. Valid values are “build”, “deploy”, or “other”.
FLOW_ENGINE If the flow makes of a specific, recognized game engine, it is identified in this variable. This variable is only defined if the requireEngine field in the flow definition is set to true. The value of this variable will always be equal to the engineName field in the flow definition. Valid values are “ue”, or “unity”.
FLOW_ENGINE_VERSION If the flow defines requireEngine as true, then this variable is defined to hold the version string of the engine as defined by the engineVersion field in the flow instance.
FLOW_ENGINE_TARGET_PLATFORM If the flow defines requireEngine as true, then this variable is defined to hold the target platform of this flow, as defined by the engineTargetPlatform field in the flow instance.
FLOW_ENGINE_TARGET_PLATFORM_X If the variable FLOW_ENGINE_TARGET_PLATFORM is defined, then this variable is the engine-specific form of that variable. For example, if FLOW_ENGINE_TARGET_PLATFORM is “windows”, and the engineName (in the flow definition) is “ue”, then FLOW_ENGINE_TARGET_PLATFORM_X would be “Win64” because that is how the Unreal Engine refers to windows.

Where flows live

Local to the project

Shared repository

Arbitrary URL

The flow registry

Variable naming conventions

Appropriately, variables are used various ways in various places. It is the fact that they are also defined in various places that makes it convenient to stick to a naming convention that lets flow developers easily know where a variable is defined.

This is the convention that WVS currently uses:

Definition source Prefix Example Notes
System WVS_ WVS_EXEC_TAG Defined by the system. No specific location.
  CI_ CI_JOB_ID  
Flow FLOW_ FLOW_ENGINE Defined by the flow in the flow definition.
    FLOW_BUILD_TARGET  
Project PROJ_ PROJ_UE_VERSION Defined in the project’s settings.
    PROJ_GITHUB_TOKEN  

This is just a convention and it is not enforced but it is followed by WVS defined flows.

The variables used internally to the flow definition, and as input and output parameters can follow any convention desired by the author. Each flow has its own scope for variables and so those variables do not need to be unique across flows.