A Tale of Two Subsystems

There are three kinds of scheduling of Pipelines in Go - Auto, Manual, Timer Triggered. Two subsystems that provide the required behaviour are Material Update System and Scheduling Subsystem. These two subsystems are interwoven in the dashboard to allow the user to work seamlessly across them.

Understanding of these two subsystems helps in using the most appropriate option to achieve the needed behavior for a pipeline.

Material Update Subsystem

This subsystem works with materials configured in Go. It polls the specified material repositories and brings in the material change details into Go. This information is stored in Go’s internal datastore. The information is used in various parts of Go - changes pop-up on dashboard, materials/overview tab on the stage details page, compare pipelines, trigger with options etc. In summary, the sole purpose of the material update system is to fetch the change details from the repository(and not the changed data itself).

The flag that controls this subsystem's behavior is the option; available on each of the configured materials. Unchecking this option will make the material update system skip querying the repository for such materials.

Scheduling Subsystem

This subsystem deals with scheduling of pipelines. The pipeline type and corresponding schedule trigger are listed below.

Auto Pipeline

This is default mode of a newly created pipeline. For such a pipeline, scheduling system watches for the change details fetched by the material update subsystem and triggers the pipeline when new changes are found.

Manual Pipeline

As we know, pipeline is a sequence of stages. A manual pipeline is a pipeline with first stage Manual. In this case the scheduling subsystem will only schedule on a user trigger. The pipeline will use all the material changes fetched by the material update system between two consecutive user triggers. If there are no new material changes, the pipeline will use the same changes as the previous execution.

In addition, the material update system is asked to check for the latest changes at the time of the trigger. Hence a user trigger puts both the material update system and the scheduling system into action.

Timer Pipeline

The pipeline will trigger based on the configured timer specification. In addition to the timer configuration, pipeline itself can have its first stage Manual or On Success.

If the first stage is On Success, the pipeline is similar to an auto pipeline; the pipeline will be scheduled on both- every new change fetched by material update subsystem and based on the timer specification. In timer initiated schedule, the pipeline may not have any new material changes in this case (since the pipeline would have already run for the material when the changes were fetched by the material update system) but the pipeline will still schedule.

If the first stage is Manual, the pipeline is similar to a manual pipeline; the pipeline will schedule based on the timer specification only. The pipeline will use all the material changes fetched by the material update system between two consecutive timer triggers.

Timer trigger never instructs the material update system to check for changes (unlike user trigger); it merely uses the data stored my material update system in Go’s datastore to continue the trigger process.

Scenarios and Configuration samples

Following are the example scenarios that demonstrates the use of the configuration options explained above.

Scenario 1: Overnight Performance or Regression testing activities

Pipeline for these jobs should always run at the specified time with the latest scripts.

Configuration:

• The pipeline should be configured with the material that contains performance/regression testing scripts.

• Cron timer for the time of the run should be specified

• Pipeline will run consistently at the given time irrespective of any material changes.

• First stage should be manual since this testing should happen only at the given time.

• This will ensure any changes to the testing scripts does not schedule the jobs out of turn.

• The material should be set to poll for new changes, so that the material update system will fetch changes to the testing scripts periodically. The timer will not fetch any new changes and will just run of the changes already fetched.

Scenario 2: Making nightly builds of an application via network share.

Pipeline for these jobs should run in case of any code changes(to produce a build) and also at a specified time to create nightly build.

Configuration:

• The pipeline should watch the material that contains the code and scripts.

• Cron timer for the time of the nightly run  should be specified

• Pipeline will run consistently at the given time irrespective of any material changes.

• First stage should be automatic so that the builds are generated on every new change too.

• The material should be set to poll for new changes, so that the material update system will fetch changes to the code that causes the pipeline to trigger. The timer will not fetch any new changes; will just run of the changes already fetched and produce the nightly build.

Scenario 3: Production team taking an application live.

Candidate build that is successfully tested and approved has to be deployed on to production. This activity usually done manually by authorized personnel.

Configuration:

• The pipeline should watch the material that contains the code and scripts.

• First stage should be manual and approval permissions to appropriate users should be set.

• The material should be set to poll for new changes, so that the material update system will fetch changes periodically. At the time of trigger, the user will select the candidate revision using the trigger with options for deployment.

Scenario 4: Go server monitoring a public repository(like github) and deploying on to cloud.

In this case, the deployment activity is manual and the user will decide when the deployment will take place. In addition, the user would not want the Go server to continuously monitor changes on public repository since this would cost him on the network load but also spam the public repo with request which might result in hosting website considering the activity as suspicious.

Configuration:

• The pipeline should watch the public repository material.

• Poll of new changes for this material should be unchecked to ensure that Go server does not communicate with the public repo on a periodic basis.

• First stage should be manual and approval permissions to appropriate users should be set.

• At the time of user trigger,  as mentioned above (Manual Pipeline section) the latest changes will be fetched, built and deployed.