There are three kinds of scheduling of Pipelines in Go - Auto, Manual, Timer Triggered. Two subsystems that provide the required behaviour are Material Update System and Scheduling Subsystem. These two subsystems are interwoven in the dashboard to allow the user to work seamlessly across them.
Understanding of these two subsystems helps in using the most appropriate option to achieve the needed behavior for a pipeline.
Material Update Subsystem
This subsystem works with materials configured in Go. It polls the specified material repositories and brings in the material change details into Go. This information is stored in Go’s internal datastore. The information is used in various parts of Go - changes pop-up on dashboard, materials/overview tab on the stage details page, compare pipelines, trigger with options etc. In summary, the sole purpose of the material update system is to fetch the change details from the repository(and not the changed data itself).
The flag that controls this subsystem's behavior is the option; available on each of the configured materials. Unchecking this option will make the material update system skip querying the repository for such materials.
Scheduling Subsystem
This subsystem deals with scheduling of pipelines. The pipeline type and corresponding schedule trigger are listed below.
Scheduling Type |
Trigger |
Auto Scheduling |
Material changes fetched by the material update system |
Manual Scheduling |
User triggered. |
Timer Scheduling |
Timer triggered. |
Auto Pipeline
This is default mode of a newly created pipeline. For such a pipeline, scheduling system watches for the change details fetched by the material update subsystem and triggers the pipeline when new changes are found.
Manual Pipeline
As we know, pipeline is a sequence of stages. A manual pipeline is a pipeline with first stage Manual. In this case the scheduling subsystem will only schedule on a user trigger. The pipeline will use all the material changes fetched by the material update system between two consecutive user triggers. If there are no new material changes, the pipeline will use the same changes as the previous execution.
In addition, the material update system is asked to check for the latest changes at the time of the trigger. Hence a user trigger puts both the material update system and the scheduling system into action.
Timer Pipeline
The pipeline will trigger based on the configured timer specification. In addition to the timer configuration, pipeline itself can have its first stage Manual or On Success.
If the first stage is On Success, the pipeline is similar to an auto pipeline; the pipeline will be scheduled on both- every new change fetched by material update subsystem and based on the timer specification. In timer initiated schedule, the pipeline may not have any new material changes in this case (since the pipeline would have already run for the material when the changes were fetched by the material update system) but the pipeline will still schedule.
If the first stage is Manual, the pipeline is similar to a manual pipeline; the pipeline will schedule based on the timer specification only. The pipeline will use all the material changes fetched by the material update system between two consecutive timer triggers.
Timer trigger never instructs the material update system to check for changes (unlike user trigger); it merely uses the data stored my material update system in Go’s datastore to continue the trigger process.
Scenarios and Configuration samples
Following are the example scenarios that demonstrates the use of the configuration options explained above.
Scenario 1: Overnight Performance or Regression testing activities
Pipeline for these jobs should always run at the specified time with the latest scripts.
Configuration:
-
The pipeline should be configured with the material that contains performance/regression testing scripts.
-
Cron timer for the time of the run should be specified
-
Pipeline will run consistently at the given time irrespective of any material changes.
-
First stage should be manual since this testing should happen only at the given time.
-
This will ensure any changes to the testing scripts does not schedule the jobs out of turn.
-
The material should be set to poll for new changes, so that the material update system will fetch changes to the testing scripts periodically. The timer will not fetch any new changes and will just run of the changes already fetched.
Scenario 2: Making nightly builds of an application via network share.
Pipeline for these jobs should run in case of any code changes(to produce a build) and also at a specified time to create nightly build.
Configuration:
-
The pipeline should watch the material that contains the code and scripts.
-
Cron timer for the time of the nightly run should be specified
-
Pipeline will run consistently at the given time irrespective of any material changes.
-
First stage should be automatic so that the builds are generated on every new change too.
-
The material should be set to poll for new changes, so that the material update system will fetch changes to the code that causes the pipeline to trigger. The timer will not fetch any new changes; will just run of the changes already fetched and produce the nightly build.
Scenario 3: Production team taking an application live.
Candidate build that is successfully tested and approved has to be deployed on to production. This activity usually done manually by authorized personnel.
Configuration:
-
The pipeline should watch the material that contains the code and scripts.
-
First stage should be manual and approval permissions to appropriate users should be set.
-
The material should be set to poll for new changes, so that the material update system will fetch changes periodically. At the time of trigger, the user will select the candidate revision using the trigger with options for deployment.
Scenario 4: Go server monitoring a public repository(like github) and deploying on to cloud.
In this case, the deployment activity is manual and the user will decide when the deployment will take place. In addition, the user would not want the Go server to continuously monitor changes on public repository since this would cost him on the network load but also spam the public repo with request which might result in hosting website considering the activity as suspicious.
Configuration:
-
The pipeline should watch the public repository material.
-
Poll of new changes for this material should be unchecked to ensure that Go server does not communicate with the public repo on a periodic basis.
-
First stage should be manual and approval permissions to appropriate users should be set.
- At the time of user trigger, as mentioned above (Manual Pipeline section) the latest changes will be fetched, built and deployed.
Comments
3 comments
Hi. Can you fix the image on this article?
I have a scenario similar to #3, except that upon approval, the job will deploy the next morning at 2am. is there a way to do this easily? I
Hi Shawn,
The setup for scenario3 would look something similar to this:
For the scenario you mentioned, the above setup could be changed to something like this:
Here, an intermediate pipeline "ApproveDeploymentToProduction" is introduced, which is a setup to be triggered manually. "DeploymentToProduction" is a manual pipeline setup to trigger based on a timer. This pipeline depends on "ApproveDeploymentToProduction". While configuring timer settings, make sure you select "Run only on new material". The important thing to note is DeploymentToProduction should not have any *pipeline* materials defined as dependency, apart from ApproveDeploymentToProduction. SCM materials can remain, but they need to be blacklisted. If you have any pipeline material defined for DeploymentToProduction, make them the material for ApproveDeploymentToProduction instead. Fetch artifacts works fine for ancestor artifacts as well.
With this setup, you could trigger ApproveDeploymentToProduction whenever you find suitable and let it go green. When the timer goes off next, it will trigger DeploymentToProduction. If you are not ready to approve the deployment to production yet, just do not trigger this pipeline. If not new builds of upstream is available, Go would not schedule the deployment pipeline when timer goes off.
Hope that helps.
PS: The Go community pages have moved to mailing lists. You can find the links here: http://www.go.cd/community. There are more people there, to help.
Please sign in to leave a comment.