Drive Go from Jenkins

Say we have a job in Jenkins (publish_artifact) that archives an artifact in Jenkins. Every time this job succeeds, we want to auto-schedule a Go pipeline (targetPipeline) that will then download the artifact and use it. There are two ways to do this. One using Go API to notify Go and another that uses a git repo that Go then polls. The first approach is simpler but the second one has the added benefit of some reverse traceability.

1. Go API Approach

Jenkins Set up

publish_artifact is a job in Jenkins that archives an artifact (artifact.txt) as a post build action (see image below). We make use of the parameterized build trigger plugin to add another post build action that triggers the handing-over job in Jenkins (talk2go).

Note that talk2Go needs to declare itself as a parameterized build to complete the binding.

talk2Go simply uses the following shell command to schedule the pipeline in Go. Refer Go Pipeline scheduling API for details.

 (some newlines added for readability, remove if copy/pasting)

curl -u user:passwd -d "variables[JENKINS_URL]=${JENKINS_URL}&

variables[JENKINS_JOB_NAME]=publish-artifact&

variables[JENKINS_JOB_NUMBER]=${PUBLISH_JOB_NUM}&

variables[JENKINS_ARTIFACT]=artifact.txt"

http://goserver:8153/go/api/pipelines/targetPipeline/schedule

Go set up

For the above call to work, we need to define pipeline level environment variables in the target Go pipeline as follows.

The Go pipeline also needs a material. Ideally this would be the repo containing deployment scripts (such as the one below - shown as an inline task here for readability). Finally, we add a Go job that downloads the artifact from Jenkins.

When set up correctly, we should see output from the Go pipeline as follows. The artifact gets saved into downloadedFile under the pipeline working directory.

[go] Start to execute task: <exec command="/bin/bash" >
<arg>-c</arg>
<arg>curl -o downloadedFile 

${JENKINS_URL}job/${JENKINS_JOB_NAME}/${JENKINS_JOB_NUMBER}/artifact/${JENKINS_ARTIFACT}</arg>
</exec>.

…

[go] setting environment variable 'JENKINS_JOB_NAME' to value ''
[go] setting environment variable 'JENKINS_ARTIFACT' to value ''
[go] setting environment variable 'JENKINS_JOB_NUMBER' to value ''
[go] setting environment variable 'JENKINS_URL' to value ''
[go] overriding environment variable 'JENKINS_URL' with value 'http://jenkins-server:8080/'
[go] overriding environment variable 'JENKINS_JOB_NAME' with value 'publish-artifact'
[go] overriding environment variable 'JENKINS_JOB_NUMBER' with value '7'
[go] overriding environment variable 'JENKINS_ARTIFACT' with value 'artifact.txt'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

2. Material Approach

Jenkins set up

In this approach, we make talk2Go provide the required information to Go via a git repo rather than via API. Say we have a git repo called testrepo with a file jenkins.properties containing the following:

$ cat jenkins.properties
JENKINS_URL=http://jenkins-server:8080/
JENKINS_JOB_NAME=publish-artifact
JENKINS_JOB_NUMBER=10
JENKINS_ARTIFACT=artifact.txt

talk2Go updates and pushes this file to git as follows. Note how we write the git checkin comment to avail of reverse traceability from Go.

#!/bin/bash

cat >/var/lib/jenkins/testrepo/jenkins.properties <<EOL

JENKINS_URL=$JENKINS_URL
JENKINS_JOB_NAME=publish-artifact
JENKINS_JOB_NUMBER=$PUBLISH_JOB_NUM
JENKINS_ARTIFACT=artifact.txt
EOL

# newline added for readability, remove if copy/pasting

cd /var/lib/jenkins/testrepo;git commit jenkins.properties -m "#$PUBLISH_JOB_NUM checkin by jenkins";

git push origin master

Go set up

The Go pipeline has testrepo as a material. It gets scheduled in due course, runs a job that exports environment variables from jenkins.properties and then makes the usual Jenkins fetch artifact API call.

/bin/bash -c

#newline added for readability, remove if copy/pasting

while read line; do export $line; done <jenkins.properties;curl -o downloadedFile

${JENKINS_URL}job/${JENKINS_JOB_NAME}/${JENKINS_JOB_NUMBER}/artifact/${JENKINS_ARTIFACT}

This approach no longer requires us to pre-define environment variables at the pipeline level. To get reverse traceability, we configure Go to link checkin comments back to Jenkins

That's it, now when publish_artifact runs in Jenkins, it then triggers talk2Go which pushes to testrepo. Go polls testrepo, detects the changes and schedules the target pipeline. The Changes section now has a helpful link back to the triggering job in Jenkins!

Handover external artifacts from Jenkins to Go

What if the artifact was generated but not stored in Jenkins? Simply use a different parameter to trigger talk2Go from publish_artifact.

ARTIFACT_URL=<path to artifact in external repo>

Soon Go pipelines will have the ability to poll package repositories. That will give us another option in addition to the above notification approach.

If, for some reason, you happen to be storing the artifacts in version control instead, you could simply create a Go pipeline to poll the VCS (material). In case of VCS, there is also the option to use a non-specific notification that just says material changed without reference to a specific pipeline.

http://www.thoughtworks-studios.com/docs/go/current/help/materials_api.html