Staggered "build on all" for jobs

Comments

5 comments

  • Avatar
    Srinivas Upadhya

    Hi Brett,

    Thank you for the suggestion. It makes sense to have this behavior.

    We have logged a feature request and will get back to you when we implement it.

    Regards,

    Srinivas

    0
    Comment actions Permalink
  • Avatar
    Patrick Schilling

    Any chance we will see this feature in an upcoming release?  It would be a big help.

    0
    Comment actions Permalink
  • Avatar
    Princy

    Hi Brett

    To get the feature working would require us to change the core working of how Go thinks of Jobs. In Go's concept jobs are run in parallel and stages are sequential. 

    We would suggest you to split the jobs such that the restart is taken care of in another stage.

    Regards,

    Ali/Princy

    0
    Comment actions Permalink
  • Avatar
    Aravind SV

    To get into a little more detail, jobs are meant to run in parallel, it's a core concept of Go. Changing this requires quite a bit of effort, and that's why it hasn't been considered yet. However, while this is the case, you can do something which gets you close to what you want (but not exactly what you want).

    You can split the agents you want to run-on-all into two, by giving half of them a resource, say, "resource1" and the other half a resource, say, "resource2". Then, if you have 2 stages, with identical jobs set to run-on-all, with the first stage set to run the job on agents with resource1 and the second stage to run the job on agents with resource2, then, half your app servers will be upgraded, while the other half are serving requests. Then, the second half of your app servers will be upgraded.

    It's a workaround, for sure. But, it might work for you.

    0
    Comment actions Permalink
  • Avatar
    Brett Cave

    Thanks for the suggestions on the workaround - we implemented a workaround within a day of posting this improvement. It involves using a task that calls a wrapper script to execute the command (e.g. service x restart), but uses a lock file on shared storage for control. There is no special configuration needed in Go then. Here's a psuedo example of how we did it:

     

        RUN_ID=${GO_PIPELINE_NAME}-$(GO_PIPELINE_COUNTER}-$(GO_STAGE_NAME}-${GO_STAGE_COUNTER}

        function do_restart() {

            service x restart

        }

        while [ -f /nfsshare/appdeploy/${RUN_ID} ]; do

           sleep 10

        done

        hostname > /nfsshare/appdeploy/${RUN_ID}

        do_restart

        sleep 10      # give the service 10 seconds to start up

        rm -f /nfsshare/appdeploy/${RUN_ID}

     

    0
    Comment actions Permalink

Please sign in to leave a comment.