Articles in this section

See more

Pipeline failing to load artifact

Avatar
Scott, Stephen
Updated
Follow 

Hi'

Running Go Enterprise 2.2 on RedHat Linux. Agnt installed on same server as server, one of my pipelines has started failing because it is not loading the artifact generated.

This pipeline has been running successfully for sometime now, it has only started behaving like this since yesterday.

I have verified that the artifact exists in the go-agent directory structure, and that it's less than 1Gb.

I can see the following error in the go-agent log file:

2011-07-20 09:04:40,676 [loopThread] ERROR thoughtworks.cruise.publishers.CruiseArtifactsManipulator:75 - Failed to upload /customer/product/go/go-agent-2/pipelines/Oldstack_CI/target/britishgas.ear.tar
java.io.IOException: No space left on device

Which points to a disk being full. However the disk go is installed on has 150Gb free (as reported both by the Agent menu in Go, and directly on the server).

My question therefore is what device does go think has reached capacity?

 

Steve

Related articles

Comments

2 comments

Sort by Date Votes
  • Avatar
    Sachin Sudheendra

    Hi Steve,


    You could get this exception if you ran out of file descriptors. You can check this by running the command "sysctl fs.file-nr"


    Also, can you please try the following to bump up the limit on file descriptors for go user:


    Check the output of 'ulimit -Hn' (this will return hard limit) and 'ulimit -Sn'(which returns soft limit). Both of these values should be set to high numbers like 20000. If its not already set to high enough number, please use these commands to set them to appropriate values



    - Check the contents of /proc/sys/fs/file-max (this should be a fairly high number, if not, please set it to 600000 using the command 'sudo sysctl -w fs.file-max=600000'


    - Add the following lines to /etc/security/limits.conf (first column is name of the user, please change it to cruise if the server runs as cruise)


    go         soft    nofile          20000


    go         hard    nofile          20000


    - Stop Go Server and all instances of Go Agent running locally


    - Terminate the shell you have used to do all these actions


    - Get another session (as user go or cruise)


    - Verify the numbers had effect (use ulimit -Hn and ulimit -Sn)


    - Start the Go Server and Go Agents

    0
    Permalink
  • Avatar
    Scott, Stephen

    Thanks Sachin,


     


    I'll look into this. However in this case we've discovered the root cause was that /tmp was100% full.


     


    Is there any way to configure go so that it uses a different temporry area?


     


    Steve

    0
    Permalink

Please sign in to leave a comment.