Pipeline failing to load artifact

Updated September 27, 2016 18:26

Hi'

Running Go Enterprise 2.2 on RedHat Linux. Agnt installed on same server as server, one of my pipelines has started failing because it is not loading the artifact generated.

This pipeline has been running successfully for sometime now, it has only started behaving like this since yesterday.

I have verified that the artifact exists in the go-agent directory structure, and that it's less than 1Gb.

I can see the following error in the go-agent log file:

2011-07-20 09:04:40,676 [loopThread] ERROR thoughtworks.cruise.publishers.CruiseArtifactsManipulator:75 - Failed to upload /customer/product/go/go-agent-2/pipelines/Oldstack_CI/target/britishgas.ear.tar
java.io.IOException: No space left on device

Which points to a disk being full. However the disk go is installed on has 150Gb free (as reported both by the Agent menu in Go, and directly on the server).

My question therefore is what device does go think has reached capacity?

Steve

Comments

2 comments

Sachin Sudheendra July 20, 2011 09:07

Hi Steve,

You could get this exception if you ran out of file descriptors. You can check this by running the command "sysctl fs.file-nr"

Also, can you please try the following to bump up the limit on file descriptors for go user:

Check the output of 'ulimit -Hn' (this will return hard limit) and 'ulimit -Sn'(which returns soft limit). Both of these values should be set to high numbers like 20000. If its not already set to high enough number, please use these commands to set them to appropriate values

- Check the contents of /proc/sys/fs/file-max (this should be a fairly high number, if not, please set it to 600000 using the command 'sudo sysctl -w fs.file-max=600000'

- Add the following lines to /etc/security/limits.conf (first column is name of the user, please change it to cruise if the server runs as cruise)

go soft nofile 20000

go hard nofile 20000

- Stop Go Server and all instances of Go Agent running locally

- Terminate the shell you have used to do all these actions

- Get another session (as user go or cruise)

- Verify the numbers had effect (use ulimit -Hn and ulimit -Sn)

- Start the Go Server and Go Agents

0

Comment actions Permalink
Scott, Stephen July 20, 2011 09:30

Thanks Sachin,

I'll look into this. However in this case we've discovered the root cause was that /tmp was100% full.

Is there any way to configure go so that it uses a different temporry area?

Steve

0

Comment actions Permalink

Please sign in to leave a comment.

Articles in this section