We've been having a problem (Now its every couple of days) where the git modification check gets stuck. The error we've been seeing in the log is "ERROR: fatal: The remote end hung up unexpectedly". I've experienced this same issue on my home server and I'm not sure what would be causing it.
The bigger issue though is that sometimes the modification check can't recover from an error. In the attached log snippet you can see it hits the error and will skip the modification check indefinitely until we reboot the server. Go should detect the error, recover and then be able to try the modification check again.
Here is our server info:
| Go Server Version: | 2.3.1(14065-5593a5e73beeca) |
| JVM version: | 14.3-b01 |
| OS Information: | Windows Server 2008 R2 6.1 |
| Usable space in artifacts repository: | 32179 Mb |
| Database schema version: | 230009 |
Comments
4 comments
Any thoughts on this? It's happended again today and my boss isn't too happy. I'm not sure how to resolve this short of moving to another CI server. Any help would be appreciated.
Hi Mike,
Sorry for the delay. Go doesn't seem to be the cause in this case. This is a command invocation that Go makes and wether modification check succeeds or not is solely dependent of the command returning successfully.
This seems like a problem with remote end killing invocations git makes on the remote end or the proceses on the other end dying for some reason. I would suggest trying with git protocol (git://) insteed, which will be dependent on git daemon running on the other side instead of invocations made over ssh.
A way to test this out is to write an infinite loop in a shell script to continue making git pull invocations with half a minute of sleep and capture stdout/stderr from these invocations to verify the issue.
It may also be a problem with the network(as in, firewall/proxy etc), so please keep the environment for the infinite loop test as similar to the server as possible.
If git:// is a no go, you may want to switch to https:// (please check the git version before you make this switch, as http support was known to be unstable before 1.7.x).
Regards,
Janmejay
The log shows a github url, so both git protocol and https are available for the repo. They can be tried out to see if they are functioning properly.
Thanks for the troubleshooting tips. Like I mentioned earlier, the bigger problem is that Go sometimes isn't able to recover from these errors (regardless of their cause). Go should be able to hit an error with git and not get stuck. It could monitor stdout/stderr to check for an error (Like you mentioned) and it could also have a timeout where it kills the call if it takes too long (This could even be configurable). Either way it should be robust enough to handle and error and recover.
Please sign in to leave a comment.