If you have agents that connect through Java Web Start (such as agents installed as Windows services), we have a good news for you.
In case of a connection loss, this type of agents has been designed to automatically attempt to reconnect to the controller. This makes sense because you want these agents to remain online all the time, even if your janitor trips over the ethernet cable. Unfortunately, it also means that over the time, these agents accumulate gunk, such as mutated static states, any left-over threads or memory leaks, or native libraries that are loaded into JVM.
To prevent that, a better approach is to restart the agent JVM (JENKINS-19055) and have the new JVM reconnect, instead of having the same JVM reconnect. That would ensure that the agent always stays clean. I’ve planned to make this change for a while now, and I’m happy to report that this change is finally landing to the upcoming 1.559.
Restarting JVM is easy on Unix, where I could just exec(3) to itself. We’ve been doing this for ages on controllers, for example when you update a plugin and tell Jenkins to restart.
The hard part is to do this for Windows, where the most of the time was spent. I had to improve windows service wrapper to support self-restarting services, which turned out to be trickier because Windows service control manager doesn’t provide "restart" as an atomic operation. It also kills not just the service process itself but all the processes in the group. So I had to double-fork the service wrapper into a separate process group just to restart a service from within itself.
In any case, the end result is that if you have installed a service through GUI, be it on Windows, Unix, or OS X, agents will restart themselves every time it gets disconnected from the controller.
I’ve also taken the opportunity to make jenkins-slave.exe on the agent self-updating. Every time it connects to the controller, it gets the latest version from the controller.
If you have installed Web Start agents as services, make sure to update the local copy of agent.jar on these agents to 2.37 or later. This "restart on reconnect" feature only kicks in when you are running this very recent version of agent.jar. And yes, we realize it’d be nice for agent.jar to update itself, which is tracked as JENKINS-22454. But that’s a work for another day.
|
Kohsuke Kawaguchi
Kohsuke is the creator of Jenkins. |