[JENKINS-39370,JENKINS-39369] - Support of work directories in Remoting

README.md

@@ -53,6 +53,8 @@ User documentation:
 * [Remoting 3 Compatibility Notes](docs/remoting-3-compatibility.md)
 * [Remoting Protocols](docs/protocols.md) - Overview of protocols integrated with Jenkins
 * [Remoting Configuration](docs/configuration.md) - Configuring remoting agents
+* [Logging](docs/logging.md) - Logging
+* [Work Directory](docs/workDir.md) - Remoting work directory (new in Remoting `TODO`)
 * [Jenkins Specifics](docs/jenkins-specifics.md) - Notes on using remoting in Jenkins
 * [Troubleshooting](docs/troubleshooting.md) - Investigating and solving common remoting issues
 

docs/logging.md

@@ -0,0 +1,46 @@
+Logging
+===
+
+In Remoting logging is powered by the standard `java.util.logging` engine. 
+The default behavior depends on the [Work Directory](workDir.md) mode.
+
+### Configuration
+
+In order to configure logging it is possible to use an external property file, path to which can be defined using the `-loggingConfig` CLI option or the `java.util.logging.config.file` system property. 
+
+If logging is configured via `-loggingConfig`, some messages printed before the logging system initialization may be missing in startup logs configured by this option.
+
+See details about the property file format
+in [Oracle documentation](https://docs.oracle.com/cd/E19717-01/819-7753/6n9m71435/index.html)
+and [this guide](http://tutorials.jenkov.com/java-logging/configuration.html).
+Note that `ConsoleHandler` won't be enabled by default if this option is specified.
+
+### Default behavior with work directory
+
+With work directory Remoting automatically writes logs to the disk.
+This is a main difference from the legacy mode without workDir.
+
+Logging destinations:
+
+* STDOUT and STDERR
+  * Logs include `java.util.logging` and messages printed to _STDOUT/STDERR_ directly.
+* Files - `${workDir}/${internalDir}/logs` directory
+  * File base name - `remoting.log`
+  * Logs are being automatically rotated.
+    By default, Remoting keeps 5 10MB files
+  * Default logging level - `INFO`
+  * If the legacy `-agentLog` or `-slaveLog` option is enabled, this file logging will be disabled.
+
+If `-agentLog` or `-slaveLog` are not specified, `${workDir}/${internalDir}/logs` directory will be created during the work directory initialization (if required).
+
+<!--TODO: Mention conflict with early initialization by java.util.logging.config.file?-->
+
+### Default behavior without work directory (legacy mode)
+
+* By default, all logs within the system are being sent to _STDOUT/STDERR_ using `java.util.logging`.
+* If `-agentLog` or `-slaveLog` option is specified, the log will be also forwarded to the specified file
+  * The existing file will be overridden on startup
+  * Remoting does not perform automatic log rotation of this log file
+
+Particular Jenkins components use external features to provide better logging in the legacy mode.
+E.g. Windows agent services generate logs using features provided by [Windows Service Wrapper (WinSW)](https://github.com/kohsuke/winsw/).

docs/workDir.md

@@ -0,0 +1,45 @@
+Remoting Work directory
+===
+
+In Remoting work directory is a storage 
+
+Remoting work directory is available starting from Remoting `TODO`.
+Before this version there was no working directory concept in the library itself;
+all operations were managed by library users (e.g. Jenkins agent workspaces).
+
+### Before Remoting TODO
+
+* There is no work directory management in Remoting itself
+* Logs are not being persisted to the disk unless `-slaveLog` option is specified
+* JAR Cache is being stored in `${user.home}/.jenkins` unless `-jarCache` option is specified
+
+### After Remoting TODO
+
+Due to compatibility reasons, Remoting retains the legacy behavior by default.
+Work directory can be enabled using the `-workDir` option in CLI or via the `TODO` [system property](configuration.md).
+
+Once the option is enabled, Remoting starts using the following structure:
+
+```
+${WORKDIR}
+  |_ ${INTERNAL_DIR} - defined by '-internalDir', 'remoting' by default
+    |_ jarCache - JAR Cache
+    |_ logs     - Remoting logs
+    |_ ...      - Other directories contributed by library users
+```
+
+Structure of the `logs` directory depends on the logging settings.
+See [this page](logging.md) for more information.
+
+### Migrating to work directories in Jenkins
+
+:exclamation: Remoting does not perform migration from the previous structure, 
+because it cannot identify potential external users of the data.
+
+Once the `-workDir` flag is enabled in Remoting, admins are expected to do the following:
+
+1. Remove the `${user.home}/.jenkins` directory if there is no other Remoting instances running under the same user.
+2. Consider upgrading configurations of agents in order to enable Work Directories
+  * SSH agents can be configured in agent settings.
+  * JNLP agents should be started with the `-workDir` parameter.
+  * See [JENKINS-TODO](TODO) for more information about changes in Jenkins plugins, which enable work directories by default.

src/main/java/hudson/remoting/Engine.java

@@ -29,9 +29,12 @@
 import java.io.File;
 import java.io.FileInputStream;
 import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
 import java.io.IOException;
+import java.io.PrintStream;
 import java.net.Socket;
 import java.net.URL;
+import java.nio.file.Path;
 import java.security.AccessController;
 import java.security.KeyManagementException;
 import java.security.KeyStore;
@@ -68,6 +71,7 @@
 import org.jenkinsci.remoting.engine.JnlpConnectionStateListener;
 import org.jenkinsci.remoting.engine.JnlpProtocolHandler;
 import org.jenkinsci.remoting.engine.JnlpProtocolHandlerFactory;
+import org.jenkinsci.remoting.engine.WorkDirManager;
 import org.jenkinsci.remoting.protocol.IOHub;
 import org.jenkinsci.remoting.protocol.cert.BlindTrustX509ExtendedTrustManager;
 import org.jenkinsci.remoting.protocol.cert.DelegatingX509ExtendedTrustManager;
@@ -150,7 +154,62 @@ public void run() {
      */
     private boolean keepAlive = true;
 
-    private JarCache jarCache = new FileSystemJarCache(new File(System.getProperty("user.home"),".jenkins/cache/jars"),true);
+
+    /**
+     * Default JAR cache location for disabled workspace Manager.
+     */
+    private static final File DEFAULT_NOWS_JAR_CACHE_LOCATION = 
+        new File(System.getProperty("user.home"),".jenkins/cache/jars");
+
+    @CheckForNull
+    private JarCache jarCache = null;
+
+    /**
+     * Specifies a destination for the agent log.
+     * If specified, this option overrides the default destination within {@link #workDir}.
+     * If both this options and {@link #workDir} is not set, the log will not be generated.
+     * @since TODO
+     */
+    @CheckForNull
+    private Path agentLog;
+
+    /**
+     * Specified location of the property file with JUL settings.
+     * @since TODO
+     */
+    @CheckForNull
+    private Path loggingConfigFilePath = null;
+
+    /**
+     * Specifies a default working directory of the remoting instance.
+     * If specified, this directory will be used to store logs, JAR cache, etc.
+     * <p>
+     * In order to retain compatibility, the option is disabled by default.
+     * <p>
+     * Jenkins specifics: This working directory is expected to be equal to the agent root specified in Jenkins configuration.
+     * @since TODO
+     */
+    @CheckForNull
+    public Path workDir = null;
+
+    /**
+     * Specifies a directory within {@link #workDir}, which stores all the remoting-internal files.
+     * <p>
+     * This option is not expected to be used frequently, but it allows remoting users to specify a custom
+     * storage directory if the default {@code remoting} directory is consumed by other stuff.
+     * @since TODO
+     */
+    @Nonnull
+    public String internalDir = WorkDirManager.DirType.INTERNAL_DIR.getDefaultLocation();
+
+    /**
+     * Fail the initialization if the workDir or internalDir are missing.
+     * This option presumes that the workspace structure gets initialized previously in order to ensure that we do not start up with a borked instance
+     * (e.g. if a filesystem mount gets disconnected).
+     * @since TODO
+     */
+    @Nonnull
+    public boolean failIfWorkDirIsMissing = WorkDirManager.DEFAULT_FAIL_IF_WORKDIR_IS_MISSING;
 
     private DelegatingX509ExtendedTrustManager agentTrustManager = new DelegatingX509ExtendedTrustManager(new BlindTrustX509ExtendedTrustManager());
 
@@ -165,12 +224,68 @@ public Engine(EngineListener listener, List<URL> hudsonUrls, String secretKey, S
     }
 
     /**
-     * Configures JAR caching for better performance.
+     * Starts the engine.
+     * The procedure initializes the working directory and all the required environment
+     * @throws IOException Initialization error
+     * @since TODO
+     */
+    public synchronized void startEngine() throws IOException {
+
+        @CheckForNull File jarCacheDirectory = null;
+
+        // Prepare the working directory if required
+        if (workDir != null) {
+            final WorkDirManager workDirManager = WorkDirManager.getInstance();
+            if (jarCache != null) {
+                // Somebody has already specificed Jar Cache, hence we do not need it in the workspace.
+                workDirManager.disable(WorkDirManager.DirType.JAR_CACHE_DIR);
+            }
+
+            if (loggingConfigFilePath != null) {
+                workDirManager.setLoggingConfig(loggingConfigFilePath.toFile());
+            }
+
+            final Path path = workDirManager.initializeWorkDir(workDir.toFile(), internalDir, failIfWorkDirIsMissing);
+            jarCacheDirectory = workDirManager.getLocation(WorkDirManager.DirType.JAR_CACHE_DIR);
+            workDirManager.setupLogging(path, agentLog);
+        } else if (jarCache != null) {
+            LOGGER.log(Level.WARNING, "No Working Directory. Using the legacy JAR Cache location: {0}", DEFAULT_NOWS_JAR_CACHE_LOCATION);
+            jarCacheDirectory = DEFAULT_NOWS_JAR_CACHE_LOCATION;
+        }
+
+        if (jarCache == null){
+            if (jarCacheDirectory == null) {
+                // Should never happen in the current code
+                throw new IOException("Cannot find the JAR Cache location");
+            }
+            LOGGER.log(Level.FINE, "Using standard File System JAR Cache. Root Directory is {0}", jarCacheDirectory);
+            jarCache = new FileSystemJarCache(jarCacheDirectory, true);
+        } else {
+            LOGGER.log(Level.INFO, "Using custom JAR Cache: {0}", jarCache);
+        }
+
+        // Start the engine thread
+        this.start();
+    }
+
+    /**
+     * Configures custom JAR Cache location.
+     * Starting from TODO, this option disables JAR Caching in the working directory.
+     * @param jarCache JAR Cache to be used
      * @since 2.24
      */
-    public void setJarCache(JarCache jarCache) {
+    public void setJarCache(@Nonnull JarCache jarCache) {
         this.jarCache = jarCache;
     }
+
+    /**
+     * Sets path to the property file with JUL settings.
+     * @param filePath JAR Cache to be used
+     * @since TODO
+     */
+    public void setLoggingConfigFile(@Nonnull Path filePath) {
+        this.loggingConfigFilePath = filePath;
+    }
 
     /**
      * Provides Jenkins URL if available.
@@ -198,6 +313,44 @@ public void setNoReconnect(boolean noReconnect) {
         this.noReconnect = noReconnect;
     }
 
+    /**
+     * Sets the destination for agent logs.
+     * @param agentLog Path to the agent log.
+     *      If {@code null}, the engine will pick the default behavior depending on the {@link #workDir} value
+     * @since TODO
+     */
+    public void setAgentLog(@CheckForNull Path agentLog) {
+        this.agentLog = agentLog;
+    }
+
+    /**
+     * Specified a path to the work directory.
+     * @param workDir Path to the working directory of the remoting instance.
+     *                {@code null} Disables the working directory.
+     * @since TODO
+     */
+    public void setWorkDir(@CheckForNull Path workDir) {
+        this.workDir = workDir;
+    }
+
+    /**
+     * Specifies name of the internal data directory within {@link #workDir}.
+     * @param internalDir Directory name
+     * @since TODO
+     */
+    public void setInternalDir(@Nonnull String internalDir) {
+        this.internalDir = internalDir;
+    }
+
+    /**
+     * Sets up behavior if the workDir or internalDir are missing during the startup.
+     * This option presumes that the workspace structure gets initialized previously in order to ensure that we do not start up with a borked instance
+     * (e.g. if a filesystem mount gets disconnected).
+     * @param failIfWorkDirIsMissing Flag
+     * @since TODO
+     */
+    public void setFailIfWorkDirIsMissing(boolean failIfWorkDirIsMissing) { this.failIfWorkDirIsMissing = failIfWorkDirIsMissing; }
+
     /**
      * Returns {@code true} if and only if the socket to the master will have {@link Socket#setKeepAlive(boolean)} set.
      *
@@ -239,6 +392,7 @@ public void removeListener(EngineListener el) {
 
     @Override
     public void run() {
+        // Create the engine
         try {
             IOHub hub = IOHub.create(executor);
             try {

src/main/java/hudson/remoting/FileSystemJarCache.java

@@ -37,11 +37,15 @@ public class FileSystemJarCache extends JarCacheSupport {
     private final Map<String, Checksum> checksumsByPath = new HashMap<>();
 
     /**
+     * @param rootDir  
+     *      Root directory.
      * @param touch
      *      True to touch the cached jar file that's used. This enables external LRU based cache
      *      eviction at the expense of increased I/O.
+     * @throws IllegalArgumentException
+     *      Root directory is {@code null} or not writable.
      */
-    public FileSystemJarCache(File rootDir, boolean touch) {
+    public FileSystemJarCache(@Nonnull File rootDir, boolean touch) {
         this.rootDir = rootDir;
         this.touch = touch;
         if (rootDir==null)
@@ -54,6 +58,11 @@ public FileSystemJarCache(File rootDir, boolean touch) {
         }
     }
 
+    @Override
+    public String toString() {
+        return String.format("FileSystem JAR Cache: path=%s, touch=%s", rootDir, Boolean.toString(touch));
+    }
+
     @Override
     protected URL lookInCache(Channel channel, long sum1, long sum2) throws IOException, InterruptedException {
         File jar = map(sum1, sum2);