Copy.copyFileToPod() hangs waiting for process to complete

[PredatorVI]

Client Version: 13.0.0
Kubernetes Version: 1.19.12-gke.2100
Java Version: Java 1.8.0_291
**Server

• OS: Ubuntu 20.04.1 LTS
• Environment: GKE Container
• Cloud : GCP/GKE

I have the copyFileFromPod() working, but copyFileToPod() hangs at Copy.java line 459 at proc.waitFor(). I've pulled source, but I don't understand the websockets well enough to know what is going on. A copy command from the command-line works fine.

I've verified that the 'tar' and 'base64' executables exist in the container:

> root@test-gcp-dev-gke-guse4a-0:/tmp# which tar
> /usr/bin/tar
> root@test-gcp-dev-gke-guse4a-0:/tmp# which base64
> /usr/bin/base64
> root@test-gcp-dev-gke-guse4a-0:/tmp# cat /etc/issue
> Ubuntu 20.04.1 LTS \n \l
> 

I've not changed the timeouts as I'd expect a simple text file copy would work within the defaults.

It never seems to timeout or throw an error so I don't have any stack traces.

Also, I tried API version 12.0.1 (throws SocketTimeout almost immediately) and 11.0.2 (hangs just like 13.0.0).

[brendandburns]

For some reason the untar command on the other side isn't completing. It's hard to know why. I don't think that this is a WebSockets issue, I think it is an issue with the process in the container that is unpacking the copied files (or alternately it is an issue with the input data not getting closed properly)

How are you supplying the data that is getting copied? Byte array? Local file path? Is it one file or many?

I will try to repro locally.

[PredatorVI]

I've started testing using a simple text file.

The test I'm doing is to first call copyFileFromPod() to grab a config file from the pod (POD:/etc/adduser.conf --> LOCAL:C:/tmp/adduser.conf). I then turn around and copy that same file using copyFileToPod() (LOCAL:C:/tmp/adduser.conf --> POD:/tmp/adduser.conf).

I am using the java.nio.file.Path variant of the method:

public void copyFileToPod(String namespace, String pod, String container, Path srcPath, Path destPath)
throws ApiException, IOException

I open to suggestions on how to narrow this down more or debug it.

Thanks!!

[PredatorVI]

I noticed running 'ps -ef | grep tar' in the container that the path was not right. I am setting the destPath (remote ubuntu container) to Paths.get("/tmp/adduser.conf"), but it seems to be using the Windows path delimiter.

root 1344130 0 0 16:33 ? 00:00:00 sh -c base64 -d | tar -xmf - -C \tmp
root 1344136 1344130 0 16:33 ? 00:00:00 tar -xmf - -C tmp

I pulled the source and added a parentPath.replace("\", "/"); just to see if that fixed it. However, it still seems to hang, but the paths look better:

root 1214170 0 0 16:33 ? 00:00:00 sh -c base64 -d | tar -xmf - -C /tmp
root 12144136 1214170 0 16:33 ? 00:00:00 tar -xmf - -C /tmp

There appear to be no variants of copyFileToPod() that take a string for the remote destPath like the copyFileFromPod had for the remote srcPath so there seems to be potential issue using native Path/File references where the source and destination systems use different path delimiters.

I also noticed now that once I kill the client process, it seems to finish and the file does show up. I didn't check the content but will once I get the pod back online.

[PredatorVI]

File content appears to be written correctly when I stop/kill the client-side process. This is where my unfamiliarity with WebSockets isn't helping. I don't know how the EOF/end of input for the remote process (sh -c base64 -d | tar -xmf - -C /tmp) is triggered.

I tried grasping at straws:

• adding a call to archiveOutputStream.finish(); per https://commons.apache.org/proper/commons-compress/apidocs/org/apache/commons/compress/archivers/ArchiveOutputStream.html
• calling proc.getOutputStream().close() in the finally block.
• wrote "CTRL-D" character to the end of the output stream (proc.getOutputStream().write('\004');)

[brendandburns]

Thanks for investigating! Basically the web socket should close when the process on the client side ends. In this case that process will only end when the stdin that it is reading closes (at least that's what I think should happen)

If you're willing/able to investigate further, one thing you might try is removing the base64 encoding. I don't actually think that it is necessary, and it's possible that it's causing the problem. Then instead of sh -c ... you could just run tar -xmf - -C /tmp

It's possible that spawning the extra shell (via sh -c ...) is what is causing stdin to hang open.

[PredatorVI]

Still the same behavior. Here is the process output showing it isn't calling 'sh -c'. Hopefully the format is correct.

root@test-gcp-dev-gke-guse4a-0:/tmp# ps -ef | grep tar
root 141926 0 0 16:05 ? 00:00:00 tar -xmf - -C /tmp

Here is the updated code:

    public Future<Integer> copyFileToPodAsync(
            String namespace, String pod, String container, Path srcPath, Path destPath)
            throws ApiException, IOException {
        // Run decoding and extracting processes
        final Process proc = execCopyToPod(namespace, pod, container, destPath);

        // Send encoded archive output stream
        File srcFile = new File(srcPath.toUri());
        try (
                ArchiveOutputStream archiveOutputStream = new TarArchiveOutputStream(proc.getOutputStream());
                FileInputStream input = new FileInputStream(srcFile)) {
            ArchiveEntry tarEntry = new TarArchiveEntry(srcFile, destPath.getFileName().toString());

            archiveOutputStream.putArchiveEntry(tarEntry);
            Streams.copy(input, archiveOutputStream);
            archiveOutputStream.closeArchiveEntry();
            archiveOutputStream.finish();

            return new ProcessFuture(proc);
        }
    }

    private Process execCopyToPod(String namespace, String pod, String container, Path destPath)
            throws ApiException, IOException {
        String parentPath = destPath.getParent() != null ? destPath.getParent().toString() : ".";
        parentPath = parentPath.replace("\\", "/");
        return this.exec(
                namespace,
                pod,
                new String[]{"tar", "-xmf", "-", "-C " + parentPath},
                container,
                true,
                false);
    }

When I kill my client process however, the file does not get created as it did before. I reverted back to using 'sh -c' but left off the base64 encoding/decoding steps and the behavior goes back to hanging, but the file does get created when I kill the my client process.

[PredatorVI]

I wrote this hack method to test it in a more 'synchronous' way hoping to be able to debug better.

My First attempt was to explicitly read the proc.getInputStream() and the proc.getErrorStream() thinking that it needed to be consumed/flushed before the process could complete, but that didn't change the behavior so I pulled that code as it didn't seem to help.

I then split the TAR creation to writing a local temporary archive thinking maybe the TarArchiveOutputStream was causing a hang-up for some reason. That didn't help either.

The code below is basically working by doing a proc.destroy() after the try-with-resources streams are closed and always returns '0' since closing the websockets appears to allow the file creation to complete.

I have not yet found the right combination of flush()/close() calls that allow the process to exit normally. I think I've exhausted the limits of my understanding for the Copy.copyFileToPod() methods. Maybe there is an issue in the WebSocket handling? I'm just starting to look down that road.

private int copyFileToPodBruteForce(
            String namespace, String pod, String container, Path srcPath, Path destPath)
            throws ApiException, IOException {
        // Run decoding and extracting processes
        final Process proc = execCopyToPod(namespace, pod, container, destPath);

        // Send encoded archive output stream
        File srcFile = new File(srcPath.toUri());
        try (ArchiveOutputStream archiveOutputStream
                = new TarArchiveOutputStream(proc.getOutputStream());
                FileInputStream input = new FileInputStream(srcFile)) {
            ArchiveEntry tarEntry = new TarArchiveEntry(srcFile, destPath.getFileName().toString());

            archiveOutputStream.putArchiveEntry(tarEntry);
            Streams.copy(input, archiveOutputStream);
            archiveOutputStream.closeArchiveEntry();
            archiveOutputStream.flush();
            archiveOutputStream.finish();
        }
        proc.destroy();
        return 0;
    }
}