Fix copyTo and copyFrom command in K8 class

[leninmehedy]

I think the way to first try and solve this is by upgrading all of our tools to the latest version of Kuberenetes/Kubectl/Kind /etc so that it will leverage the websocket, which might seem to solve the problem.

K8 › should be able to copy a file to and from a container

    ZlibError: zlib: unexpected end of file

      at Unzip.write (node_modules/minizlib/index.js:154:22)
      at Unzip.flush (node_modules/minizlib/index.js:105:10)
      at Unzip.end (node_modules/minizlib/index.js:111:10)
      at Unpack.end (node_modules/tar/lib/parse.js:544:21)
      at Pipe.end (node_modules/fs-minipass/node_modules/minipass/index.js:75:17)
      at ReadStream.[emitEnd2] (node_modules/fs-minipass/node_modules/minipass/index.js:522:9)
      at ReadStream.[emitEnd] (node_modules/fs-minipass/node_modules/minipass/index.js:507:21)
      at ReadStream.emit (node_modules/fs-minipass/node_modules/minipass/index.js:458:27)
      at ReadStream.emit (node_modules/fs-minipass/index.js:175:22)
      at ReadStream.[maybeEmitEnd] (node_modules/fs-minipass/node_modules/minipass/index.js:440:12)
      at ReadStream.emit (node_modules/fs-minipass/node_modules/minipass/index.js:474:27)
      at ReadStream.emit (node_modules/fs-minipass/index.js:175:22)
      at ReadStream.[resume] (node_modules/fs-minipass/node_modules/minipass/index.js:312:10)
      at Unpack.Pipe.ondrain (node_modules/fs-minipass/node_modules/minipass/index.js:64:37)
      at ReadEntry.<anonymous> (node_modules/tar/lib/parse.js:439:47)
      at ReadEntry.emit (node_modules/tar/node_modules/minipass/index.js:498:23)
      at ReadEntry.[flush] (node_modules/tar/node_modules/minipass/index.js:378:62)
      at ReadEntry.[resume] (node_modules/tar/node_modules/minipass/index.js:337:41)
      at WriteStream.Pipe.ondrain (node_modules/tar/node_modules/minipass/index.js:74:37)
      at WriteStream.emit (node_modules/fs-minipass/index.js:257:18)
      at WriteStream.[_onwrite] (node_modules/fs-minipass/index.js:345:16)
      at node_modules/fs-minipass/index.js:325:21

• requires: In the README maintain a compatibility matrix for the version of Solo with the version of Hedera, and other technologies. #503 (if we are going to do the upgrade to solve the problem)

[leninmehedy]

Remove async: https://github.com/hashgraph/solo/blob/f4f8527fc30eb8fe8d7900fdf57158119e2f0181/src/core/k8.mjs#L485C11-L488C14

It seems tar is probably failing to run when we are uploading a large file like build.zip (~90Mb) because of pipe-buffer issue. We could use dd/tee to first write to a remote file and then subsequently untar that file.

Similar issues: kubernetes/kubernetes#60140

[leninmehedy]

The error log in this ticket is confusing because copyTo and copyFrom doesn't use Zlib. However, we need to test thoroughly with a large file and find out why copyTo and copyFrom fails.

[matteriben]

This may have been fixed in the kubernetes 1.30 release (ga 4/17/24).
kubernetes/kubernetes#60140 (comment)
https://www.kubernetes.dev/resources/release/

Note:

• This fix was verified in 1.30.0-alpha
• Both the cluster and kubectl to use version 1.30

[jeromy-cannon]

A few more notes. 1.30+ is available. I think we would need to update Kind version, update the readme, and also update the kubernetes node client.

We could also add retry logic, here is the source code of the Java libraries:

• https://github.com/kubernetes-client/javascript : src/web-socket-handler.ts
• https://github.com/kubernetes-client/javascript : src/exec.ts

[jeromy-cannon]

@instamenta , on the copyFrom, just drop the z from the tar, we'll merge that, close this issue, then we can reopen this issue if it starts happening again.

[jeromy-cannon]

potential future solutions (if we need to reopen this ticket in the future), if we delay, then 1.30.0 deployments to GKE and Latitude will make this issue go away:

1. tar the file first, then send the bytes back/down and decompress locally (current version is tar'ing with compression and piping to stdout, which feels the buffer and creates instability)
2. make a list of files, and do each one individually one at a time (without tar)
3. add retry and resume logic (resume if multiple files)

[jeromy-cannon]

a different error, but copyFrom is failing more frequently

[swirlds-automation]

🎉 This issue has been resolved in version 0.30.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀