[ECS/CodeDeploy] [request]: Support uncompressed streaming zip

[sveniu]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Please fix CodePipeline's Amazon ECS (Blue/Green) action so that it can unzip files that are produced by streaming and use STORE as the archive method. These files cannot be extracted by Java's standard java.util.zip, which appears to be in use by the action.

Which service(s) is this request for?
AWS CodePipeline's Amazon ECS (Blue/Green) action, affecting CodeDeploy and ECS.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
The CodePipeline action Amazon ECS (Blue/Green) does not support zip files that are created by streaming and use STORE (not DEFLATE) as the archive method. The error produced by the action is:

Exception while trying to read the task definition artifact file from: <source action name>

The zip file is produced by Terraform's archive_file provider, but it has been patched with 64kramsystem/terraform-provider-archive-dev@347a1af to ensure that file metadata is normalized, i.e. that they are the same across platforms. To do this, it sets mtime and permissions, but also forces the STORE archive method to avoid minute differences in compression output across platforms.

See hashicorp/terraform-provider-archive#47 for details on the patch.

The combination of Go's streaming zip writing (used by archive_file under the hood) and the STORE archive method, seems to be the the core of the problem.

Speculation: The AWS CodePipeline Amazon ECS (Blue/Green) action may be using the standard java.util.zip package to extract the archive containing appspec.yaml and taskdef.json for CodeDeploy and ECS respectively.

A test program using java.util.zip reproduces the problem:

java.util.zip.ZipException: only DEFLATED entries can have EXT descriptor
        at java.base/java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:311)
        at java.base/java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:123)
        at com.meh.test.UnzipFiles.unzip(UnzipFiles.java:30)
        at com.meh.test.UnzipFiles.main(UnzipFiles.java:17)

Relevant links:

• https://bugs.openjdk.java.net/browse/JDK-8143613
• https://groups.google.com/g/golang-nuts/c/0iae5Ng-I-0
• https://github.com/30x/zipper
• https://stackoverflow.com/questions/47208272/android-zipinputstream-only-deflated-entries-can-have-ext-descriptor

Are you currently working around this issue?
We have reverted to not use the patched archive_file Terraform provider, and rather dealing with file permission problems manually, which sucks. The main challenge is differences in default umasks between OSX, Ubuntu and other Linux distributions.

I have reported the issue upstream to the patched Terraform provider too: 64kramsystem/terraform-provider-archive-dev#1

Attachments
The attached test.zip has been created by the patched archive_file Terraform provider. It reproduces the CodePipeline exception, and the java.util.zip exception.

test.zip
UnzipFiles.zip

$ javac UnzipFiles.java
$ java -cp . UnzipFiles
java.util.zip.ZipException: only DEFLATED entries can have EXT descriptor
        at java.base/java.util.zip.ZipInputStream.readLOC(ZipInputStream.java:311)
        at java.base/java.util.zip.ZipInputStream.getNextEntry(ZipInputStream.java:123)
        at UnzipFiles.unzip(UnzipFiles.java:23)
        at UnzipFiles.main(UnzipFiles.java:12)

$ java --version
openjdk 11.0.8 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode)

Please note: While the zip only contains a dummy file test1.txt with body hello, it produces exactly the same error as a perfectly valid pair of appspec.yaml and taskdef.json files.