Symlinks are flattened in the layers because of a SAM bug

[bubba-h57]

Here is a link to the open issue over at aws/aws-sam-cli#878

The bottom line is that sam invoke local fails to unpackage symlinks properly, which means you can not test anything locally that relies on a layer (or function for that matter) that has symlinks.

This is only an issue for local invokation. Everything unpackages properly in the cloud. However, it is a major aggrevation for anyone who is expecting to test/debug localy with SAM.

[mnapoli]

If I recall correctly this is a problem in the new runtime because of the symlinks right? (#132)

On aws/aws-sam-cli#878 they say that files are copied instead of symlinked. Don't the binary still work after being copied? Or maybe they don't work because they cannot load related libraries?

Can you confirm that sam local and the new runtimes are incompatible until they fix the issue in SAM?

[bubba-h57]

Confirmed that the new runtim is incompatible with sam local. We have also confirmed that everything works as expected in the cloud.

Technical Details
The unzip they are using flattens the symlink effectively making it nothing more than a text file. This effects us in two ways.

1. The /opt/bin/* symlinks that should point to /opt/bref/bin/* are flat text files, not binaries nor scripts.
2. The shared libraries we are building against use symlinks extensively to track versions of the shared object. LIke so:

lrwxrwxrwx 1 sbx_user1051  495      16 Jan  8 17:15 libcurl.so -> libcurl.so.4.5.0
lrwxrwxrwx 1 sbx_user1051  495      16 Jan  8 17:15 libcurl.so.4 -> libcurl.so.4.5.0
-rwx------ 1 sbx_user1051  495  572239 Jan  8 17:15 libcurl.so.4.5.0
lrwxrwxrwx 1 sbx_user1051  495      19 Jan  8 17:17 libsodium.so -> libsodium.so.23.1.0
lrwxrwxrwx 1 sbx_user1051  495      19 Jan  8 17:17 libsodium.so.23 -> libsodium.so.23.1.0
-rwx------ 1 sbx_user1051  495 2457307 Jan  8 17:17 libsodium.so.23.1.0

However, the actualy results on sam local look like:

-rwx------ 1 sbx_user1051  495      16 Jan  8 17:15 libcurl.so
-rwx------ 1 sbx_user1051  495      16 Jan  8 17:15 libcurl.so.4
-rwx------ 1 sbx_user1051  495  572239 Jan  8 17:15 libcurl.so.4.5.0
-rwx------ 1 sbx_user1051  495      19 Jan  8 17:17 libsodium.so
-rwx------ 1 sbx_user1051  495      19 Jan  8 17:17 libsodium.so.23
-rwx------ 1 sbx_user1051  495 2457307 Jan  8 17:17 libsodium.so.23.1.0

Most code is going to link against the base library like libcurl.so or against that major version of the library libcurl.so.4 which are symlinked against a full version like libcurl.so.4.5.0. So when PHP attemts to link to the libcurl.so it finds a text file instead and everything falls over.

[mnapoli]

That will be a big problem unfortunately 🤔

Do you think there is any way we can avoid using symlinks in the (new) runtime you are working on? At least for the first versions of the runtime while we wait for the issue in SAM to be resolved.

(by the way slightly related aws/aws-sam-cli#477 we don't seem to be affected by this given what you said above, but I'm sharing the link just in case)

[bubba-h57]

I have the new runtime working with sam invoke local for now. We basically make copies of everything and duplicate (or triplicate) the space used. But I am able to keep it under 100M on disk.

Longterm, that bug will be debilitating is we stick with AWS SAM. Symlinks are our best weapon against bloat in the layer.

FWIW, I do not believe I had the same problems with Serverless.

[nealio82]

@mnapoli @bubba-h57 can this be closed? (related to / fixed by aws/aws-sam-cli#1140..? / no longer relevant as we're back on serverless...)

[bubba-h57]

I do not see any reason to leave it open.

[mnapoli]

Should we remove hard copies now and keep the symlinks? That should make layers much lighter right?

[bubba-h57]

@mnapoli Abso-stinking-lutely!!!!!

[mnapoli]

Cool!

[mnapoli]

I have opened #509 to try to address this.

[mnapoli]

Closing this issue, this is no longer an issue in Bref v2.