Build Memory Consumption

[kixorz]

Describe the bug

Building of the libaws-cpp-sdk as a shared lib requires more than 4GB of memory.

Scanning dependencies of target aws-cpp-sdk-ec2
[ 28%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/EC2Client.cpp.o
virtual memory exhausted: Cannot allocate memory

This is from a machine with 4GB of physical memory.

Expected behavior

It should be possible to build the SDK with much less memory.

Current behavior

The SDK requires more than 4GB of memory to build.

Steps to Reproduce

Test building the library with 2 or 4GB of memory.

Possible Solution

One solution could be splitting EC2Client.cpp.o into multiple smaller objects.

AWS CPP SDK version used

aws-sdk-cpp-1.7.266

Compiler and Version used

gcc (Debian 8.3.0-6) 8.3.0

Operating System and version

Linux 4.19.0-14-arm64

[jmklix]

Building the whole sdk can use a lot of memory as you have noticed. Can you try using the BUILD_ONLY flag with cmake to reduce the size of what you are trying to build? You will only need to select the services that you want to use.

It might look something like this: -DBUILD_ONLY="s3;ec2;ebs"

[kixorz]

Thanks for the response. I wasn't immediately aware of this switch option.
However, I don't believe it will solve the issue with the EC2 service client requiring more than 4GB of RAM to build.

[jmklix]

If you haven't already can you please try building with that cmake flag. We are currently trying to improve the build process. More details about this can be found here. Also any feedback or suggestions are welcome.

[kixorz]

I tested the suggested approach with -DBUILD_ONLY="ec2" with the same result as before. The EC2Client.cpp.o is too large to compile with 4GB of RAM. The SDK builds successfully with 8GB of RAM so the minimum build machine memory requirement is 8GB of RAM.

[  5%] Building CXX object aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/EC2Client.cpp.o
virtual memory exhausted: Cannot allocate memory
make[5]: *** [aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/build.make:64: aws-cpp-sdk-ec2/CMakeFiles/aws-cpp-sdk-ec2.dir/source/EC2Client.cpp.o] Error 1

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.