Could not upload to S3 with ECS Fargate

[zeroiszero]

Hi, I am running WordPress with ECS Fargate and TaskRoleArn defined in Task Definition

Version of AWS SDK for PHP?

aws/aws-sdk-php": "~3.18"

Version of PHP (php -v)?

PHP 7.3.6 (cli) (built: Jun 12 2019 20:40:46) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.6, Copyright (c) 1998-2018 Zend Technologies

What issue did you see?

Fatal error: Uncaught GuzzleHttp\Exception\ConnectException: cURL error 7: (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) in /var/www/html/wp-content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:185 Stack trace: #0 /var/www/html/wp-content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(149): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array) #1 /var/www/html/wp-content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(102): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory)) #2 /var/www/html/wp-content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(181): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlMultiHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory)) #3 /var/www/html/wp-content/plugins/s3-u in /var/www/html/wp-content/plugins/s3-uploads/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 185

Steps to reproduce

Run S3 putObject as in this example.

Credentials not provided as I assume the SDK will retrieve from the IAM role.

Also tried the task role attached with the policy AmazonS3FullAccess. Still the same error.

[diehlaws]

Hi @zeroiszero, thanks for reaching out to us. cURL Error 7 is caused by a failure to connect to the target host, which suggests there is something blocking or dropping the connection from your Fargate container to S3. Are you able to reach the S3 endpoint for the region in which your bucket resides using your container without using the AWS SDK for PHP? E.g. using curl or nc from the shell, or using fsockopen() or a GuzzleHttp client within PHP.

As a side note, I see you're using version 3.18 of the AWS SDK for PHP. While this shouldn't affect your ability to connect to S3, this version was released in April of 2016 and is 90 minor versions behind the latest release. I strongly suggest updating the SDK to a newer version as there have been quite a few bug fixes and improvements implemented between these two versions that should result in an overall better experience using the SDK.

[zeroiszero]

Hi, @diehlaws

Checked. No issue connect to S3 enpoint. Updated the SDK version as well, still the same.

As I traced the code, its turn out that cURL error is coming from the SDK when its trying to connect http://169.254.169.254/latest/meta-data/iam/security-credentials to retrieve the credentials / permissions from the IAM attached to Fargate task.

[zeroiszero]

Solved.

It was caused by php-fpm configuration that deleted the environment variables when docker started php-fpm processes.

AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is required to retrieve credentials from Task Role as mentioned here.

To fix this, set clear_env to no in php-fpm configurations:

; Clear environment in FPM workers
; Prevents arbitrary environment variables from reaching FPM worker processes
; by clearing the environment in workers before env vars specified in this
; pool configuration are added.
; Setting to "no" will make all environment variables available to PHP code
; via getenv(), $_ENV and $_SERVER.
; Default Value: yes
clear_env = no

[mvn-bachhuynh-dn]

hi @zeroiszero , I try like you did, but still error
aws/aws-sdk-php-laravel#176

[Steven-Wright]

To fix this, set clear_env to no in php-fpm configurations:

I am so glad I found this post, I was really beginning to lose hope.

I'm not sure if it matters much, but rather than preventing FPM from clearing environment variables you could pass AWS_CONTAINER_CREDENTIALS_RELATIVE_URI specifically.

I added the line to /etc/php-fpm.d/www.conf:

env[AWS_CONTAINER_CREDENTIALS_RELATIVE_URI] = $AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Below the section:

; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from
; the current environment.
; Default Value: clean env
;env[HOSTNAME] = $HOSTNAME
;env[PATH] = /usr/local/bin:/usr/bin:/bin
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp