NoCredentialProviders

[deacix]

I get this error:
"Failed to initialize logging driver: NoCredentialProviders: no valid providers in chain"
if i try to start a stack with awslogs driver. Single docker container call like this:

docker -D run --rm -it
--log-driver=awslogs
--log-opt awslogs-region=eu-central-1
--log-opt awslogs-group=docker-log
--log-opt awslogs-stream=hello-world hello-world

works fine.

docker-compose yml log config:

log_driver: "awslogs"
log_opt:
    awslogs-region: eu-central-1
    awslogs-group: docker-log
    awslogs-stream: hello-world

[dnephin]

According to https://docs.docker.com/engine/reference/logging/awslogs/#credentials the credentials only need to be provided to the daemon, so I'm not sure why the difference in client matters.

Where is the docker daemon running? Do you notice any difference in the docker inspect of the container created by docker run vs the container created by docker-compose ?

[unoexperto]

I get this error despite the fact that I created ~/.aws/credentials file per your documentation. How would you recommend to pinpoint what exactly is missing ?

Following sample application in Scala works and can find credentials file

    val conf = ConfigFactory.load().getConfig("case")
    val aws_region = RegionUtils.getRegion(conf.getString("aws_region"))
    val client = new AWSLogsClient(new ProfileCredentialsProvider("default"))
    client.setRegion(aws_region)
    val req = new PutLogEventsRequest()
    req.setLogGroupName("case")
    req.setLogStreamName("api-server")
    val arr: util.Collection[InputLogEvent] = new util.ArrayList[InputLogEvent]()
    val event = new InputLogEvent()
    event.setMessage("test message!")
    event.setTimestamp(ZonedDateTime.now().toEpochSecond * 1000)
    arr.add(event)
    req.setLogEvents(arr)
    val regs = client.putLogEvents(req)
    println(regs)

I'm on Docker 1.9.1

[mayeco]

ping

[dnephin]

It's important to remember that this credential file needs to be made available to the docker engine not the client. It's the engine (the daemon) that is going to connect to aws.

If you create that file as a user, it may not be available to the engine. If you're running docker-machine and the engine is in the VM, you'll need to move that credentials file into the VM for the root user.

[chdsbd]

This is still a problem. Even with moving the credentials to /root/.aws/credentials, I still get this error.

[jasco]

In my case too the above error was caused by not having service provider credentials properly set. (I do not configure default credentials but rather set a profile as needed.)

If you are having trouble figuring out whether the credentials are configured, you might install the python awscli module and invoke something like aws ec2 describe-instances. The error messages from that application seem clearer to me.

Before using docker-compose I set the environment with docker-machine. Both docker-machine ls and docker-machine env $CONTAINER_NAME produced errors even though neither command seemed like it should need credentials. It caused some confusion when those seemingly innocuous commands worked one moment and stopped the next because I had changed to a terminal window without AWS_PROFILE defined.

Neither of the above commands required service provider credentials when I used the DigitalOcean driver.

For reference, the error I observed on invoking env was

Error checking TLS connection: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[sharq1]

I think I tried everything - setting up ~/.aws/credentials and ~/.aws/config for both myself and root, setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION as env variables for myself and root and on the project level in .env file, adding that to docker-compose.yml, reviewed IAM permissions, but getting NoCredentialProviders over and over.

Why can't compose config accept key id and secret just as it accepts region??
I would pass these from my .env file and problem solved. But now I need to find another way for logging...

[AlexZeitler]

@sharq1 Did you find a solution? I'm facing the same issue.

[jasco]

@AlexZeitler have you confirmed the credentials are working correctly with the AWS CLI?

[AlexZeitler]

@jasco Yes, they're working fine.

[AlexZeitler]

This tutorial (using the systemd part on Ubuntu 18.04) did the trick: https://wdullaer.com/blog/2016/02/28/pass-credentials-to-the-awslogs-docker-logging-driver-on-ubuntu/

[MichaelLeeHobbs]

Still an issue.
Linux dockerSwarm01.vns.local 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Docker version 18.09.0, build 4d60db4

Works without issue: eval $(aws ecr get-login --no-include-email --region us-gov-west-1)

echo $AWS_ACCESS_KEY_ID
A.....
echo $AWS_SECRET_ACCESS_KEY
h.....

cat ~/.aws/config
[default]
region = us-gov-west-1

cat ~/.aws/credentials
[default]
aws_access_key_id=A....
aws_secret_access_key=h....

Jan 15 17:00:03 dockerSwarm01.vns.local dockerd[30809]: time="2019-01-15T17:00:03.373268493-05:00" level=error msg="fatal task error" error="starting container failed: failed to initialize logging driver: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" module=node/agent/taskmanager node.id=1f5t37qqgdpwz3qimiwpteqp0 service.id=9cb8od8kj177ak74kokxtp9bu task.id=kg21is0n80pic9rmojb9fkdhs Jan 15 17:00:36 dockerSwarm01.vns.local dockerd[30809]: time="2019-01-15T17:00:36.531371548-05:00" level=error msg="fatal task error" error="starting container failed: failed to initialize logging driver: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" module=node/agent/taskmanager node.id=1f5t37qqgdpwz3qimiwpteqp0 service.id=9cb8od8kj177ak74kokxtp9bu task.id=rt4zrs4fz3aepau5ybn02eylb Jan 15 17:00:54 dockerSwarm01.vns.local dockerd[30809]: time="2019-01-15T17:00:54.714278599-05:00" level=error msg="fatal task error" error="starting container failed: failed to initialize logging driver: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors" module=node/agent/taskmanager node.id=1f5t37qqgdpwz3qimiwpteqp0 service.id=9cb8od8kj177ak74kokxtp9bu task.id=lhn4wk5l6xcexq8ug0jbdtkbs

[ehrlichja]

Same issue, here. Credentials in /root/.aws/credentials but not picked up (Fedora Atomic 28).

[felix-xuu]

hi @ehrlichja , maybe you can try:

1. create file /etc/systemd/system/docker.service.d/aws-credentials.conf
2. add content in file aws-credentials.conf:
   [Service]
   Environment="AWS_ACCESS_KEY_ID=<aws_access_key_id>"
   Environment="AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>"
3. exec sudo systemctl daemon-reload to reload daemon config
4. exec sudo service docker restart to restart docker
5. exec systemctl show --property=Environment docker to see whether the env variables existed.

[jbq]

This blog post helped me: https://wdullaer.com/blog/2016/02/28/pass-credentials-to-the-awslogs-docker-logging-driver-on-ubuntu/ previous comment is right, but the blog post covers more cases.

[andreclaro]

Hello, Is there any plan to fix this issue? Thanks!

[adiii717]

any update on this issue?

[justengland]

FWIW: I am using the SSM Agent on an onprem instance, then I set the user in the docker service file.
vi /usr/lib/systemd/system/docker.service

[Service]
User=root

[AusPrinzip]

bump

[AusPrinzip]

Please this is still relevant, any help?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not recent activity during the stale period.

[paulo-digital]

The bot close an big issue with NO fixings. Still an issue !

[rajeshkp]

we are still observing this issue

[FrancescoSaverioZuppichini]

same here

[phitoduck]

Same issue :( Mine is on a mac. It looks like people have found workarounds for Ubuntu. I develop on a mac, but then run docker containers on all sorts of OS-es. Strictly speaking, I don't need awslogs to work from my mac, but it makes the feedback loop during development soo much tighter if I can be running my stack locally.

[MusHusKat]

I have the same issue. While I was able to get this working on our production ec2 instance, I can't replicate this locally on Mac which I use for dev work.

[SplashMatrix]

Any workaround for docker in Windows?
WSL 2
ubuntu 18

[anking]

Not working on windows trying to deploy to ECS:
NoCredentialProviders: no valid providers in chain. Deprecated.
Very frustraiting

[StevePorter92]

I've tried multiple ways to get this to work with no luck on Windows without using a workaround.

The only way I have managed to get this to work is to use the aws-vault server mode to spin up a local ec2 metadata server that the Docker daemon picks up. It works but isn't ideal

aws-vault exec profile --server -- docker-compose up

As far as I can tell, it would seem the AWS logging driver is not supported on windows docker/for-win#9684

[kamdar]

Just posting this in hopes it can help someone. We have ec2 instances that are running imds V1, and we needed to switch to imds v2. When we switched to imds v2, we started to see the error in various parts of the code. If you run the below command:
aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,MetadataOptions.HttpTokens]"

And instead of "optional" you see "required", then you are on V2. If you switch to optional, then it might help you.