Cannot get gMSA krb ticket

[vaujo6y]

I have setup gMSA domained-joined and domainless without an issue on Windows. I started to repeat the same process for a Linux instance in the same way, but everything I do I can't get passed the following ecs task error Rpc error: code = Internal desc = ERROR: Cannot get gMSA krb ticket

The logging looks like:
cat credentials-fetcher.log

Thread 0: top of stack near 0x7ffb713a0968; argv_string=krb_ticket_refresh_thread
Thread 0: top of stack near 0x7ffb6bffea88; argv_string=truncate_file_logs
2024-09-10 16:36:34     INFO: Server listening on unix:/var/credentials-fetcher/socket/credentials_fetcher.sock
2024-09-10 16:36:34     INFO: CallDataCreateKerberosLease 0x7ffb6c02a980status: 0
2024-09-10 16:36:34     INFO: AddNonDomainJoinedKerberosLease 0x7ffb6c02b330status: 0
2024-09-10 16:36:34     INFO: RenewNonDomainJoinedKerberosLease 0x7ffb6c02bcf0status: 0
2024-09-10 16:36:34     INFO: CallDataDeleteKerberosLease 0x7ffb6c02c690status: 0
2024-09-10 16:36:34     INFO: CallDataHealthCheck 0x7ffb6c02d050 status: 0
2024-09-10 16:36:34     INFO: CallDataCreateKerberosArnLease 0x7ffb6c02d9d0status: 0
2024-09-10 16:36:34     INFO: RenewKerberosArnLease 0x7ffb6c02e3a0status: 0
2024-09-10 16:38:22     INFO: AddNonDomainJoinedKerberosLease 0x7ffb6c02b330status: 1
2024-09-10 16:38:22     INFO: AddNonDomainJoinedKerberosLease 0x7ffb6c02f2e0status: 0
INFO: prompt at 0x970ea702, 0x400,
INFO: prompt at 0x970ea702, 0x400,
2024-09-10 16:38:22     ERROR: getting FQDN from domain ip
2024-09-10 16:38:22     ERROR: getting FQDN from domain ip
2024-09-10 16:38:22     ERROR: getting FQDN from domain ip
2024-09-10 16:38:22     ERROR: getting FQDN from domain ip
2024-09-10 16:38:22     ERROR: getting FQDN from domain ip
2024-09-10 16:38:22     ERROR: getting FQDN from domain ip
2024-09-10 16:38:22     ERROR: Cannot get gMSA krb ticket
2024-09-10 16:38:22     INFO: AddNonDomainJoinedKerberosLease 0x7ffb6c02b330status: 2
2024-09-10 16:46:34     INFO: renewal started

cat credentials-fetcher.log
cat: credentials_fetcher.sock: No such device or address

journalctl -u credentials-fetcher

Sep 10 16:36:34 ip-10-X-3-X.domain.local systemd[1]: Starting credentials-fetcher.service - credentials-fetcher systemd service unit file....
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: credentials-fetcher daemon has started and running
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: on request failures check logs located at /var/credentials-fetcher/logging
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: krb_files_dir = /var/credentials-fetcher/krbdir
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: logging_dir = /var/credentials-fetcher/logging
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: unix_socket_dir = /var/credentials-fetcher/socket
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: Thread 0: top of stack near 0x7ffb71da1a18; argv_string=grpc_thread
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: grpc pthread is at 0x5637762ac7b0
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: krb refresh pthread is at 0x5637762a6da0
Sep 10 16:36:34 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: truncate file logs is at 0x5637762a8090
Sep 10 16:36:34 ip-10-X-3-X.domain.local systemd[1]: Started credentials-fetcher.service - credentials-fetcher systemd service unit file..
Sep 10 16:38:22 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: watchdog enabled with interval value = 5000000Using default cache: persistent:0:0
Sep 10 16:38:22 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: Using principal: svc-gmsakey-octopus@xxxxx
Sep 10 16:38:22 ip-10-X-3-X.domain.local credentials-fetcherd[4482]: Authenticated to Kerberos v5

host:
amazon/al2023-ami-ecs-hvm-2023.0.20240905-kernel-6.1-x86_64

If you could provide me with a better way to troubleshoot this issue, it would be much appreciated.

Thanks

[smhmhmd]

Hi @vaujo6y
cc: @palldavi

ERROR: getting FQDN from domain ip

We have fixes in this area, we used to do reverse DNS in the previous release.
We now use the Microsoft MSDS method here

[smhmhmd]

@vaujo6y
Until we make the new release, you can build credentials-fetcher on Ubuntu using the Docker files here

[smhmhmd]

@vaujo6y

I can't get passed the following ecs task error Rpc error: code = Internal desc = ERROR: Cannot get gMSA krb ticket

Since you are using ECS, you can set DHCP option set in your VPC. This works in the previous release. Please try and let us know.

[vaujo6y]

@vaujo6y Until we make the new release, you can build credentials-fetcher on Ubuntu using the Docker files here

Not having much luck on the compile

4.422 [ 47%] Built target proto-objects
4.437 1 warning and 2 errors generated.
4.446 make[2]: *** [api/CMakeFiles/cf_gmsa_service_private.dir/build.make:79: api/CMakeFiles/cf_gmsa_service_private.dir/src/gmsa_service.cpp.o] Error 1
4.446 make[2]: Leaving directory '/root/credentials-fetcher/build'
4.446 make[1]: Leaving directory '/root/credentials-fetcher/build'
4.446 make[1]: *** [CMakeFiles/Makefile2:247: api/CMakeFiles/cf_gmsa_service_private.dir/all] Error 2
4.446 make: *** [Makefile:149: all] Error 2

I will continue on with DHCP option set.

Thanks

[smhmhmd]

@vaujo6y Feel free to email me at samiull at amazon dot com, I can setup a chime meeting to screen share with you,

[smhmhmd]

Changes and RPM are at https://github.com/aws/credentials-fetcher/tree/fixes_for_DNS_and_distinguishedName/rpm
We plan to release them.

[mfann-or]

Any ETA on when this will be released into the Fargate baselines?

[smhmhmd]

We are planning to release in the first quarter of 2025