Use kernel 4.18 in EKS and ECS Amazon Linux AMIs to solve CFS throttling issues.

[willejs]

Apologies as this is kind of an Amazon linux 2 issue, but directly effects EKS and ECS.
Does anyone know when the Linux kernel version 4.18 will be shipped in Amazon Linux 2?
When it is, can the EKS and ECS images be rebuilt?

torvalds/linux@512ac99#diff-1c5364196d98130348bddabaad0a701f
The patch above should fix the issue with CFS quotas leading to process throttling. This should enable us to use limits everywhere again without setting off prometheus alerts or degrading performance in latency sensitive components!
kubernetes/kubernetes#67577

[zflamig]

+1 this, would be great for FUSE too that shipped in 4.18.

[BrianChristie]

According to an excellent detailed response from AWS Support, the sched/fair: Fix bandwidth timer clock drift condition patch [0] is merged into the Linux kernel version 4.14.y branch and is available as part of 4.14.95 release [1].

The latest EKS Optimized Amazon Linux 2 AMI has a Linux kernel version which includes the patch.

# uname -a
Linux ip-192-168-213-62.eu-west-1.compute.internal 4.14.97-90.72.amzn2.x86_64 #1 SMP Tue Feb 5 20:46:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Reference:
[0] torvalds/linux@512ac99#diff-1c5364196d98130348bddabaad0a701f

[1] Linux kernel patch 4.14.95
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/?h=linux-4.14.y&ofs=1000

I believe this issue can be closed.

[szymonpk]

I have switched all the nodes to 4.14.104-95.84.amzn2.x86_64, it didn't helped much. I still see throttled containers where CPU usage is minimal, mostly node-exporters.

[otterley]

It was subsequently reported that torvalds/linux@512ac99 introduced a regression. A patch to correct this can be found at https://lkml.org/lkml/2019/5/17/581 (not yet merged AFAICT).

[whereisaaron]

Not merged. The author, Dave Chiluk, is looking for support on the LKML to get attention to this patch, to get it reviewed and merged.

[willejs]

It looks like that patch is getting closer to being merged from that thread. I can imagine it's still probably months away from getting into the amazon linux build though.

[whereisaaron]

Reports of great improvements using 4.14.133 with 512ac99 and Dave Chiluk’s patch backported into it: kubernetes/kubernetes#67577 (comment) https://gist.github.com/PaulFurtado/ff6c67ec87416b66ba1c6fc70f7beec1

Hopefully test results like these will help it get merged soon.

[andrew-howden]

In our case we're looking for:

https://github.com/torvalds/linux/commit/79e9fed46038/

As we're exhausting the ephemeral TCP port range in containers.

Also in 4.18.

[hugoprudente]

@andrew-howden,

As per release-notes the Amazon Linux 2, base image for the EKS/ECS Optmized AMI's already have available the kernel 4.19.x[1]

I have upgraded my EKS Opmtized AMI with the 4.19.x kernel to confirm the version/patch available.

$ uname -r
4.19.72-25.58.amzn2.x86_64

With that I was able to confirm that the net-tcp: extend tcp_tw_reuse sysctl to enable loopback only optimization. patch that you required is available already on the 4.19 kernel provided by the AL2

Now is a question of time to the Amazon Linux 2 to start use the Kernel 4.19 as default, or you can use the https://github.com/awslabs/amazon-eks-ami, and build your own AMI with the AWS Supported Kernel as default.

Sadly the Dave Chiluk’s patch is not backported to neither versions of the kernel till now. Altought the version 512ac99 is available on both 4.19, 4.14

Reference
[1]: https://aws.amazon.com/amazon-linux-2/release-notes/

[alakesh]

Amazon linux extras provide kernel-ng which is a 4.19 based kernel.
$ sudo amazon-linux-extras install kernel-ng

[whereisaaron]

Dave Chiluk’s patch is en route to be in the 5.4 kernel (the next release now that 5.3 out), and should shortly be available in 5.4-rc1.
kubernetes/kubernetes#67577 (comment)

This has now been merged into Linus' tree and should be released with 5.4. I also just submitted it to linux-stable, and assuming that goes smoothly all distros that correctly follow stable process should start picking it up shortly.

[hugoprudente]

Hi all,

The Dave Chiluk’s patch was back-ported today, it's available from version v4.14.154 && v4.19.84 towards.

Now we just need to way to be available on AmazonLinux2 to yum update it and use the feature.

I'll keep track of it and let you guys know.

[hugoprudente]

Amazon Linux 2 just had rolled out the new kernel version 4.14.154 is available for usage.

[ec2-user@ip-172-31-38-202 ~]$ uname -a
Linux ip-172-31-38-202.eu-west-1.compute.internal 4.14.154-128.181.amzn2.x86_64 #1 SMP Sat Nov 16 21:49:00 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[f3d3Cz]

thanks mate, I've just created a new image based on amazon-eks-ami and that kernel got installed: amazon-ebs: ---> Package kernel.x86_64 0:4.14.154-128.181.amzn2 will be installed

…

On Wed, 27 Nov 2019 at 00:46, Hugo Prudente ***@***.***> wrote: Amazon Linux 2 just had rolled out the new kernel version 4.14.154 is available for usage. ***@***.*** ~]$ uname -a Linux ip-172-31-38-202.eu-west-1.compute.internal 4.14.154-128.181.amzn2.x86_64 #1 SMP Sat Nov 16 21:49:00 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#175?email_source=notifications&email_token=ALD7MZTKC4YK4ZWESQDNRTDQVUSE5A5CNFSM4G2Q4FT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFGBSRY#issuecomment-558635335>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALD7MZWQFKZZD6GNG5PSYEDQVUSE5ANCNFSM4G2Q4FTQ> .

[juniorz]

EKS AMIs have just been released with Kernel 4.14.154

sh-4.2$ uname -a
Linux ip-10-0-0-35.ec2.internal 4.14.154-128.181.amzn2.x86_64 #1 SMP Sat Nov 16 21:49:00 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

See ami-087a82f6b78a07557 in us-east-1 for example.

[mikestef9]

Closing this issue as resolved with the release of the latest EKS AMI. See release notes here

https://github.com/awslabs/amazon-eks-ami/releases/tag/v20191213