ARM64/Graviton support

[jasonmccallister]

AWS launched ARM64 support on Lambda (https://aws.amazon.com/blogs/aws/aws-lambda-functions-powered-by-aws-graviton2-processor-run-your-functions-on-arm-and-get-up-to-34-better-price-performance/).

Would be useful to get Bref functions and container images setup with ARM64 variants... Happy to assist where I can.

[mnapoli]

AFAIK PHP isn't running best on ARM (e.g. JIT not supported). I may be wrong, but it might be worth digging into that?

[deleugpn]

I think the first step is to wait Cloudformation support? aws-cloudformation/cloudformation-coverage-roadmap#926

[jasonmccallister]

@deleugpn for Lambda layers yes, but I think there should not be anything holding up pushing the ARM images to Docker Hub for container lambda functions?

[dytyniuk]

@deleugpn people claim in aws-cloudformation/cloudformation-coverage-roadmap#926 (comment) that CloudFormation API already supports the Architecture property.

@mnapoli maybe, this helps to melt your concern about running PHP on ARM -- https://aws.amazon.com/tr/blogs/compute/improving-performance-of-php-for-arm64-and-impact-on-amazon-ec2-m6g-instances/

[mnapoli]

@dytyniuk definitely, thank you. I was wrong, ARM can be interesting for PHP.

Also relevant: https://twitter.com/sebastienhouze/status/1443742924534132843

And that ARM JIT support is coming at least with 8.1

Also FYI: CloudFormation support is in (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html#cfn-lambda-function-architectures) and we're planning to add support in Serverless Framework early next week.

[willvincent]

So if the roadblocks to this were cloudformation and the serverless framework supporting arm - both of which now do.. is there any remaining issue or is this good to go now? And if so.. how so? 😁

I'm about to dig into a new project and I think I'd like to use the graviton lambdas if possible.

[deleugpn]

Compiling php is not a trivial process. I've been looking into simplifying it and only now realized that remi-collect does not support aarch64 builds which put a wrench in my experiment.

[dytyniuk]

Hey @deleugpn
Maybe, this repository can help -- https://repo.drpixel.fr/enterprise/7/drpixel-php74/aarch64/ ?

[deleugpn]

I have been pointed that address but I haven't figured out what to do with it. I tried enabling it and installing packages following remi-collect naming convention and no luck

[deleugpn]

I managed to install php with amazon-linux-extras but its locked into php 8.0.8 which is a few months behind. I'm not sure it would be a good switch as it would delay patch releases beyond our control. If anyone have a Dockerfile recipe on how to install php using the al2-arm64 base image and wants to share, I'd be grateful!

[deleugpn]

Another thing that might be relevant, if I understand https://repo.drpixel.fr/ correctly, it only support PHP 8.0.1, which is 10 point-releases behind so even if I did manage to install with it, it seems it would be very slow to move forward with PHP upgrades. I'm open to suggestions and considering our options. I might have to dig deep into our current compile php ourselves and abort the attempt to simplify it with a pre-package distribution

[dytyniuk]

@deleugpn yeah, now I feel your shoes. At least, to some extent.

IMHO, any effort to reuse pre-built binaries for this or that repository will bring a concrete PHP version and lock further updates. From this point of view, layers' updates will be postponed until maintainers of the repository incorporate an update.

Despite the pain of the compilation, I think, this is the only way to provide best layers.

[deleugpn]

remi-collect offers blazing-fast compilation and even release-candidate binaries. The only problem is that there's no arm64 support there. Swapping our current (complex) compilation process with a much-simpler process (https://github.com/deleugpn/bref-runtime-internal/blob/main/PHP80.Dockerfile#L6-L11) without arm64 support kinda feels like touching what doesn't need to be touched. OTOH, if I do use remi-collect as a simplified packaging and then use amazon-linux-extras for arm64, I fall into having to maintain 2 different processes that are not consistent with each other and put the Graviton layers behind on point releases which could be good, but kinda awkward.

[dytyniuk]

@deleugpn I might be wrong, but I thought the point of a compilation is to install everything to the same path (/opt) and maintain all the relations and links when binaries and modules are extracted and zipped into a layer.

[deleugpn]

The folks at Vercel are doing some amazing stuff that works great (https://github.com/deleugpn/bref-runtime-internal/blob/main/php80.build.sh#L31).

I copied this strategy from https://github.com/juicyfx/libphp/blob/master/libs/amazon-linux-2-v80/Makefile#L53

[dytyniuk]

@deleugpn okay, I understand there's a way to extract an artefact from any place in the container. But, is an explicit listing more convenient rather than copying every binary/module installed on a build time? Imagine you have an auxiliary module of a file built for an extension, and it's missing from your list. How long will it take to pinpoint a root cause of a broken extension?

[deleugpn]

I guess that's subjective. I have some extensive experience on PHP, but I find the code compiling PHP hard to read / understand / maintain. The copy/paste strategy seems much more straightforward / easier to read and understand. It gets boring, but you can always run ldd /usr/lib64/php/modules/curl.so and find the necessary shared libraries to build the layer. I'm not sure I will replace Bref build process at all. This is just wild speculation while I'm studying / learning while trying to build Graviton support.

[georgeboot]

How can I help getting this shipped?

PHP 8.1 just released which has native support for the JIT on ARM.

[deleugpn]

I've been working on #1078 for quite a while now but docker-compose bug on ARM prevents it from building arm layers at the moment. In the meantime I'm trying to finish up how to integrate the console layer into the new proposal so that we can start beta test it while we wait on docker/compose#8804

If you're interested, I can post the arn on the x86 layers that needs some QA testing before we're confident to move forward with the new runtime.

[allan-simon]

@deleugpn interested in testing the x86 layers

[deleugpn]

arn:aws:lambda:us-east-1:978790411843:layer:prototype-arm64-php74-function:1
arn:aws:lambda:us-east-1:179453031647:layer:prototype-arm64-php74-fpm:1

[allan-simon]

@deleugpn do you have them in 8.0 by chance ? our code base use some 8.x-only features :)

[deleugpn]

I will look into building it this week. Sorry for the short answer, I'm on holidays between Christmas and new years

[allan-simon]

no problem, enjoy your vacations :)

[deleugpn]

sorry the long time without replies. I have actually tried to release ARM64 prototype layers across regions multiple times in the past 3 weeks and no luck as of yet.

Some exotic AWS regions seems to be bugged when trying to publish Graviton layers because it always gives "The security token in the request is invalid" even though the build process is a perfect replica between x86 and arm64. I tried disabling these regions and now there is an unknown error crashing the build process just after the layers are published which I'm still unable to track down. In the meantime, docker-compose is still bugged on arm64 when trying to build using buildkit, so builds are much slower for arm64.

Unfortunately the progress here is a bit slow as there's a lot of moving parts that are still not working well with arm64 and sometimes don't even give clear errors.

[allan-simon]

thanks for keeping us updated , especially as it looks like a very tedious and "death by a thousand cut" work , hope you will succeed :)

[jasonmccallister]

@deleugpn I'm going to have some focus time at work to dedicate to this, happy to collab on what you have done so far and help push this over the finish line.

[wlami]

Some exotic AWS regions seems to be bugged when trying to publish Graviton layers because it always gives "The security token in the request is invalid" even though the build process is a perfect replica between x86 and arm64.

Please note that Graviton2 processor support for Lambda is not available in all regions (see):

You can use Lambda Functions powered by Graviton2 processor today in US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Frankfurt), Europe (Ireland), EU (London), Asia Pacific (Mumbai), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo).

@deleugpn Are there any errors when deploying in the supported regions?

[mnapoli]

We have beta ARM layers ready to test 🎉

I've started benchmarking them at brefphp/benchmarks#5, but it benchmarks cold starts (which get slightly faster) and warm executions which are really fast anyway. So I'd love to get numbers from real applications, as well as see if these layers miss anything (e.g. extensions).

Anyone interested in testing ARM layers?

Here's how to proceed:

1. Upgrade to Bref v2 (dev version): "bref/bref": "v2.x-dev" in composer.json (and run composer update)
2. In serverless.yml, add architecture: arm64 on the function you want to test (or add it in provider: to apply to all functions)
3. In serverless.yml, add arm- as a prefix for the layer, for example ${bref:layer.arm-php-80} or ${bref:layer.arm-php-80-fpm} (note: ARM layers only exist for PHP 8.0)
4. Deploy

I hope I haven't forgotten anything. Let me know if you see improvements to the execution time. If you can, please let us know how many invokes you tested with (if you tested with 10-100 invocations then the numbers might not be very reliable).

If you want to measure cold starts specifically, here is a CloudWatch Insights query you can run:

filter @type = “REPORT” and @initDuration
| stats
 count(@type) as count,
 min(@billedDuration) as min,
 avg(@billedDuration) as avg,
 pct(@billedDuration, 50) as p50,
 max(@billedDuration) as max
by @log
| sort @log

[mnapoli]

Here's a reminder that you can test ARM runtimes right now. Switch to v2 in composer.json:

        "bref/bref": "v2.x-dev"

Then composer update. Update serverless.yml to use ARM runtimes:

functions:
    api:
        handler: public/index.php
+        architecture: arm64
        layers:
-            - ${bref:layer.php-80-fpm}
+            - ${bref:layer.arm-php-80-fpm}
        events:
            -   httpApi: '*'

There shouldn't be any big breaking change in v2 worth mentioning here (upgrade guide here).

I am updating the benchmarks to include ARM results: brefphp/benchmarks#5

Today I managed to slim down ARM layers drastically (brefphp/aws-lambda-layers#33). Cold starts are slightly improved for ARM runtimes.

Still it doesn't seem like PHP runs faster on ARM, but my benchmarks are very limited. Real apps would be much more useful here.

Also one last thing: APCu is now provided in ARM layers. That was the last missing thing.

---

Moving forward for Bref v2, do you think ARM layers should be marked as "beta", or as "stable"? Given the situation, I'd feel more comfortable with marking them as "beta", but if some of you start using them and are happy with them, we could mark them stable.

[SerkanYildiz]

Hi @mnapoli,
Thanks for your great work. I've an application running on aws lambda / bref & symfony. The latter prevents me from installing the development version, therefore I couldn't test this. If you've plans to support (or I can do something to help you here) symfony with ARM layers, lmk. Thanks all

[mnapoli]

@SerkanYildiz If you have requirement conflicts, a quick and dirty option is:

        "bref/bref": "v2.x-dev as 1.99.99"

[mnapoli]

I pushed some documentation in ceb4e64.

I also tagged Bref 2.0.0-beta1, which lets you use the ARM layers: https://github.com/brefphp/bref/releases/tag/2.0.0-beta1

I will also tag new versions of the Symfony and Laravel bridge so that you can install the beta without conflicts.

I'll be closing this one, please open new issues if you see any issues with them! Thanks all for your involvement!

[mnapoli]

And an update just 2 hours later: it turns out we'll be able to support PHP 8.1 with Graviton too 🎉

Coming up in brefphp/aws-lambda-layers#40

[shouze]

Support of php 8.2 on Graviton still miss me but part of the game, about 6 months to wait for now, not so long.

[aran112000]

Support of php 8.2 on Graviton still miss me but part of the game, about 6 months to wait for now, not so long.

I've raised a re:Post here in the hope if enough of us star it, they might not take the normal 6 months to introduce support:
https://repost.aws/questions/QUlGjGHlhgQNWukZ_IB1cw9g/add-php-8-2-to-amazon-linux-extras

[shouze]

@aran112000 very good initiative, I’ve upvoted.

[pesseyjulien]

Hi,

thanks for the work !

Quick question, do we also need to add 'arm' to the extra layer.

Exemple :

${bref-extra:redis-php-81} to ${bref-extra:redis-arm-php-81} ?

Thanks in advance for your help,
Julien

[mnapoli]

@pesseyjulien ARM extensions aren't built yet (it will take a few weeks probably, we need to solve a few things first), and yes I do think we need to rebuild all of them for ARM unfortunately.

[eexit]

Hello,

Would it be possible to provide the ARM version of the Docker bref/fpm-dev-gateway image?
Also, why changing the image name for different arch? DockerHub natively handles multi-arch images...

Thanks!

[mnapoli]

@eexit This image will no longer be useful in v2. I am working on the v2 blog post, but in the meantime here is a link that should explain what to do: https://github.com/brefphp/bref/blob/v2/docs/upgrading/v2.md#development-docker-images

Also, why changing the image name for different arch? DockerHub natively handles multi-arch images...

Interesting question 🤔

Originally the images/layers were to be very different, so having different Docker images made sense. Now I get why you are asking.

I could imagine the scenario where someone deploys to x86, and wants to run that env locally on Mac M1 (for example) -> they don't want to run the ARM image locally.

I'm not 100% sure that what I'm saying is a good argument, we could still consider merging all images into one, feel free to weigh in everyone. I don't think it will block v2, we can do that later.

[eexit]

Hello @mnapoli,

Thanks for your quick answer! I spent last evening trying to improve the performance of my PHP stack on my Mac M1, so that's the reason I'm asking 😃

Originally the images/layers were to be very different, so having different Docker images made sense. Now I get why you are asking.

I could imagine the scenario where someone deploys to x86, and wants to run that env locally on Mac M1 (for example) -> they don't want to run the ARM image locally.

Docker automatically pulls the right arch:

When running an image with multi-platform support, docker automatically selects the image that matches your OS and architecture.

https://docs.docker.com/build/building/multi-platform/

This means, you can have the very same image name, this prevents per-arch stack config 💌 .

I'll give a shot to the :2 image then.
Thanks!

[mnapoli]

Docker automatically pulls the right arch:

Yes but that's what I wanted to avoid, because you'd be working locally with the ARM image and deploying to x86 Lambda.

[shouze]

Docker automatically pulls the right arch:

Yes but that's what I wanted to avoid, because you'd be working locally with the ARM image and deploying to x86 Lambda.

@mnapoli I'm not agree. It's nice & docker-ish way to have multi arch support for a given image, and under the same tag. Tag names shouldn't be used to specify an arch.
In the same time, it's the team decision to choose if depending of the local development/developer laptop arch, you want to force to use a sticky arch or let docker dynamically use x86 or arm64 depending of the arch of this developer laptop. And it works by design into docker.

For example, in docker-compose.yaml:

1. If you want to pin the platform:

version: '3.9'
   services:
      foo:
         platform: linux/amd64
         image: bref/php:82
         

1. If you want to let docker dynamically use the platform of developer laptop (if arch is available in docker multi arch image):

version: '3.9'
   services:
      foo:
         platform: linux/amd64
         image: bref/php:82
         

[georgeboot]

Agreed.

@shouze your 2nd example is identical to first one. You probably meant to omit the platform in 2nd example.

The images probably should be very similar, so it shouldn't matter what platform you're on locally.

If you do care however, set the --platform linux/amd64 flag on the cli or the corresponding flag in the docker compose file.

[mnapoli]

OK, that's interesting, thanks all for chiming in.

So if I sum it up, it would make sense to drop all the bref/arm-xxx images and instead publish cross-platform images?

We don't foresee any downside (except having to set --platform linux/amd64 explicitly for those who want to force a platform locally)?

[jasonmccallister]

@mnapoli that makes sense, but one thing that might be strange is that ARM support for multiple PHP versions is limited (unless that changed) so someone trying to use ARM and deploying bref/php:82 might be surprised when there is no tag for the ARM platform?

[georgeboot]

So if I sum it up, it would make sense to drop all the bref/arm-xxx images and instead publish cross-platform images?

That is correct!

[mnapoli]

👍

and @jasonmccallister now all PHP versions are supported in both x86 and ARM, with no practical differences between the two.

I'll take note and will look into this: brefphp/aws-lambda-layers#62

[mnapoli]

FYI "multi-platform images" won't be implemented in the 2.0 release today (see brefphp/aws-lambda-layers#62 (comment) for more details, it's a bit harder than I thought), but I will work on this afterward.

The AWS Lambda layers will be perfectly fine, and running images locally will work, it will just be a bit slower until we have the multi-platform optimization.