[Proposal] Runtime changes #1078

deleugpn · 2021-10-27T18:32:46Z

This pull request is an ongoing work with an attempt to drastically change the Runtime system.

Why?

It's an attempt to make things easier to change. Although it was triggered by AWS releasing Graviton support on Lambda, this pull request isn't entirely about supporting Graviton, but in fact making it easier and more accessible to build, test and reproduce the layers provided by Bref.

What are the changes?

The expected result is ann easier to reproduce system by leveraging primarily Docker and Docker Compose.
Instead of compiling PHP, we delegate that work to reliable sources like Remi Collect (recommendation)
Simplify the process to become more accessible for us PHP Developers
~~Steal~~ Take inspiration from the folks at Vercel (Source)

What does it mean for new PHP releases?

Although compiling PHP means we're in control of compiling point releases as soon as they come, remi-collect has a great track record of releasing new version on the same day (or even the day before) a new version is announced on the PHP internal list. The downside is that Remi doesn't provide aarch64 support, which means we would need to rely on amazon-linux-extras for Graviton support. Currently Amazon only provides PHP 8.0.8 (about 4 months behind). My take is that it's better to provide delayed aarch64 than not provide at all or provide it using emulators or in a hard-to-maintain process. Given that Apple is pushing arm64 with the new M1 laptops and AWS is pushing it via Graviton support, it's a matter of time until there's better release cycle around arm64.

How does it work?

The project structure is divided into common, phpxx, tests and the root folder.

The common folder contains files for the fpm layer, function layer and the scripts for publishing the layers. The FPM and Function layers are built by using composer install and generating a vendor folder that will be embedded and available inside the Lambda Layer.

The php74, php80 and php81 folders are suppose to be incredibly identical and easy to test and replicate. Each PHP Folder has it's own Makefile so that contributors can make function and publish their own layer in their own account. This allows for easy testing for layers modifications.

The tests folder contains assertions that will be executed inside the container of each layer so that we can "prove" that the system works.

Finally, the root folder contains a Makefile and a docker-compose file that allows for parallelization of building all layers / php versions and publishing them on all regions.

Special thanks to Aran Reeks for teaching me about ldd to show linked libraries for the php binary and its extensions.

What is the `/opt/bref-internal-src` folder?

This is an attempt to make PHP on Lambda easier to onboard and test. It allows to replicate the experience of other Runtime. You go to the AWS Console, create a new Function, attach Bref's layer to it, write a hello.php file and run it. No need for composer or serverless. Just plain old PHP. It greatly improves the User Experience when testing things out.

How can I test this myself?

The biggest motivation for this pull request is to make it easier for anyone/everyone build their own layer so that you can test things out yourself. Making it more reproducible / accessible will help contributors changing the layer to test things. Here are the steps:

git clone git@github.com:brefphp/bref.git
cd bref
git checkout runtime
cd runtime
cd php81

# Modify the Makefile with your AWS credentials

# Run the project tests
make test 

# Publish your own `function` layer
make function

# Publish your own `fpm` layer
make fpm

These steps will publish a new layer into your own AWS Account so that you can test them.

Comparison

bref/php-81                             latest      127c04144153   333MB
bref/x86-php81-function                 latest      c46d3edc8846   361MB

bref/php-80                             latest      1300e903c67e   333MB
bref/x86-php80-function                 latest      3562db33bbad   359MB

bref/php-74                             latest      a508288169c9   331MB
bref/x86-php74-function                 latest      83031f11c8d2   357MB

Checklist

CI / CD

Public AWS CodeBuild
make everything publishing all layers in parallel
Brefphp Bot PR
layers.x86.json
Update serverless-bref plugin to be architecture-aware
Run automation tests when pull request is opened.
Process to publish Docker Images on Docker Hub
Tag Docker Images on Release (GitHub Actions)

Extensions

Important Notes

Currently the process is blocked by docker/compose#8804 for graviton support.

shouze · 2021-10-28T06:37:38Z

runtime/Makefile

-	php layers/tests.php
+function:
+	# Build the base image individually so that the extensions can be built using it
+	docker-compose -f docker-compose.compile.yml build php-base


Would be nice to use and try docker compose (docker compose 2.0 next gen coded in go vs python for the first one)

Do you have a good hello world that I can check?

And maybe the new version wouldn't suffer the docker compose issue you had

mnapoli · 2021-10-28T08:02:44Z

🎉 thank you @deleugpn for starting this huge work. Simplifying the build process will let us be more confident, and spend less time maintaining all that.

One thing that might be worth adding to the TODO list (and early in the list preferably) is testing performances. And by that, I mean (mostly?) checking out the size of the generated layer: is it bigger/smaller than the same layer on master?

Related to that: is there any positive/negative impact (or an opportunity) on Bref's composer package size? (since it's included in all applications, it is part of the cold start time as well) I'm asking this because this PR looks into copying files into the layer directly: is there any file we can exclude from publishing to Composer to lighten the package? Maybe not, I'm just curious ^^

deleugpn · 2021-10-28T09:02:03Z

One thing that might be worth adding to the TODO list (and early in the list preferably) is testing performances. And by that, I mean (mostly?) checking out the size of the generated layer: is it bigger/smaller than the same layer on master?

I just realized that doing that too early is a bit misleading because we don't have all extensions in the image yet.

aran112000 · 2021-10-29T18:56:36Z

Currently Amazon only provides PHP 8.0.8 (about 4 months behind)

I believe Amazon backport any critical security patches to packages from their repos, will check our Yum logs shortly to confirm if that appears to be the case as can't find anything to confirm online.

Update: This doesn't appear to be the case after all and this would be prohibitive for my personal usage of Graviton/ARM sadly

shouze · 2021-11-02T12:45:06Z

runtime/configuration/bref-ext.ini

+extension=/opt/php-modules/dom.so
+extension=/opt/php-modules/exif.so
+extension=/opt/php-modules/fileinfo.so
+extension=/opt/php-modules/ftp.so


💡 I doubt that everybody wants to load ftp extension. Would be nice to give the choice between a "batteries included" with all extensions loaded, and a "minimal" profile with for example only opcache for example, so everybody's in charge then to load their required extensions then.

@mnapoli we already had this discussion in the past when I wanted to make bref as lightweight as possible and I'm still agree with you that the entry level user of Bref should have all batteries included, yet it should be possible to push forward, as loading fewer extentions worth considering it also reduce the attack surface of the λ.

Let's take only dom extension. For the record it's disabled by default on Ubuntu 20, especially because it relies on libxml... which is unfortunately known to have a very impressive CVEs record track 💥 .

In any case, removing extensions right now would be a breaking change, so I don't think it's worth addressing in this PR.

For me it's ok to have every of these core extensions available in the layer FS, but not loaded by default with the "minimal" profile. Because as the layers limit is set to 5, we cannot put every ext into a dedicated layer otherwise every power user will reach such limit quickly.

deleugpn · 2021-11-13T19:17:56Z

I just did a rebase and pushed my latest modifications to the project infrastructure. The project now has 1 Makefile for each PHP Version + 1 Makefile at the root of the Runtime for parallel build and publishing.

There's more code duplication now, but also easier to replicate and evolve together with new PHP Versions. A new version can easily be obtained by copy/pasting the php folder and doing a search/replace with the PHP Version. Older versions are easier to delete as it only requires removing their entire folder + their declaration on the root runtime.

Also, I dialed back in dividing things into multiple Dockerfile as they ended up being more confusing than easier to navigate. The current file structure is almost a top-down build process. It only "fails" to be top-down once it reaches the FPM Layer which then requires jumping a bit back so that the FPM Layer won't have unnecessary files that were loaded into the Function layer.

The "How does it work" on the first post has been updated and the replication of layers in your own AWS account is sketched.

Things still left to do:

Figure out how to feed the layers.json file
Console Layer
Write up a buildspec.yml that will be the AWS CodeBuild that will be running the Makefile
Docker Hub login & docker push of the Docker Images
testing, testing, testing

t-richard · 2021-11-14T00:17:11Z

It seems like intl, apcu and pdo_pgsql are missing in extensions while they are currently present in the bref layers and listed here https://bref.sh/docs/environment/php.html#extensions-installed-but-disabled-by-default

deleugpn · 2021-11-14T17:31:51Z

thanks @t-richard I had forgotten about those. They're ready now.

deleugpn · 2021-11-14T19:23:53Z

Sadly, intl adds 28MB into the size of the layer :(
It took me an hour to pinpoint where the growth had come from, but we're better because of it as I managed to optimize the layers even further. I'll be writing reports of layers sizes soon, but unfortunately for now (for BC compatibility) the layers would end up being slightly bigger because of intl.

shouze

For schema, would be nice to use drawio for example. Thus saving under a .drawio.png extension and you can then edit back into drawio later on as it saves all metadata used to produce the png inside png headers

deleugpn · 2021-11-21T15:52:15Z

@shouze that's a good tip and I didn't know that! However, I'm THE WORST when it comes to visual representations and I really like whimsical because they make it easy for me to make "pretty-ish" charts. Maybe that's something that the community might feel like contributing if we manage to land these changes

shouze · 2021-11-29T21:56:04Z

@deleugpn I've just seen this post, maybe interesting you 😉

deleugpn · 2022-01-31T23:50:55Z

The first set of ARM Layers for prototyping are finally here
#1155

mnapoli

🎉 🎉

I did a first pass of review, awesome job @deleugpn! Most of my comments are about getting the "why" of everything written down (to ease maintenance down the road).

General feedback: could we get a comment at the top of most scripts/config files that explain what they are for?

Another general comment: the wording "php-module" confuses me. I'm more used to see these called "extensions", is the name "module" the name used by remi-collect?

Next steps for me will be reviewing the "embedding" of Bref's source code in the images, I haven't managed to wrap my head around this but will get there!

mnapoli · 2022-03-05T13:01:13Z

index.js

@@ -32,20 +44,27 @@ class ServerlessPlugin {
                    // and guarantees to return a fully resolved form (even if property is configured with variables)
                    const region = options.region || await resolveConfigurationProperty(['provider', 'region']);

+                    // Allows compatible Bref Deployment with custom layers built by the user.
+                    const account = process.env.AWS_ACCOUNT ?? '209497400698'


We should probably use a unique variable like BREF_LAYERS_ACCOUNT?

mnapoli · 2022-03-05T13:02:10Z

index.js

+
+        // TODO:
+        // const arm64 = this.path.resolve(__dirname, 'layers.arm64.json');
+        // const layers_arm64 = JSON.parse(this.fs.readFileSync(filename));


I see the file exists and contains layers, should we uncomment this?

How stable is this?

mnapoli · 2022-03-05T13:31:59Z

runtime/common/publish/Makefile

@@ -0,0 +1,48 @@
+


If I understood correctly this file is autogenerated?

If so it might be worth adding a big-ass comment at the top that mentions this and encourages contributors to not modify it? (and point instead to the docs/script they should look at instead)

It used to be, but through several iteractions it ended up much simpler and not really worth generating it in case of region changes.

runtime/common/publish/generate-makefile.php

mnapoli · 2022-03-05T13:38:18Z

runtime/php74/config/bref-opcache.ini

-error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
-
-memory_limit=3008M
+zend_extension=/opt/php-modules/opcache.so


This line used to be zend_extension=opcache.so which prompts me to wonder: any reason to put extensions in /opt/php-modules and not the previous extension directory that is automatically picked up by PHP?

PHP has an order of folder scan to load extensions. I believe we can still make it work with just the extension name here, but during iteractions of it I found it much easier to debug if I know exactly which folder was expected to have the extension and why it crashed / was not loaded.
By placing just the extension name we end up relying on PHP looking on multiple folders in the PATH before giving up.
Let me know if I should remove the full path and leave just the relative path.

Sorry, I just read your comment again and I noticed I didn't fully address it.

So, when we were compiling PHP, we could choose some paths that would be available for scanning PHP extensions. This is no longer true for installing from a distribution. Remi Collet and Amazon Linux have different configuration betweem themselves (by default), so that would mean we would need different config files for ARM and x86. Furthermore, Amazon Linux places extension files under /etc/... and we can't overwrite that from AWS Lambda.

I did replace this with /opt/php-extensions/.

mnapoli · 2022-03-05T13:55:44Z

runtime/ping/statsd.config.json

-    "cloudwatch": {
-      "region": "EU_WEST_1"
-    }
-  }


I think we want to keep that file around.

mnapoli · 2022-03-05T14:16:09Z

layers.x86.json

@@ -0,0 +1,116 @@
+{


For simplicity and avoiding unexpected breaks (e.g. https://runtimes.bref.sh) could we keep this file as layers.json? (defaults will be x86 anyway)

mnapoli · 2022-03-05T14:20:41Z

runtime/README.md

+### The php{xx} folder
+
+This is the heart of Bref Runtime Layers. We configure php using the `config` folder to store `php.ini` files that
+will be copied into `/opt/php-ini`. Note that we configure `/opt/bootstrap` to execute PHP with `PHP_INI_SCAN_DIR=/opt/php-ini:/var/task/php/conf.d/`.


The directory seems to have changed compared to master right?

I think we need to keep loading ini from the same directories (IIRC it was /opt/bref/etc/php/conf.d/), I don't think we gain anything by changing and that avoids BC break (e.g. for https://github.com/brefphp/extra-php-extensions or any custom layer already existing)? (was there a reason to change?)

mnapoli · 2022-03-05T14:22:22Z

runtime/plugins/console/bootstrap.sh

+# Fail on error
+set -e
+
+export PHP_INI_SCAN_DIR="/opt/php-ini/:/var/task/php/conf.d/"


On master we didn't need a bootstrap.sh file (the bootstrap file was a PHP file directly) -> do we need it now to set the PHP_INI_SCAN_DIR variable?

I'm guessing we are now forced to set PHP_INI_SCAN_DIR manually because we don't compile PHP anymore, is that the reason? Is there any other way we can set PHP_INI_SCAN_DIR once? (I don't think there is, but just making sure)

I replied to this comment here: https://github.com/brefphp/bref/tree/runtime/runtime#bootstrapsh-file and I also added some comments to the file itself, let me know what you think

mnapoli · 2022-03-05T14:23:54Z

runtime/plugins/console/Dockerfile

@@ -0,0 +1,30 @@
+FROM alpine:3.14


I was confused by the plugins word -> the console layer is a normal layer (unless it changed here?), I don't think we should have a concept of plugins, that's one more concept to keep track off? We might as well have this layer in runtime/ directly?

The problem with the console layer is that it's not 1 layer per PHP version and it's also not one layer per CPU architecture. But in any case this was an old idea that ended up adding more complexity than necessary. I deleted the entire folder.

Co-authored-by: Marco Deleu <deleugyn@gmail.com>

mykiwi · 2022-07-22T11:14:10Z

👋 hi,

I played a lot with docker buildx bake lately, and I would like to know if you were interested of having a contribution to refacto:

the runtime/Makefile logic
the github action to have one output per version (instead of only 1 big output)
add the ability to build on ARM architecture family?

I saw this PR after preparing this as an issue, but instead I post my message directly here
This PR seems to be blocked because of docker compose, maybe this would not be the case with docker buildx bake?

I'm open to help / do an example with the master base if you want 🙂

mnapoli · 2022-08-20T07:57:47Z

@mykiwi what does buildx bake bring? I'm not familiar with it, trying to figure out what's next for this refactoring.

mykiwi · 2022-08-20T09:13:32Z

This small documentation explain what is aim https://github.com/docker/buildx/blob/master/docs/guides/bake/index.md
But what I like about it is that we can nested builds without having to do multiple docker build xxx manually, and also have the HCL file format which can give use dynamic variables/functions

mnapoli · 2022-10-18T15:33:45Z

Hey everyone, I have picked up this PR and I am working in a different repository (with all these changes).

More details here: #1299

mnapoli · 2022-12-09T10:16:12Z

Closing, all these commits have not been lost, they are in https://github.com/brefphp/aws-lambda-layers 🎉

We're getting closer to Bref 2.0, thank you @deleugpn for the fantastic work!

More details in #1299

shouze reviewed Oct 28, 2021

View reviewed changes

mnapoli added the enhancement label Oct 28, 2021

deleugpn force-pushed the runtime branch 2 times, most recently from b34f27a to 57ddd0b Compare October 30, 2021 08:56

shouze reviewed Nov 2, 2021

View reviewed changes

deleugpn force-pushed the runtime branch from d417cce to 022f174 Compare November 13, 2021 19:02

deleugpn force-pushed the runtime branch from bf85ede to 34bbe03 Compare November 20, 2021 21:15

shouze reviewed Nov 21, 2021

View reviewed changes

deleugpn mentioned this pull request Nov 29, 2021

ARM64/Graviton support #1043

Closed

deleugpn force-pushed the runtime branch from ad44484 to e67ab67 Compare January 1, 2022 17:31

deleugpn mentioned this pull request Feb 12, 2022

SCRAM authentication requires libpq version 10 or above in #965

Closed

mnapoli reviewed Mar 5, 2022

View reviewed changes

deleugpn added 8 commits March 5, 2022 15:46

Clean up the runtime folder

4788ba1

Tests for the Runtime

f6de57d

PHP 8.0 Runtime Sample

1b6ac26

Runtime Configuration

2eb5a8a

Docker Compose orchestration

176424c

Makefile to build layers

e22d21e

Extension builder

0e14460

Zip & Publish

b1dbd90

deleugpn and others added 14 commits March 5, 2022 15:46

fix path

c1db026

temporarily disable some regions to test STS Token

309de1c

reduce parallelism

8d592de

Publish only to some regions

3bc66ca

Remove parallel execution

b06c907

Fix docker tags

7aba28f

Remove silent from Makefile

eadd155

current folder

ab72fbc

god forgive me for I have 1 month of hatred accumulated

73be427

Thank you so much, @mattrenner

da4b8c8

layers.json update

333579b

Co-authored-by: Marco Deleu <deleugyn@gmail.com>

layers.json update

bbd841a

Co-authored-by: Marco Deleu <deleugyn@gmail.com>

layers.json update

1316f50

Co-authored-by: Marco Deleu <deleugyn@gmail.com>

layers.json update

dacf7f0

Co-authored-by: Marco Deleu <deleugyn@gmail.com>

deleugpn force-pushed the runtime branch from 6a578df to dacf7f0 Compare March 5, 2022 14:47

deleugpn added 7 commits March 5, 2022 16:02

undo bad rebase

4c1422f

Add comment about PHP_INI_SCAN_DIR & PHP Bootstrapping argument

da42471

Add bootstrap.sh to the Decision Log

356ff17

Remove file no longer needed

83b054f

Add some more comments

30b4e2a

Add comment about isolation

7d2cf18

Rename modules with extensions

000ca16

mnapoli mentioned this pull request Oct 18, 2022

Bref 2.0: rewritten runtimes #1299

Closed

mnapoli self-assigned this Nov 5, 2022

mnapoli closed this Dec 9, 2022

[Proposal] Runtime changes #1078

[Proposal] Runtime changes #1078

Conversation

deleugpn commented Oct 27, 2021 • edited Loading

Why?

What are the changes?

What does it mean for new PHP releases?

How does it work?

What is the /opt/bref-internal-src folder?

How can I test this myself?

Comparison

Checklist

CI / CD

Extensions

Important Notes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mnapoli commented Oct 28, 2021

deleugpn commented Oct 28, 2021

aran112000 commented Oct 29, 2021 • edited Loading

shouze Nov 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deleugpn commented Nov 13, 2021 • edited Loading

t-richard commented Nov 14, 2021

deleugpn commented Nov 14, 2021

deleugpn commented Nov 14, 2021

shouze left a comment

Choose a reason for hiding this comment

deleugpn commented Nov 21, 2021

shouze commented Nov 29, 2021

deleugpn commented Jan 31, 2022

mnapoli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deleugpn Mar 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mykiwi commented Jul 22, 2022

mnapoli commented Aug 20, 2022

mykiwi commented Aug 20, 2022

mnapoli commented Oct 18, 2022

mnapoli commented Dec 9, 2022

deleugpn commented Oct 27, 2021 •

edited

Loading

What is the `/opt/bref-internal-src` folder?

aran112000 commented Oct 29, 2021 •

edited

Loading

shouze Nov 2, 2021 •

edited

Loading

deleugpn commented Nov 13, 2021 •

edited

Loading

deleugpn Mar 5, 2022 •

edited

Loading