Feature/magento2 artifact deployment

[wilfriedwolf]

• New feature?

• BC breaks?

• Tests added?

• Docs added?

  Please, regenerate docs by running next command:
  $ php bin/docgen
  

[wilfriedwolf]

Hi there,

After a discussion in the magento_de slack channel I was wondering, if it makes sense to directly include

tasks:
  consumers:kill:
    - run: 'pgrep -f "[q]ueue:consumers:start" | tee /dev/stderr | awk ''{print $1}'' | xargs -r kill'
[...]

after:
  cache:clear:opcache: consumers:kill

in the recipe? Question probably for @peterjaap

[erfanimani]

Good work. I've found the jalogut one is now incompatible with PHP 8.1, so I might try this PR out instead of creating a PR over there.

[antonmedv]

Let’s wait review from the @peterjaap

[wilfriedwolf]

I was discussing this MR lately in the magento_de slack group, and there will be some improvements coming from that side.
Basically to allow more detailed fine tuning and use the contributed cachetool reset functionality. Also some discussions going on about killing consumers and maybe indexers.
So yes, lets wait fore @peterjaap and meanwhile improve the MR.

[osrecio]

Exists an option to reset opcache with Deployer:
https://deployer.org/docs/7.x/contrib/cachetool

[wilfriedwolf]

Yeah, @sprankhub told me that already, I should put that out.

[schmengler]

which we needed to improve though: #3363

[peterjaap]

Killing consumers shouldn't be necessary. Magento should start consumers when they're needed and have them exit when there's nothing in the queue.

I guess the exception is when you would set https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/files/config-reference-envphp.html#consumers_wait_for_messages to 0 but that is an exception and in that case you could add the task to kill them yourself.

[schmengler]

Why do you create an env.php file for the build? There is no database etc, everything we need should be in config.php. At least this should be optional

[erfanimani]

Can second this

[wilfriedwolf]

Ok, the idea behind this is/was the way static assets are generated.

Since on production and dev they differ, I thougth it would be a good Idea, to have them in an env.php which broke my deployment. But probably you are right, If env.php settings override conf.php settings, then we should just omit this part, do they.

[erfanimani]

env.php overrides config.php indeed, and should not be committed. config.php (if configured correctly) is sufficient to run static content deploy and is committed to the repo.

[wilfriedwolf]

So, OK, I'll put this out soon then...

[wilfriedwolf]

Killing consumers shouldn't be necessary. Magento should start consumers when they're needed and have them exit when there's nothing in the queue.

I guess the exception is when you would set https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/files/config-reference-envphp.html#consumers_wait_for_messages to 0 but that is an exception and in that case you could add the task to kill them yourself.

@peterjaap Well, I don't mind putting this out of the MR, anyhow I needed this during a deployment phase, where I was heavily working on consumers. They had like 50 tasks to complete and were "laying" there waiting for new messages, which they would complete with the old code rather then the new changed one, which caused big trouble.

[erfanimani]

@peterjaap @wilfriedwolf the default setting for consumers is to wait until 1000 messages before restarting when cron runs. However, as far as I know, there is a poison pill for consumers to kill themselves when the static content deploy hash is changed. That said, I agree that I'm not sure whether killing consumer processes should be in the scope of this module. (Unless if it somehow interferes with the actual deployment process itself.)

Also see https://developer.adobe.com/commerce/php/development/components/message-queues/object-states/

edit: okay so it turns out the current poison pill implementation isn't great as it requires the var dir to be shared across al releases (which still probably is fine for people that use this module):

magento/magento2#23540 (comment)

[wilfriedwolf]

Regarding the consumers problem and the poison pill: Anyhow it was a question whether to put the code in the MR or not, actually it is not, so I just leave it out and keep it (and the fruitful discussion here) in mind.

[wilfriedwolf]

So to conclude:

• I'll remove the env.php part.
• I'll remove the opcache part.
• I won't add the consumers part.

Anything I oversaw?

[erfanimani]

Actually, after reading magento/magento2#23540 (comment) I think I've changed my mind, maybe leave the consumers part in... some good points are discussed there, and it seems to also be included in the capistrano extension. Does anyone else have any thoughts?

[schmengler]

would you add default values for these in the configuration section and document them there? This is a commonly needed feature and should not stay undocumented

[wilfriedwolf]

Not quite sure if I get it. 'shared_files' and 'shared_dirs' are defined in the standard magento2 recipe. So I think I should not comment these. and the 'additional_shared_files' and 'additional_shared_dirs' are just provided to add some nonstandard files or dirs coming from your shop.
So anyhow I can comment this, but how would I do it, since the documentation is created 🤔

[schmengler]

The documentation is created from comments before set() calls. I'll do it in our fork and show you

Other changes I already implemented: https://github.com/integer-net/deployer/pulls?q=is%3Apr+

[schmengler]

See here: https://github.com/integer-net/deployer/pull/11/files

[wilfriedwolf]

BTW: I had to move the add('...-part in a task (depoy:additional-shared), just like in your PR it did not add the yml settings, I guess because of timing.

desc('Adds additional files and dirs to the list of shared files and dirs');
task('deploy:additional-shared', function () {
    add('shared_files', get('additional_shared_files'));
    add('shared_dirs', get('additional_shared_dirs'));
});

[schmengler]

We have a problem with all those invoke(), apparently they only execute this one task and not anything added via before() or after(), which makes the recipe hard to extend for custom needs.

Maybe task()->select() can be used to only run it on non localhost? If not, is this restriction so important?

[wilfriedwolf]

Me too (just noticed, because I had to add magepack:build :)
Could the problem possibly be solved by defining a task

desc('Forces the tasks to be executed locally and sets the pathes');
task('build:force-local', function () {
    if(currentHost()->get('local')) {
        set('deploy_path', '.');
        set('release_path', '.');
        set('current_path', '.');
    } else {
        throw new GracefulShutdownException("Artifact can only be built locally, you provided a non local host");
    }
});

desc('Builds an artifact.');
task('artifact:build', [
    'build:force-local',
    'build:remove-generated',
    'deploy:vendors',
    'magento:compile',
    'magento:deploy:assets',
    'artifact:package'
]);

[wilfriedwolf]

Would also apply to the deployment which must not be local!

[schmengler]

yes, here's my working solution: integer-net#10 (didn't bother to force non-local, but the same would work there too)

[schmengler]

I've asked Anton for clarification here about local tasks: #2679 (comment)

but if there's no cleaner way I would turn this into an array of tasks without invoke() too (see below)

[wilfriedwolf]

So, I removed and added as discussed.
Since no consent was found concerning the consumers I did not add them.
Thanks to all discussing with me.

[wilfriedwolf]

@antonmedv @peterjaap Anything else here to do? I'm using this version (my own composer package) for a month now and I do not see any open questions problems (except maybe a final statement from @peterjaap)?

[antonmedv]

I'm ok to merge. @peterjaap ?

[schmengler]

Thank you @antonmedv 🧡

Brace yourself for more PRs

[guvra]

@wilfriedwolf Do you have any example of deploy.php file that implements the artifact tasks?

I'm trying to use it but I don't know how to build the artifact (the task artifact:build wants a local host, but I did specify one with dep artifact:build localhost and it stills trigger the error message).

edit: "solved" it with the following code:

localhost()
    ->set('local', true);

[wilfriedwolf]

@guvra well, actually I use deploy.yaml to define the hosts, and there it would need:

hosts:
  localhost:
    local: true
  devbox:
    local: true

which basically is the same as your "workaround". The sense behind that restriction is/was to avoid accidently building artifacts on remote machine. Don't know if this is necessary..

[wilfriedwolf]

@antonmedv Thanks a lot for merging!

[antonmedv]

Let’s add some documentation in recipe as comments.

[guvra]

That would be neat 👍

I managed to make it work on a gitlab ci automated deployment, but I got a few issues with the magento:deploy:assets task.
This task was failing because Magento uses the default mode by default, but the command setup:static-content:deploy can only be run in production mode if the -f option was not specified).

I solved it by creating a dummy env.php file in the gitlab ci script:

echo "<?php return ['MAGE_MODE' => 'production'];" > app/etc/env.php
deployer artifact:build localhost
deployer artifact:deploy $ENV_NAME

[antonmedv]

Can we add this pseudo env as a step in recipe?

[wilfriedwolf]

@guvra @antonmedv Actually this is not necessary. I had that approach (providing a special app/etc/env.php for the artifact:build step) in one of the first versions of this pull requests.
After discussing with some people (@schmengler and @sprankhub) on slack, I changed it in favor of defining everything in app/etc/config.php (and overriding the settings in the local env.php if needed).

For the assets you would need MAGE_MODE and the system settings defining how to deploy the assets...

    [....]
    'MAGE_MODE' => 'production',
    'system' => [
        'default' => [
            'dev' => [
                'js' => [
                    'enable_js_bundling' => '0',
                    'minify_files' => '1',
                    'merge_files' => '0'
                ],
                'css' => [
                    'minify_files' => '0'
                ],
                'static' => [
                    'sign' => '1'
                ],
                'template' => [
                    'minify_html' => '1'
                ]
            ],
           [....]

[guvra]

@wilfriedwolf Thanks for the heads up, it works perfectly 👍

FYI, I added a section in my deploy.php file that allows to build the artifact from a development environment (e.g. local docker compose container):

set('repository', 'git@xxxxxxxxx');

// Custom code for deployment from dev environment
set('build_dir', null);
after('build:prepare', function () {
    $buildDir = (string) get('build_dir');
    if ($buildDir=== '') {
        return;
    }

    set('deploy_path', $buildDir);
    set('current_path', $buildDir);
    set('current_path', $buildDir);
    run('rm -rf {{release_or_current_path}}');
    run('git clone {{repository}} {{release_or_current_path}}');
    run('git checkout --force {{target}}');

Then I just have to specify build_dir in the command line to trigger this custom code.
It clones the repo in a temporary directory before creating the artifact.

e.g. deployer artifact:build localhost --option build_dir=/tmp/magento-build --branch production

[akosglue]

how do we use artifact deployment exactly? does someone have a sample deploy.php file maybe?

[antonmedv]

@wilfriedwolf will be cool to add some docs right in the recipe comments.

[wilfriedwolf]

Hej @antonmedv

I am definitvely willing to do so, but I have no clue how. Looking on the generated magento2.md I find a longer explanation which is not in the recipe but which might be good for the artifact deployment.
Also I find some but not all desc() texts I have written above the tasks and only a subset of tasks was added at all.
So if you could point me out how to do (even a manual or so would do) I am off..

Best regards,

Willi

[Schrank]

@peterjaap @schmengler Do you have a hint?

[antonmedv]

You can just add a comment at the top of the recipe.

[akosglue]

What I could figure out until now is that I basically need to run these 2 command's:

(for the moment ignoring #3454)

./vendor/bin/dep artifact:build -vvv --option build_from_repo=true
./vendor/bin/dep artifact:deploy -vvv

However it fails for me with this:
Could not open input file: .../releases/26/bin/magento (just when trying the setup:db:status command)

It is a little strange because the artifact file is made like this:
/usr/bin/tar --exclude-from=artifacts/excludes -czf artifacts/artifact.tar.gz artifacts/repo

And when I run the untar:

/usr/bin/tar -xzpf ... artifact.tar.gz -C .../releases/26

then the repo folder is untarred into artifacts/repo inside .../releases/26, instead of on top of .../releases/26, so that's why the magento bin is not found.

Edit: all this in MacOs Ventura

[guvra]

@akosglue You must build the artifact file from a local environment.

First, you need to add this line in deploy.php:

localhost()
    ->set('local', true);

If you're using a yaml file instead of deploy.php, IIRC the syntax is the following:

hosts:
    localhost:
        local: true

---

To build the artifact, there are two modes:

• Building from the current working directory (useful if you're building from a CI, e.g. github actions)

  vendor/bin/dep artifact:build localhost
  

• Building from a temporary directory (deployer will clone the repo before building the artifact)

  vendor/bin/dep artifact:build localhost --option build_from_repo=1 --tag <mytag>
  

Notes:

• DO NOT use the first mode if you're building from a development environment.
• To use the 2nd mode, you also need to set the repository var in deploy.php.

---

Then, to deploy:

vendor/bin/dep artifact:deploy <host>

---

Warning: config.php file must contain the list of the stores/websites, as well as the CSS/JSS compilation settings. This is because Magento won't have access to a database during the artifact generation, so the relevant data must be stored in config.php instead.

Example of config.php file:

return [
    'modules' => [
        // ...
    ],
    'MAGE_MODE' => 'production',
    'scopes' => [
        'websites' => [
            'admin' => [
                'website_id' => '0',
                'code' => 'admin',
                'name' => 'Admin',
                'sort_order' => '0',
                'default_group_id' => '0',
                'is_default' => '0'
            ],
            'base' => [
                'website_id' => 1,
                'code' => 'base',
                'name' => 'Main Website',
                'sort_order' => '0',
                'default_group_id' => '1',
                'is_default' => '1'
            ]
        ],
        'groups' => [
            [
                'group_id' => '0',
                'website_id' => '0',
                'code' => 'default',
                'name' => 'Default',
                'root_category_id' => '0',
                'default_store_id' => '0'
            ],
            [
                'group_id' => '1',
                'website_id' => '1',
                'code' => 'main_website_store',
                'name' => 'Main Website Store',
                'root_category_id' => '2',
                'default_store_id' => '1'
            ]
        ],
        'stores' => [
            'admin' => [
                'store_id' => '0',
                'code' => 'admin',
                'website_id' => '0',
                'group_id' => '0',
                'name' => 'Admin',
                'sort_order' => '0',
                'is_active' => '1'
            ],
            'default' => [
                'store_id' => '1',
                'code' => 'default',
                'website_id' => '1',
                'group_id' => '1',
                'name' => 'Default Store View',
                'sort_order' => '0',
                'is_active' => '1'
            ]
        ]
    ],
    'system' => [
        'default' => [
            'dev' => [
                'js' => [
                    'enable_js_bundling' => '0',
                    'minify_files' => '1',
                    'merge_files' => '1'
                ],
                'css' => [
                    'minify_files' => '1',
                    'merge_css_files' => '1'
                ],
                'static' => [
                    'sign' => '1'
                ],
                'template' => [
                    'minify_html' => '1'
                ]
            ]
        ]
    ]
];

[antonmedv]

Let’s add it as documentation in recipe.

[akosglue]

Thanks @guvra

I also needed to add '--strip=2' to the artifact:extract command but I think this is a MacOS specific issue.

[wilfriedwolf]

@antonmedv Submitted a PR with the documentation after digging a little in the bin/docgen command, which i modified a little to allow the artifact deployment to be explained more prominently. Hope this is o.k.

[wilfriedwolf]

@akosglue Do you use gtar or tar ? I thougt it would work with gtar under MacOS?

[akosglue]

I used tar (/usr/bin/tar)
I have just seen your docs update that I should use gtar.

anyways, it work also with tar, I just needed to make this task override.

//this override is only needed when the artifact is made from repo: artifact:build --option build_from_repo=1 //not needed to strip path when the artifact is made from current directory: artifact:build --option build_from_repo=0 /* desc('Extracts artifact in release path.'); task('artifact:extract', function () { run('{{bin/tar}} --strip=2 -xzpf {{release_path}}/{{artifact_file}} -C {{release_path}}'); run('rm -rf {{release_path}}/{{artifact_file}}'); }); */

[wilfriedwolf]

I guess you can use tar on mac, but then you maybe should add some tar_extract_extra_options which resolve to '--strip=2' when tar is used with mac?

[akosglue]

Yes, it seems to be cleaner.

I guess you can use tar on mac, but then you maybe should add some tar_extract_extra_options which resolve to '--strip=2' when tar is used with mac?

[wilfriedwolf]

But I don't know if there is a check for MacOS, so better use gtar on Mac :)