Add guidance for updating bundled packages in Kibana

[joshdover]

As of 8.2, Kibana ships some packages bundled in the Kibana release (see elastic/kibana#112095). As part of the release process, the fleet_packages.json file in the Kibana repo needs to be updated prior to the final BC for a release version. Currently, this is a manual process, but some assistance in the elastic-package CLI would go a long way to help. We could do something simple for now or go one step further to automate this process.

Give guidance on how to manually update

It would be helpful to add guidance to the elastic-package publish command to help package owners be sure to update this file in the Kibana repo, when appropriate. This should show after elastic-package publish succeeds for a package in the following list:
- apm, elastic_agent, fleet_server, synthetics, endpoint

We could show something like this in the stdout:

You have published a package that is bundled by Kibana. There are additional steps required to update this package in Kibana snapshot, BC, and release builds.

If this version should be shipped in the next BC or release, update https://github.com/elastic/kibana/blob/<release branch>/fleet_packages.json on the appropriate release branch.

If this version should be used in development for next snapshot builds, update
https://github.com/elastic/kibana/blob/main/fleet_packages.json

Automate opening a PR

Even better would be to open a PR against the Kibana repo for these changes, including against the correct release branch. One issue is that I think this would require asking the package owner for the intended Stack release version on the CLI or maybe we could use the constraint.kibana.version to infer this.

[joshdover]

cc @kpollich

[mtojek]

Hi Josh,

thanks for raising the issue. Before we analyze this use case, let's first remind few assumptions about Package Storage v2:

1. Package Storage structure is flat. No more promotions between stages will be performed. Once the package is published, it will be exposed to the world.
2. All published package revisions are idempotent and it won't be possible to overwrite content under the same version. The entry point to the storage is a signed, zipped package. Any packages that don't follow this rule, will be simply rejected without any exceptions. This is the root cause of all inconsistencies between EPR deployments.

Back to the issue:

This should show after elastic-package promote -p succeeds for a package in the following list AND the package has been published to snapshot:

elastic-package publish publishes a package to snapshot branch. This command is executed by CI only.
elastic-package promote can promote from snapshot to staging or staging to production, or snapshot to production (not recommended).

Which step do you refer to?

[joshdover]

• Package Storage structure is flat. No more promotions between stages will be performed. Once the package is published, it will be exposed to the world.

Understood, I updated the issue to remove the mention of stages. We currently pull from the snapshot registry. Is my understanding correct that once v2 storage is active, a published package will be in all registry stages or only the production one?

• All published package revisions are idempotent and it won't be possible to overwrite content under the same version. The entry point to the storage is a signed, zipped package. Any packages that don't follow this rule, will be simply rejected without any exceptions. This is the root cause of all inconsistencies between EPR deployments.

I don't see this really impacting this issue / workflow at this point. I do think we need some guidance on how to use prerelease packages for Stack BCs and then promote them to final versions for the final Stack BC(s). As of now, the only package that I am aware of that really requires this now is the APM package since they desire that their package version is always aligned with the Stack version. I'll work with them directly on this.

elastic-package publish publishes a package to snapshot branch. This command is executed by CI only. elastic-package promote can promote from snapshot to staging or staging to production, or snapshot to production (not recommended).

Which step do you refer to?

Too many s-named stages, got staging and snapshot mixed up. publish would be fine in that case, we currently pull bundled packages from snapshot. I've updated the description accordingly.

[mtojek]

Understood, I updated the issue to remove the mention of stages. We currently pull from the snapshot registry. Is my understanding correct that once v2 storage is active, a published package will be in all registry stages or only the production one?

The final result would be a composition of packages from all stages, but there will be an option to skip/include prerelease ones. This is still a plan, but it will be consistent to expose the same Package Registry instance (unified stage) behind all endpoints - snapshot, staging, production).

I don't see this really impacting this issue / workflow at this point. I do think we need some guidance on how to use prerelease packages for Stack BCs and then promote them to final versions for the final Stack BC(s). As of now, the only package that I am aware of that really requires this now is the APM package since they desire that their package version is always aligned with the Stack version. I'll work with them directly on this.

I think this is a good topic for the Tech Leads weekly sync. I'm afraid that with the bundling feature, we tangled dependencies between all components. Maybe we shouldn't publish bundled packages to the registry at all? (probably a separate topic)

Too many s-named stages, got staging and snapshot mixed up. publish would be fine in that case, we currently pull bundled packages from snapshot. I've updated the description accordingly.

As the "publish" operation is mostly dedicated to the machine execution, I guess that we could open another PR to the Kibana repository, similarly as we open one for "snapshot". But I'm afraid that the devil is hidden in details, we can cover only these packages: elastic_agent, fleet_server, synthetics. Publishing of packages apm, endpoint is managed manually by their teams. I don't think they use the elastic-package publish, but I might be wrong.

[akshay-saraswat]

Since we created elastic/kibana#127974, I believe we can close this issue. Is there anything additional that we want to achieve with this GH issue?