[Fleet] Support stack-aligned versioning scheme for bundled APM package

[kpollich]

Replaces #136834

In the process of adapting various Elastic Agent packages to the new Package Storage v2 architecture, we've identified some needs specifically for the APM package to continue allowing the maintainers of the APM integration to work within their current processes:

• Kibana should bundle the latest compatible version of APM, including prerelease versions
• The latest compatible version of APM should be considered valid using the package verification logic logic added in 8.4
• Kibana should rewrite a prerelease version of APM (e.g. 8.4.0-dev1) to be presented as 8.4.0 to preserve APM's versioning needs

To meet these goals, we're proposing the following implementation:

1. Expose a usesStackAlignedExperimentalReleases flag in fleet_packages.json
2. If usesStackAlignedExperimentalReleases is set for a bundled package, Kibana will fetch the latest compatible prerelease version from EPR when bundling, regardless of the version value specified in fleet_package.json
3. After the package is fetched, unpacked, and verified, Kibana will remove the prerelease tag from the package version from the package version

So, if APM opts in to this behavior, we'd have the following process:

• Fetch against EPR with ?experimental=true query string
• Retrieve apm-8.5.0-dev1 version of APM, for example
• Verify apm-8.5.0-dev1 version using the returned archive + signature file from EPR
• Install the apm-8.5.0-dev1 package
• Manually update the version of the installed package to 8.5.0 so it's displayed as such in the UI, etc

Show original description

To accomplish this, we're proposing an implementation as follows

1. Use the existing bundling build step to pull the package versions and signatures listed in fleet_packages.json from the snapshot registry
2. Add a new build step that overwrites some packages from the first step with packages from another directory (configured by CI).
   • We'd use this to overwrite the APM package from snapshot with one built from the https://github.com/elastic/apm-server build artifacts. This step would be a no-op in local builds not built on CI with access to apm-server's build artifacts.
3. Continue to do verification at install time for bundled packages

@axw @Mpdreamz @joshdover - Please let me know if I've captured the needs here accurately, and if the proposed implementation from Josh's email looks sufficient.

@elastic/kibana-operations - What is the feasibility of being able to access artifacts built by CI in the apm-server repo during Kibana's CI builds? If we want to access these artifacts what would be the best way to do it? Could we have Kibana CI depend on a build task run in apm-server, and then be able to access the build artifacts it generates? This will be a process that may need to be made generic to support similar implementations for cases like endpoint.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[spalger]

There are buildkite APIs to access artifacts from specific builds, so if we knew the buildkite build id for the apm-server job that built the artifact then accessing the artifacts would be trivial. Knowing the build id is the hard part. You might want to use a simple gcs artifact/manifest that records the most recent build ID for a specific branch or something, allowing a very simple record of the correct build id to use.

buildkite-agent artifact download "apm-server-build.tar.gz" . --build "$APM_SERVER_BUILD_ID"

Alternatively, if you want to just upload the artifacts to a gcs bucket and then just download the most recent version of the artifact from a well-known object name, like:

gsutil cp gs://elastic-apm-server-ci-builds/${branch}/latest.tar.gz apm-server-build.tar.gz

We can help setup the GCS buckets you'd like to use

[axw]

@kpollich I think that all sounds sensible. We'll need to sign the packages we store in build artifacts, which I think is something that @mtojek is looking into anyway.

@spalger we are using Jenkins, so unfortunately we can't use buildkite build artifacts.

For unified release builds, I would expect that Kibana would bundle the apm package from the the specific APM Server build for that unified release, rather than the latest branch build. Would it make sense to have Kibana build after APM Server in unified release, and fetch the artifacts that way? This is how binary artifacts are bundled in elastic-agent.

[jbudz]

@mgreau can you help triage this? We can re-add support for synchronous artifacts, or implement an async check, just want to make sure we're not overlapping projects.

@kpollich what timeline are we targeting here?

[kpollich]

@kpollich what timeline are we targeting here?

I think sometime in 8.5 would be ideal, but @axw or someone else on the APM side might be able to better answers this. Should this be considered a blocker for package storage v2 shipping?

[axw]

Should this be considered a blocker for package storage v2 shipping?

Yes, I think so. So 8.5 would be the target.

[kpollich]

I spoke with @joshdover a bit offline, and I think we can avoid any involvement with CI or operations here by doing the following.

1. Expose a usesStackAlignedExperimentalReleases flag in fleet_packages.json
2. If usesStackAlignedExperimentalReleases is set for a bundled package, Kibana will fetch the latest compatible prerelease version from EPR when bundling, regardless of the version value specified in fleet_package.json
3. After the package is fetched, unpacked, and verified, Kibana will remove the prerelease tag from the package version from the package version

So, if APM opts in to this behavior, we'd have the following process:

• Fetch against EPR with ?experimental=true query string
• Retrieve apm-8.5.0-dev1 version of APM, for example
• Verify apm-8.5.0-dev1 version using the returned archive + signature file from EPR
• Install the apm-8.5.0-dev1 package
• Manually update the version of the installed package to 8.5.0 so it's displayed as such in the UI, etc

I think this avoids the risk of shipping a prerelease versions in production (assuming the last prerelease == the production release) while affording us the ability to use prerelease versions of APM during the BC period.

@axw - Does this satisfy the APM team's needs here without needing to involve more moving parts involving the operations folks, CI pipelines, etc?

[axw]

@kpollich I think it's reasonable. I'd say it's not perfect, as it's depending on something outside of the unified release process, which could break this assumption

assuming the last prerelease == the production release

... but I think this is better than what we have today, and addresses our immediate needs. I'd say we should go with this, and if we need to we could come back to the more complicated CI pipeline orchestration.

[jsoriano]

Fetch against EPR with ?experimental=true query string

Take into account that experimental flag is deprecated, prerelease=true should be used instead (more context in elastic/package-spec#225).

[kpollich]

Hi folks, I've updated the description + title here based on #137759 (comment), and this should no longer involve the operations team so I've removed their label here. Thanks to the operations folks that responded above to provide some clarity, but we should be able to move forward with a more isolated implementation in Fleet's package bundling process.

[cauemarcondes]

Just to clarify, based on what's been discussed there's no action for the APM UI team, am I right?

[kpollich]

Just to clarify, based on what's been discussed there's no action for the APM UI team, am I right?

This is correct based on the implementation we've discussed in this issue. The Fleet team would be driving the implementation of the prerelease version rewriting, support for an opt-in flag in fleet_packages.json, etc.

---

@axw I've spent some time thinking about bundled packages and speaking with @ruflin about this in more details, and I'd be very interested to hear more about APM's specific pain points with the current processes around bundling packages + how they relate to package storage v2. Upon reviewing the email chain that prompted this issue, I came across this list of requirements you drafted:

1. In Cloud, have the package installed by default and auto-upgraded on stack upgrade.
2. On-prem, have the package installed on demand, and then auto-upgraded on stack upgrade.
3. Align the APM package version with the stack. The package, APM Server code, and Kibana code are tightly coupled.
4. Integrate with the unified release process, and use the same package promotion process. This will enable us to make fixes to the package after any BC if it's a release blocker

Of these, numbers 1 and 2 are already fully supported and working as intended with regards to Fleet and APM. I hope I'm not missing something there, but I think we're aligned there and meeting the APM team's needs via cloud's preconfiguration configs and our support for "auto-update" packages + "auto-upgrade" package policies.

Numbers 3 and 4 are a bit more confusing to me, and I hope you can provide a little more clarity so I can understand the motivations here, and whether our path forward laid out in this issue is the right one.

3. Align the APM package version with the stack. The package, APM Server code, and Kibana code are tightly coupled.
4. Integrate with the unified release process, and use the same package promotion process. This will enable us to make fixes to the package after any BC if it's a release blocker

I'm assuming this means that when we're shipping Kibana 8.4.0, we're also shipping APM server 8.4.0 and the APM integration 8.4.0 - essentially treating all of these individual services as stack components.

My main question here is whether this is a functional or aesthetic need. For example, would it be possible to push an emergency release to the APM integration and publish it to EPR as 8.4.0-patch1 or similar? Does this emergency release process have an analog in APM server, or is APM server always released as part of the unified release cycle?

---

Overall, I'm concerned about baking too much logic into the package bundling process in Kibana. I am wondering if it's worth revisiting #124342 and considering storing bundled packages directly in Kibana's source code.

Storing bundled packages directly in Kibana and updating via Git replaces the manual pre-release checklist of bundled package updates with a more automation-friendly process. e.g. elastic/integrations repo CI can push bundled package updates to Kibana via GitHub PR. Similarly, the elastic/apm-server can push to EPR (via elastic-package upload) and Kibana (via its own CI + GitHub PR).

A process like this removes the "build step" to resolve + download bundled packages every time a Kibana distributable is built, and makes it simpler to manage bundled packages via Git and GitHub instead of custom logic and tooling.

Of course there are many bits of nuance and further thinking to do to support this, so I don't want to forego breaking down any immediate blockers the APM team is dealing with. Is the prerelease version rewriting logic captured here necessary to unblock the APM team in the coming releases where package storage v2 will be live?

I know this is a conversation that has happened a few times now, but I'm curious if there are any new thoughts here. Does this issue run counter to a potentially more elegant solution involving Kibana's source code directly containing bundled packages?

[axw]

Of these, numbers 1 and 2 are already fully supported and working as intended with regards to Fleet and APM. I hope I'm not missing something there, but I think we're aligned there and meeting the APM team's needs via cloud's preconfiguration configs and our support for "auto-update" packages + "auto-upgrade" package policies.

That's correct, I just included them for completeness.

Numbers 3 and 4 are a bit more confusing to me, and I hope you can provide a little more clarity so I can understand the motivations here, and whether our path forward laid out in this issue is the right one.

3. Align the APM package version with the stack. The package, APM Server code, and Kibana code are tightly coupled.
4. Integrate with the unified release process, and use the same package promotion process. This will enable us to make fixes to the package after any BC if it's a release blocker

I'm assuming this means that when we're shipping Kibana 8.4.0, we're also shipping APM server 8.4.0 and the APM integration 8.4.0 - essentially treating all of these individual services as stack components.

That's correct.

My main question here is whether this is a functional or aesthetic need. For example, would it be possible to push an emergency release to the APM integration and publish it to EPR as 8.4.0-patch1 or similar? Does this emergency release process have an analog in APM server, or is APM server always released as part of the unified release cycle?

APM Server is always released as part of the unified release.

It would be possible to publish a patch to the integration package out of band, but we don't want to introduce version skew. There would be more variables, decreasing supportability. It also means the APM UI code cannot assume anything about the version that is installed, increasing its complexity (see #124342 (comment)).

We have not yet needed to push an emergency fix for the integration package. Not to say that it can't or won't happen, but it's the wrong thing to optimise for. Issues with assets installed by the integration package are much more easily worked around compared to bugs in APM Server code.

Overall, I'm concerned about baking too much logic into the package bundling process in Kibana. I am wondering if it's worth revisiting #124342 and considering storing bundled packages directly in Kibana's source code.

Is it that the change to rewrite the version sigificant? Or is the concern about the implications of bundling at build time, and making the build non-reproducible? I assume the latter.

I don't want to move the source of the APM integration package to Kibana in the foresseable future, as working across repos like that will introduce toil for APM Server developers and reduce our velocity.
I'd be happy to consider syncing the source for bundling though.

Storing bundled packages directly in Kibana and updating via Git replaces the manual pre-release checklist of bundled package updates with a more automation-friendly process. e.g. elastic/integrations repo CI can push bundled package updates to Kibana via GitHub PR. Similarly, the elastic/apm-server can push to EPR (via elastic-package upload) and Kibana (via its own CI + GitHub PR).

A process like this removes the "build step" to resolve + download bundled packages every time a Kibana distributable is built, and makes it simpler to manage bundled packages via Git and GitHub instead of custom logic and tooling.

I would be fine with having apm-server's CI push the package to elastic/kibana. Can we avoid having to go through PRs & additional code reviews? Integration package changes are reviewed in APM Server PRs, so I expect that should be sufficient.
This does seem like a better option to me, since it enables reproducible builds.

Of course there are many bits of nuance and further thinking to do to support this, so I don't want to forego breaking down any immediate blockers the APM team is dealing with. Is the prerelease version rewriting logic captured here necessary to unblock the APM team in the coming releases where package storage v2 will be live?

I know this is a conversation that has happened a few times now, but I'm curious if there are any new thoughts here. Does this issue run counter to a potentially more elegant solution involving Kibana's source code directly containing bundled packages?

It's not strictly necessary to do the version rewriting. If we push a timstamped package to package-storage v2 (because immutability) and a non-timestamped version to kibana (because stack version alignment), then I believe we'll be unblocked.
Now it's just a question of what we can get done in time to switch over to storage v2. We need to ensure the bundling is done right before then, as otherwise we'll run into problems given that we cannot overwrite versions in storage v2.

[jbudz]

Storing bundled packages directly in Kibana and updating via Git replaces the manual pre-release checklist of bundled package updates with a more automation-friendly process

I'd rather we poll the unified release API for the latest release during development/builds than store an archive in the Kibana repo. At ~25mb per archive the repo size will end up quite large. We can have APM CI trigger Kibana CI when a new build is published. We will have to make sure the released version is the exact same as the bundled version.

I would be fine with having apm-server's CI push the package to elastic/kibana. Can we avoid having to go through PRs & additional code reviews?

Maybe - I'm not sure how integrated APM server is. Elasticsearch changes break Kibana tests relatively frequently, so we don't allow automatic updates. We do package beats in our Cloud image without verification, but that's really just running a separate process with no integration.

[axw]

I'd rather we poll the unified release API for the latest release during development/builds than store an archive in the Kibana repo.

Another option, which is a kind of combination of all of the above: we could have APM Server's CI send a PR to Kibana which will update it to a specific pre-release/snapshot package version after it is published to package storage. That way we would have reproducible builds. We would still want to rewrite the version to match Kibana's.

Something like this:

1. APM Server runs CI on main + active release branches, builds and tests its integration package
2. Package is published as a prerelease version, e.g. 8.5.0-preview-123456
3. A PR is sent to Kibana updating a config file (maybe just https://github.com/elastic/kibana/blob/main/fleet_packages.json), bumping the version to 8.5.0-preview-123456. The PR would be created with auto-merge, and would merge as long as the tests pass.

When Kibana is built off that commit, it wouldn't pull the latest package, but it would pull that specific package version from storage. It would rewrite the version of the package to match its own version (8.5.0-preview-123456 => 8.5.0).

Maybe - I'm not sure how integrated APM server is. Elasticsearch changes break Kibana tests relatively frequently, so we don't allow automatic updates. We do package beats in our Cloud image without verification, but that's really just running a separate process with no integration.

@jbudz if we could create a PR which doesn't need review, but waits for tests to pass and then auto-merges - would that be OK?

[jbudz]

@jbudz if we could create a PR which doesn't need review, but waits for tests to pass and then auto-merges - would that be OK?

Yep that should be fine - I don't think we have a merge without approval button, but we've automated approvals in some cases e.g. #138866

[kpollich]

I'd rather we poll the unified release API for the latest release during development/builds than store an archive in the Kibana repo. At ~25mb per archive the repo size will end up quite large. We can have APM CI trigger Kibana CI when a new build is published. We will have to make sure the released version is the exact same as the bundled version.

What about storing the non-archived packages and generating .zip archives as part of the build? We might need to introduce a dependency on https://github.com/elastic/elastic-package to perform the package build, but it might save on space here. This would also allow package maintainers to actively develop within the Kibana repo if they chose to do so, rather than syncing via CI from another location.

Another option, which is a kind of combination of all of the above: we could have APM Server's CI send a PR to Kibana which will update it to a specific pre-release/snapshot package version after it is published to package storage. That way we would have reproducible builds. We would still want to rewrite the version to match Kibana's.

This would definitely solve the issue with manually updating bundled package versions, but we'd still be doing the on-the-fly version rewriting specific to APM's (and maybe Endpoint's?) needs. What if, as part of the "bundled package contract", any package whose source is stored in elastic/kibana would have its version forcibly set to align with Kibana's? e.g. all bundled packages would ship with Kibana with a version like 8.4.0 or 8.4.0-SNAPSHOT in the case of a build candidate. Then, it'd be up to the integration maintainers to ensure a same-versioned package exists on the registry to avoid version divergence.

[mtojek]

Hi @axw and @kpollich,

I'm back from PTO and trying to keep up with this thread.

Another option, which is a kind of combination of all of the above: we could have APM Server's CI send a PR to Kibana which will update it to a specific pre-release/snapshot package version after it is published to package storage. That way we would have reproducible builds. We would still want to rewrite the version to match Kibana's.

The Infra layer of the Package Storage v2 contains a set of jobs that can be triggered on different events. We could add a new job responsible for syncing bundled packages (or just the apm package?) with the Kibana repository. It can be started by the indexing job or on daily basis (schedule-daily job).

It isn't anything new for the infra, as we have also a different job to refresh kubernetes templates. The job opens PRs against the agent (or beats?) repository.

Let me know what do you think.

[ruflin]

It can be started by the indexing job or on daily basis (schedule-daily job).

Nit: I prefer we optimise for "continuously" instead of daily, just to get into the habit.

[mtojek]

Nit: I prefer we optimise for "continuously" instead of daily, just to get into the habit.

Yes, if we want to explore the option with the sync job. Let's wait for @kpollich to chime in.

[kpollich]

The Infra layer of the Package Storage v2 contains a set of jobs that can be triggered on different events. We could add a new job responsible for syncing bundled packages (or just the apm package?) with the Kibana repository. It can be started by the indexing job or on daily basis (schedule-daily job).

It would be great to continuously update the APM package stored in Kibana's source via CI like this. The k8's template job is a pretty direct comparison, I think.

To clarify, I've created two diagrams to capture the before/after workflows around bundled packages.

Current bundled package workflow

Proposed bundled package workflow

---

So, the new process would essentially allow the APM team to publish any version of the APM package and a PR would automatically be filed against Kibana updating the package in main. Kibana would then be responsible for building the package as an additional build step, including a forcibly-aligned version number based on the generated Kibana distributable.

Let me know if I've captured your thinking here, @mtojek.

[mtojek]

These are really fancy graphs :) Please share what you've used to prepare them @kpollich.

Let me rephrase it into components we own.

1. Package Storage indexing-job triggers "sync-bundled-packages" job.
2. Sync-bundled-packages:
   2.1. Open PR against Kibana to update package sources.
3. Kibana takes it from there: build, release, etc.

Questions:

1. Are package sources stored in Kibana today? Most likely I missed the source reference.
2. Not sure why there is a both-directions arrow between Kibana and "PR updating package". Can we focus on one direction as MVP?
3. The package version is hardcoded in manifest.yml and in changelog.yml. Should the sync-bundled-packages job replace/alter the package version? It's different depending on the package. For example: apm-8.5.0-preview-1660193585 vs. 8.5.0-dev.0.
4. Which packages should be bundled? APM, Endpoint, Synthetics, anything else?
5. We don't need to sign bundled packages right? These will ultimately be considered to be "safe".

I'm concerned about increasing complexity around the solution, but hopefully, we're closer than further. cc @jlind23

[kpollich]

https://excalidraw.com/ is the tool in question 🙂

Are package sources stored in Kibana today? Most likely I missed the source reference.

Package sources aren't stored in Kibana today, only a reference to a package. Kibana fetches the built package archives from EPR at build time for a configurable list of bundled packages.

Not sure why there is a both-directions arrow between Kibana and "PR updating package". Can we focus on one direction as MVP?

Yes this is just a typo - PR's only go one direction it turns out 😵

The package version is hardcoded in manifest.yml and in changelog.yml. Should the sync-bundled-packages job replace/alter the package version? It's different depending on the package. For example: apm-8.5.0-preview-1660193585 vs. 8.5.0-dev.0.

I think this is one of the big asks from @axw here: to forcibly align APM's version to the stack. See his responses above regarding this version alignment need: #137759 (comment). The plan would be to overwrite the version in manifest.yml when building the package, yes.

Which packages should be bundled? APM, Endpoint, Synthetics, anything else?

The packages we bundle today are captured here https://github.com/elastic/kibana/blob/main/fleet_packages.json

• APM
• Elastic Agent
• Endpoint
• Fleet Server
• Synthetics

We don't need to sign bundled packages right? These will ultimately be considered to be "safe".

Yes I think we'd want these packages to skip the signing/verification process. Since their source code ships directly with Kibana it seems unnecessary to verify them in the same way we verify packages downloaded from EPR. This will probably allow us to skip over some corner cases related to verification.

[mtojek]

I'm not sure if I just circle back the discussion, but I will try this:

If we stick with package references and fleet_packages.json file, but have the logic to select latest/indexed revisions from the Package Storage v2. The job can be responsible for updating the fleet_packages.json file, based on its knowledge of EPR search indices.

During the build time, Kibana would pull referenced .zip files, unpack them, and alter package versions if needed (configurable for apm, not sure about others).

Long story short, we would need:

1. Sync-bundled-packages to update fleet_packages.json.
2. Modified Kibana build logic to pull packages from a different place (v2) and update the package version.

[axw]

I think what @mtojek wrote above sounds like a good approach. We would still need to verify the integrity of the package with this approach, since there's a download involved - but this would be done at bundling time rather than run-/installation time.

Perhaps we could extend fleet_packages.json with:

• a specific package download URL (maybe EPR v1, maybe EPR v2)
• SHA256 of package zip
• whether or not to rewrite version to Kibana's

[mtojek]

Sounds like a plan. I will start preparing the sync job in the package-storage-infra.

[kpollich]

I'll take a pass at adding this new functionality to fleet_packages.json today.

[kpollich]

I put together a first pass at the "align package version to Kibana's" behavior here: #139567. @axw I think this satisfies the initial asks from this issue (prior to our discussions about higher level rethinking of how we work on bundled packages), but please let me know if there's anything else we can do in the immediate term to unblock APM in 8.5.

[dannycroft]

@kpollich I'm removing the Team:apm label from this as it seems that APM UI isn't needed here. Please re-add if I'm mistaken.

[axw]

I put together a first pass at the "align package version to Kibana's" behavior here: #139567. @axw I think this satisfies the initial asks from this issue (prior to our discussions about higher level rethinking of how we work on bundled packages), but please let me know if there's anything else we can do in the immediate term to unblock APM in 8.5.

@kpollich thanks, looks good overall!

We can always rework it later, but I think it would be better to have forceAlignStackVersion only rewrite the version, and not influence the version of the package pulled from EPR. For the latter, can we specify "version": "latest" in fleet_packages.json?

That would take us in the direction of #137759 (comment), where we would set a specific version in fleet_packages.json to pull from EPR for reproducibility.

[mtojek]

For the latter, can we specify "version": "latest" in fleet_packages.json?

I understand that "version": "latest" would download the latest available package? This could possibly replace the sync-bundled-packages job and let us get rid of automated PRs, but not sure if it's transparent and reproducible enough (which package has been pulled out eventually).

That would take us in the direction of #137759 (comment), where we would set a specific version in fleet_packages.json to pull from EPR for reproducibility.

👍

Let me know what you think as it impacts the work I've planned to do (implement the sync job).

BTW Could we remove that comment block in fleet_packages.json and put it aside :)? It doesn't look like a valid JSON...

[axw]

I understand that "version": "latest" would download the latest available package? This could possibly replace the sync-bundled-packages job and let us get rid of automated PRs, but not sure if it's transparent and reproducible enough (which package has been pulled out eventually).

@mtojek I only meant as an interim step, until the sync-bundled-packages work is done. I still think it would be a good idea to pin kibana to specific integration package versions.

[kpollich]

BTW Could we remove that comment block in fleet_packages.json and put it aside :)? It doesn't look like a valid JSON...

We can make the file .jsonc to be explicit (will also make GitHub not complain), but we parse it with https://json5.org/ like we do most other Kibana config files (tsconfig.*.json for example), in order to support comments. We don't have any other .jsonc files in the repo though that I'm aware of, and it seems to be an established convention in Kibana to use .json even if we're parsing with JSON5.

We can always rework it later, but I think it would be better to have forceAlignStackVersion only rewrite the version, and not influence the version of the package pulled from EPR.

This makes sense, and I'd rather have every package in .fleet_packages.json define a version anyway.

For the latter, can we specify "version": "latest" in fleet_packages.json?

If the sync packages CI task work isn't too far out, then I don't see a need to have support for latest in fleet_packages.json. It sounds like APM isn't going to use latest, and I don't know that any of the other packages will need to do so. Seems safer to just wait for the CI job and continue supporting only pinned versions in the interim.

[mtojek]

This makes sense, and I'd rather have every package in .fleet_packages.json define a version anyway.

👍

If the sync packages CI task work isn't too far out, then I don't see a need to have support for latest in fleet_packages.json. It sounds like APM isn't going to use latest, and I don't know that any of the other packages will need to do so. Seems safer to just wait for the CI job and continue supporting only pinned versions in the interim.

I'm working on this here. I'm if we should expand the file structure:

[
  {
    "name": "apm",
    "version": "8.4.0-preview-1234567",
    "source": {
       "repository_name": "package-storage-v2",
       "download_url": "https://package-storage.elastic.co/artifacts/packages/apm-8.4.0-preview-1234567.zip",
       "signature_url": "https://package-storage.elastic.co/artifacts/packages/apm-8.4.0-preview-1234567.zip.sig"
    },
    ...
  },

This way we could get rid of the logic pointing to epr-snapshot from Kibana and start using signed artifacts (a concern raised by @axw).

[kpollich]

We can always rework it later, but I think it would be better to have forceAlignStackVersion only rewrite the version, and not influence the version of the package pulled from EPR.

Updated #139567 with this behavior in mind. We require a version in .fleet-packages.json and Kibana will only fetch that version. The rewriting of the archive path/version logic remains in tact.

[mtojek]

The sync-bundled-packages job is waiting for reviews too (internal link above).

[mtojek]

@kpollich I'm polishing the implementation of the sync-bundled-packages-job, but it looks like every necessary piece is in place now. Do you think that we can resolve this issue?

[kpollich]

Yep that sounds good to me. I think we can close this out.