Support package deprecation

[mtojek]

With 8.x.x we may want to deprecate some package or data streams and create new alternatives. We need to discuss many things around, including publishing and installation rules of deprecated policies and "deprecated" captions.

[mtojek]

There is a similar thread started here: #192 , but I guess we need a more formal design (nothing adhoc).

[jsoriano]

It'd be nice to have some metadata for the deprecation, so maybe the UIs can provide a better experience to users. Without explicit metadata it may be difficult for Kibana to show something meaningful to users unless they explicitly check the titles or descriptions package by package.

If explicit metadata exists, Kibana may warn the user about needed actions as soon as it discovers that an installed package is deprecated.

For example having something like this in the manifest could indicate the user what to do, even providing links to the new packages or even eventually providing automatic migrations.

deprecation:
  reason: replaced
  comment: "The package has been split into separate packages: Microsoft DHCP and Microsoft Defender Endpoint."
  replacedBy:
    - microsoft_dhcp
    - microsoft_defender_endpoint

If done at the data stream level, and assuming a compatible configuration, it might be easier to provide an automated migration path. With the following information at some point kibana could know how to automatically upgrade from the old microsoft.dhcp datastream to the new microsoft_dhcp.log:

deprecation:
  reason: replaced
  comment: "This data stream has been moved to a new package: Microsoft DHCP"
  replacedBy:
    package: microsoft_dhcp
    data_stream: log

For packages (or data streams) that are just discontinued:

deprecation:
  reason: discontinued
  comment: "This package has been discontinued, it won't receive further updates and may stop working in the future."

[mtojek]

I like this approach. It's consistent with the "skip" structure for system tests: https://github.com/elastic/integrations/blob/153945daf72d6ef0f6fce53346daf0e4ef9b1753/packages/aws/data_stream/ec2_metrics/_dev/test/system/test-default-config.yml#L1

[jlind23]

Could we also add a new reason like "Going to be deprecated soon" if it is linked to a product that will soon be EOL for example?

[jsoriano]

Could we also add a new reason like "Going to be deprecated soon" if it is linked to a product that will soon be EOL for example?

Umm, thinking about this, I think that we should consider a package or data stream marked as deprecated as "Going to be unsupported and/or unmaintained soon because of reason". The reason should help the user (and the UI or other tooling) to know what to expect and/or what to do.
Adding another level of "going to" may be not neccesary, it would also mean that we need to communicate two things: when the package is going to be deprecated and when the package is actually deprecated, it can be confusing.

If the deprecation is caused because the monitored product is going to be EOL, we can use the discontinued reason and use the comment to explain the situation:

deprecation:
  reason: discontinued
  comment: "_The product_ will be EOL in _some date_ and this package won't be further maintained" 

There may be the case where a package is deprecated because is going to be reimplemented, but the replacement doesn't exist yet, or there is no automatic migration path, in that case I think that it would be ok to don't declare a replacement, just:

deprecation:
  reason: replaced
  comment: "This data stream is going to be reimplemented"

This could be understood as: "the data stream is going to be removed, a new implementation will be provided but it is not ready yet". A message like this could be literally shown to a new user as a warning when configuring a policy with this data stream.
Later on a new version could update this information when the replacement is more clear.

@jlind23 would these definitions cover your use case?

[jlind23]

@jsoriano perfectly indeed! Another question that we should ask to @akshay-saraswat:

• Shall we let inside the registry packages that are over deprecation date?
  Work on some kind of clean up policy. wdyt?

[jsoriano]

• Shall we let inside the registry packages that are over deprecation date?
  Work on some kind of clean up policy. wdyt?

I would leave this for a future discussion. In general I think that it is not nice to remove published versions (apart of removing harmful or illegal content), but we may consider a clean up policy for practical reasons.

[ruflin]

One core feature related to this needed in Fleet is that even if a package is removed from the registry, Kibana will still keep working assuming the package was installed before. The package cannot be upgraded anymore or reinstalled, but things keep working. Ideally Fleet would know from which registry it was installed and can indicate, that the package does not exists anymore in the remote registry.

[ruflin]

As a follow up, we likely have to expand the deprecation discussion to more granular parts which could be deprecated:

• integration policy
• data stream
• field
• ?

Each likely needs its own story on how it can be deprecated. Ideally we provide framework and guideline for each. Especially around fields discussions are already happening in various places.

[mtojek]

As an integration policy, do you mean a policy template?

In terms of spec changes, it should be relatively easy.

Changes in data streams and packages can be quick wins, as they both contain manifests. We can augment these files with a special "deprecation" structure.
Theoretically, we can introduce a similar flag to "field" (like we do in Beats).

I don't have any preference regarding Kibana/Fleet behavior. I guess we could redirect users of one package to another? or just suggest it to them?

[ruflin]

@mtojek Yes, policy templates. Should we discuss each of these topics in a separate issue as some might need work in Fleet, others don't and we can close one by one.

[jsoriano]

@mtojek Yes, policy templates. Should we discuss each of these topics in a separate issue as some might need work in Fleet, others don't and we can close one by one.

@ruflin what would be the use case of deprecating policy templates without deprecating the data_stream that contains it?

[klacabane]

Hi there, our team may have a use case for deprecated inputs.

The packages for the stack components (elasticsearch/kibana/logstash) currently have two inputs, one for metrics collection via a metricbeat module and one for logs. The metricbeat collection will eventually be superseded by a new approach that uses different metrics schema, assets and collection method, while the logs collection remains the same. When the new data streams are ready the package would allow a transition period where both metrics collection are available, with the metricbeat one marked as deprecated.
Once the deprecation period is done and we ship a new version that removes the input, it'd be nice to surface that in the UI when trying to upgrade to that version (maybe with a delta of the current/next version inputs, or an additional metadata flag) so that users can take actions before committing to a breaking change.

Does that sound like a valid use case ?

[jsoriano]

This sounds like a valid use case for this, yes 👍

I guess that in this case it would be nice to mark the input as deprecated as soon as the other one is ready, with a replacedBy rule like the ones proposed in #227 (comment). Then users would have a period while they can migrate.
I think the UI should warn about this as soon as it sees the deprecation notice.

[klacabane]

It's possible that the replacement won't be an elastic-agent input that collects the metrics, but a push-based model where the stack components would send metrics to apm server. The package would only be a shell for the assets in that case. The replacedBy setting could potentially point to the apm-server package but that doesn't sound like the right way. The replacedBy could point to the new data streams in the same package, but those would be automatically installed on upgrade so I don't see a benefit.
We probably don't need anything sophisticated for the deprecation metadata here besides a way to provide guidance on the next steps (a message like "the metrics collection now require a running apm-server and updates to the elasticsearch.yml file. Refer to the package documentation for the steps to follow").

What we could use is a dependency relationship with another package. If the new metrics collection needs another package to be setup, do we currently have a way to surface that ?

[jsoriano]

We probably don't need anything sophisticated for the deprecation metadata here besides a way to provide guidance on the next steps (a message like "the metrics collection now require a running apm-server and updates to the elasticsearch.yml file. Refer to the package documentation for the steps to follow").

Ah yes, then a single comment that is displayed to users may be enough.

What we could use is a dependency relationship with another package. If the new metrics collection needs another package to be setup, do we currently have a way to surface that ?

We don't have support for any dependencies between packages at the moment.

[jsoriano]

The need for package deprecation appeared again in elastic/integrations#3262 (comment).