[ML] Add an option to ML get APIs to return only the fields required by the corresponding put API #63055

droberts195 · 2020-09-30T10:35:00Z

Over the years many users have been frustrated that it is hard to programmatically get an ML job config from one cluster and then put it into a different cluster because the get jobs API returns augmented config documents containing more fields than the config required by the put job API. The put job API will reject configs that contain unknown fields. This creates a burden on the client implementing the transfer process to know which fields need to be kept, and this is especially difficult because the list varies from release to release.

We can address this by adding an option to the get jobs endpoint to return configs containing all the fields required by put job, but no more.

The same option and filtering functionality should be added to every other get endpoint we have that returns more than the corresponding put endpoint requires.

elasticmachine · 2020-09-30T10:35:02Z

Pinging @elastic/ml-core (:ml)

This adds the new `for_export` flag to the following APIs: - GET _ml/anomaly_detection/<job_id> - GET _ml/datafeeds/<datafeed_id> - GET _ml/data_frame/analytics/<analytics_id> The flag is designed for cloning or exporting configuration objects to later be put into the same cluster or a separate cluster. The following fields are not returned in the objects: - any field that is not user settable (e.g. version, create_time) - any field that is a calculated default value (e.g. datafeed chunking_config) - any field that would effectively require changing to be of use (e.g. datafeed job_id) - any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by) closes #63055

When exporting and cloning ml configurations in a cluster it can be frustrating to remove all the fields that were generated by the plugin. Especially as the number of these fields change from version to version. This flag, remove_generated, allows the GET config APIs to return configurations with these generated fields removed. relates to elastic#63055

…in GET config APIs (#63899) When exporting and cloning ml configurations in a cluster it can be frustrating to remove all the fields that were generated by the plugin. Especially as the number of these fields change from version to version. This flag, exclude_generated, allows the GET config APIs to return configurations with these generated fields removed. APIs supporting this flag: - GET _ml/anomaly_detection/<job_id> - GET _ml/datafeeds/<datafeed_id> - GET _ml/data_frame/analytics/<analytics_id> The following fields are not returned in the objects: - any field that is not user settable (e.g. version, create_time) - any field that is a calculated default value (e.g. datafeed chunking_config) - any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by) relates to #63055

…63092) This adds the new `for_export` flag to the following APIs: - GET _ml/anomaly_detection/<job_id> - GET _ml/datafeeds/<datafeed_id> - GET _ml/data_frame/analytics/<analytics_id> The flag is designed for cloning or exporting configuration objects to later be put into the same cluster or a separate cluster. The following fields are not returned in the objects: - any field that is not user settable (e.g. version, create_time) - any field that is a calculated default value (e.g. datafeed chunking_config) - any field that would effectively require changing to be of use (e.g. datafeed job_id) - any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by) closes elastic#63055

…in GET config APIs (elastic#63899) When exporting and cloning ml configurations in a cluster it can be frustrating to remove all the fields that were generated by the plugin. Especially as the number of these fields change from version to version. This flag, exclude_generated, allows the GET config APIs to return configurations with these generated fields removed. APIs supporting this flag: - GET _ml/anomaly_detection/<job_id> - GET _ml/datafeeds/<datafeed_id> - GET _ml/data_frame/analytics/<analytics_id> The following fields are not returned in the objects: - any field that is not user settable (e.g. version, create_time) - any field that is a calculated default value (e.g. datafeed chunking_config) - any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by) relates to elastic#63055

This adds a new flag `exclude_generated` for GET transform API. This flag is useful for when a transform needs to be cloned within a cluster or exported/imported between clusters. It removes certain fields that are not able to be set via the PUT api (e.g. version, create_time). relates #63055

…63093) This adds a new flag `exclude_generated` for GET transform API. This flag is useful for when a transform needs to be cloned within a cluster or exported/imported between clusters. It removes certain fields that are not able to be set via the PUT api (e.g. version, create_time). relates elastic#63055

…63947) This adds a new flag `exclude_generated` for GET transform API. This flag is useful for when a transform needs to be cloned within a cluster or exported/imported between clusters. It removes certain fields that are not able to be set via the PUT api (e.g. version, create_time). relates #63055

…ields in GET config APIs (#63899)(#63092) (#63177) * [ML] adding for_export flag for ml plugin GET resource APIs (#63092) This adds the new `for_export` flag to the following APIs: - GET _ml/anomaly_detection/<job_id> - GET _ml/datafeeds/<datafeed_id> - GET _ml/data_frame/analytics/<analytics_id> The flag is designed for cloning or exporting configuration objects to later be put into the same cluster or a separate cluster. The following fields are not returned in the objects: - any field that is not user settable (e.g. version, create_time) - any field that is a calculated default value (e.g. datafeed chunking_config) - any field that would effectively require changing to be of use (e.g. datafeed job_id) - any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by) closes #63055 * [ML] adding new flag exclude_generated that removes generated fields in GET config APIs (#63899) When exporting and cloning ml configurations in a cluster it can be frustrating to remove all the fields that were generated by the plugin. Especially as the number of these fields change from version to version. This flag, exclude_generated, allows the GET config APIs to return configurations with these generated fields removed. APIs supporting this flag: - GET _ml/anomaly_detection/<job_id> - GET _ml/datafeeds/<datafeed_id> - GET _ml/data_frame/analytics/<analytics_id> The following fields are not returned in the objects: - any field that is not user settable (e.g. version, create_time) - any field that is a calculated default value (e.g. datafeed chunking_config) - any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by) relates to #63055

…in GET config APIs (elastic#63899) When exporting and cloning ml configurations in a cluster it can be frustrating to remove all the fields that were generated by the plugin. Especially as the number of these fields change from version to version. This flag, exclude_generated, allows the GET config APIs to return configurations with these generated fields removed. APIs supporting this flag: - GET _ml/anomaly_detection/<job_id> - GET _ml/datafeeds/<datafeed_id> - GET _ml/data_frame/analytics/<analytics_id> The following fields are not returned in the objects: - any field that is not user settable (e.g. version, create_time) - any field that is a calculated default value (e.g. datafeed chunking_config) - any field that is automatically set via another Elastic stack process (e.g. anomaly job custom_settings.created_by) relates to elastic#63055

…63093) This adds a new flag `exclude_generated` for GET transform API. This flag is useful for when a transform needs to be cloned within a cluster or exported/imported between clusters. It removes certain fields that are not able to be set via the PUT api (e.g. version, create_time). relates elastic#63055

droberts195 added >enhancement :ml Machine learning labels Sep 30, 2020

benwtrent self-assigned this Sep 30, 2020

This was referenced Sep 30, 2020

[ML] adding for_export flag for ml plugin GET resource APIs #63092

Merged

[Transform] add new exclude_generated flag to GET transform #63093

Merged

benwtrent closed this as completed in #63092 Oct 2, 2020

benwtrent mentioned this issue Oct 19, 2020

[ML] add new flag exclude_generated that removes generated fields in GET config APIs #63899

Merged

stevejgordon mentioned this issue Dec 17, 2020

7.11.0 Meta Ticket elastic/elasticsearch-net#5198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Add an option to ML get APIs to return only the fields required by the corresponding put API #63055

[ML] Add an option to ML get APIs to return only the fields required by the corresponding put API #63055

droberts195 commented Sep 30, 2020

elasticmachine commented Sep 30, 2020

[ML] Add an option to ML get APIs to return only the fields required by the corresponding put API #63055

[ML] Add an option to ML get APIs to return only the fields required by the corresponding put API #63055

Comments

droberts195 commented Sep 30, 2020

elasticmachine commented Sep 30, 2020