Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Adding job overrides to the module setup endpoint #42946

Conversation

jgowdyelastic
Copy link
Member

@jgowdyelastic jgowdyelastic commented Aug 8, 2019

Adds the ability to supply the /api/ml/modules/setup/{moduleId} endpoint with job and datafeed overrides in the form or partial job and datafeed configurations.
e.g. this will apply model_memory_limit: 13mb to all jobs in the module.

"jobOverrides": {
  "analysis_limits": {
     "model_memory_limit": "13mb"
  }
}

The overrides can be a single object or array of objects. If the object contains a job_id or datafeed_id, the overrides contained in that object will only be applied to the matching job or datafeed in the module.
If no id is specified, the overrides in the object will be applied to all jobs or datafeeds in the module.

The below example will set the bucket_span to 15m in all jobs and the model_memory_limit to 13mb for the job visitor_rate_ecs
It will also set the scroll_size to 1001 for all datafeeds and the frequency to 30m for the datafeed for the job visitor_rate_ecs
Rather than using the job_id in the datafeed, you could also have used:
"datafeed_id": "datafeed-visitor_rate_ecs"

{
  "jobOverrides":[{
    "job_id": "visitor_rate_ecs",
    "analysis_limits": {
      "model_memory_limit": "13mb"
    }
  }, {
    "analysis_config" : {
      "bucket_span": "15m"
    }
  }],
  "datafeedOverrides": [{
    "scroll_size": 1001
  }, {
    "job_id": "visitor_rate_ecs",
    "frequency": "30m"
  }]
}

These overrides are applied before any of the existing overridable options (e.g. useDedicatedIndex, groups, indexPatternName etc) and so can be overridden themselves.
In the below example, the match_all query will be used over the query passed in via the datafeedOverrides

{
  "query": {"bool":{"must":[{"match_all":{}}]}}
  "datafeedOverrides": {
    "query": {}
  }
}

Functional tests are needed for this change, which will be added in a follow up PR.

cc @weltenwort

Fixes #42409

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

- [ ] This was checked for cross-browser compatibility, including a check against IE11
- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support

For maintainers

@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui

@jgowdyelastic jgowdyelastic marked this pull request as ready for review August 8, 2019 15:57
@jgowdyelastic jgowdyelastic requested a review from a team as a code owner August 8, 2019 15:57
@elasticmachine
Copy link
Contributor

💔 Build Failed

@jgowdyelastic
Copy link
Member Author

retest

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Copy link
Contributor

@alvarezmelissa87 alvarezmelissa87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall 👍 - just added a couple of small comments

Copy link
Contributor

@walterra walterra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jgowdyelastic jgowdyelastic merged commit 701996d into elastic:master Aug 9, 2019
jgowdyelastic added a commit to jgowdyelastic/kibana that referenced this pull request Aug 9, 2019
* [ML] Adding job overrides to the module setup endpoint

* removing text code

* throw error for incompatible data

* changes based on review
jloleysens added a commit to jloleysens/kibana that referenced this pull request Aug 9, 2019
…p-metrics-selectall

* 'master' of github.com:elastic/kibana: (306 commits)
  [ML] Adding job overrides to the module setup endpoint (elastic#42946)
  [APM] Fix missing RUM url (elastic#42940)
  close socket timeouts without message (elastic#42456)
  Upgrade elastic/charts to 8.1.6 (elastic#42518)
  [ML] Delete old AngularJS data visualizer and refactor folders (elastic#42962)
  Add custom formatting for Date Nanos Format (elastic#42445)
  [Vega] Shim new platform - vega_fn.js -> vega_fn.js , use ExpressionFunction (elastic#42582)
  add socket.getPeerCertificate to KibanaRequest (elastic#42929)
  [Automation] ISTANBUL PRESET PATH is not working fine with constructor(private foo) (elastic#42683)
  [ML] Data frames: Updated stats structure. (elastic#42923)
  [Code] fixed the issue that the repository can not be deleted in some cases. (elastic#42841)
  [kbn-es] Support for passing regex value to ES (elastic#42651)
  Connect to Elasticsearch via SSL when starting kibana with `--ssl` (elastic#42840)
  Add Elasticsearch SSL support for integration tests (elastic#41765)
  Fix duplicate fetch in Visualize (elastic#41204)
  [DOCS] TSVB and Timelion clean up (elastic#42953)
  [Maps] [File upload] Fix maps geojson upload hanging on index step (elastic#42623)
  [APM] Use rounded bucket sizes for transaction distribution (elastic#42830)
  [yarn.lock] consistent resolve domain (elastic#42969)
  [Uptime] [Test] Repurpose unit test assertions to avoid flakiness (elastic#40650)
  ...
jgowdyelastic added a commit that referenced this pull request Aug 9, 2019
* [ML] Adding job overrides to the module setup endpoint

* removing text code

* throw error for incompatible data

* changes based on review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[ML] Allow for injection of job parameters upon module setup
5 participants