Instrument Kibana with Elastic APM #43548

watson · 2019-08-19T21:13:14Z

Summary

Instruments Kibana with Elastic APM by adding the Node.js agent to the source code. The agent is not turned on by default but can be enabled by setting the environment variable ELASTIC_APM_ACTIVE=true or by creating an apm config file called config/apm.dev.js and setting active: true inside of it.

This implementation is not meant to be used by end-users of Kibana as it lacks integration with the regular Kibana config file. For now, this is meant as a useful internal tool for Elastic employees when developing Kibana.

By default, it's pre-configured with a serverUrl pointing to an APM Server hosted on Elastic Cloud. The data is stored in an ES cluster accessible only by Elastic employees. These defaults can easily be overwritten using environment variables or via the custom config file.

Closes #32491

Checklist

For maintainers

This was checked for breaking API changes and was labeled appropriately
~~- [ ] This includes a feature addition or change that requires a release note and was labeled appropriately~~

mistic

LGTM from the operations team changes perspective

jbudz · 2019-09-03T14:29:35Z

I know there was some discussion here and will defer to @elastic/kibana-security, but I'd push for the default localhost or the apm server to allow anonymous requests. Mostly to avoid us showing up on any of the lists that scrape github.

legrego · 2019-09-03T14:38:55Z

I know there was some discussion here and will defer to @elastic/kibana-security, but I'd push for the default localhost or the apm server to allow anonymous requests. Mostly to avoid us showing up on any of the lists that scrape github.

I agree with @jbudz that we should either default to localhost, or include no default and require the developer to specify their own endpoint if they choose to enable APM. End-users would need to do this anyway if/when we make this a public feature, so I don't feel like it's that big of an ask. Exposing the URL via environment variable makes it easy for folks to setup their development environment once, without messing with their config files if they don't want to.

tylersmalley · 2019-09-03T14:43:51Z

@legrego, the idea for when this would be made public would be that we would be the ones collecting this information to understand the performance metrics of Kibana in the wild. Having the default provided seems to be the only way to accomplish this. Is there a security issue we see with including this default?

tylersmalley · 2019-09-03T14:46:47Z

Tested the PR, looks good to me. There are a couple things which would be nice to have that can be a follow-up PR which I am happy to do if it's not done here:

Enable for CI: We can set environment variables here
Add hostname/IP tagging to help isolate your own metrics.

elasticmachine · 2019-12-02T11:43:09Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 4f34863

rudolf

👍 For Platform changes

sorenlouv

☕️+ 📊= 🚀

watson · 2019-12-04T13:06:09Z

@elasticmachine merge upstream

elasticmachine · 2019-12-04T14:17:46Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 8bd4762

Instruments Kibana with Elastic APM by adding the Node.js agent to the source code. The agent is not turned on by default but can be enabled by setting the environment variable `ELASTIC_APM_ACTIVE=true` or by creating an apm config file called `config/apm.dev.js` and setting `active: true` inside of it. This implementation is not meant to be used by end-users of Kibana as it lacks integration with the regular Kibana config file. For now, this is meant as a useful internal tool for Elastic employees when developing Kibana. By default, it's pre-configured with a `serverUrl` pointing to an APM Server hosted on Elastic Cloud. The data is stored in an ES cluster accessible only by Elastic employees. These defaults can easily be overwritten using environment variables or via the custom config file.

mistic · 2019-12-04T16:55:36Z

@watson I know that was already merged but after passing a second time on the code I don't think it is a good idea to have a secret token stored in the repo. It's not the end of the world but I think it is not a good idea by 2 reasons:

1 - anyone can broke our collected data with wrong data if they want
2 - if someone decides to play with it we can have an undesired bill on GCP

I think maybe we should consider creating a new secret token, open a new PR and replace the secret token with a place holder and comment with a link to the place on the GCP where we could get the secret token by ourselves if we want. Finally I think we should also add apm.dev.js to the gitignore so no one could add again the secret token by mistake into the repo.

Let me know your thoughts!

tylersmalley · 2019-12-04T17:04:44Z

@mistic there is some background on that decision here: #43548 (comment)

kobelb · 2019-12-04T17:08:58Z

Reading back through the previous discussions, I don't think we fully evaluated the following risk which @mistic raised, which we should consider:

2 - if someone decides to play with it we can have an undesired bill on GCP

The previous discussions assumed that if we could allow anonymous users to submit APM data, that we would like to. Are there any limits put in place on APM or GCP which prevent gross abuse which would lead to an undesirable bill in GCP?

watson · 2019-12-04T17:16:37Z

Even without a secretToken, anybody who knows the APM Server URL can send data to it that will be stored in Elasticsearch as the RUM agent doesn't require one (I originally wanted to remove the requirement to have a secretToken for the Node.js agent as well, but this isn't possible on Elastic Cloud). This is a similar problem that Google has with Google Analytics and there's in general no good solution for it.

The reason why we hardcoded both serverUrl and secretToken in this PR was to improve our chances that our devs would actually use it (by lowering the barrier of entry). We did consider that someone might misuse it and send us garbage data, but our thought at the time was that we would cross that bridge when we got to it. In that case, we could decide to shut down the cluster and make a new one with a URL we didn't share publicly. But if it turns out to not be an issue, we would rather have the lower barrier to entry.

So as long as we pay attention to the cluster usage, my vote is still to keep it as is. What do you think?

kobelb · 2019-12-04T19:05:35Z

So as long as we pay attention to the cluster usage, my vote is still to keep it as is. What do you think?

I think this is reasonable. Is this just a normal Elasticsearch Service cluster? If so, it should already have a fixed disk/CPU, so we're not talking about an unbounded amount of resources, right?

watson · 2019-12-04T20:19:43Z

@kobelb That's my understanding yes, but @sqren has access to the Cloud admin panel for the cluster and can confirm.

sorenlouv · 2019-12-04T23:09:00Z

Yes, just the standard APM instance on ESS. I'll keep an eye on it. Maybe setup some alerts for specific OS metrics.

fbaligand · 2019-12-11T05:49:15Z

Hi,

Very interesting feature! Not only for Kibana team but also for Kibana plugin developers (like me) !

What is the environment variable to set APM server URL? Is this documented?

sorenlouv · 2019-12-11T08:25:43Z

@fbaligand Yes, it's documented in https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#instrumenting-with-elastic-apm.

If you want to send the data to your own apm server you can do it by creating config/apm.dev.js with:

module.exports = {
  active: true,
  serverUrl: 'your apm server url'
};

You can find the default values here

fbaligand · 2019-12-11T09:14:19Z

Great! Thanks!

watson · 2019-12-11T09:24:34Z

@fbaligand We're also working on adding client-side instrumentation with the RUM agent in this PR: #44281 - this might also be relevant for you as a plugin developer

fbaligand · 2019-12-11T10:00:02Z

Yes! Clearly!
Especially for my plugin that is full front-side :)

* [SIEM] [Detection Engine] Add edit on rule creation (#51670) * Add creation rule on Detection Engine * review + bug fixes * review II + clean up * fix persistence saved query * fix eui prop + add type security to add rule * fix more bug from review III * review IV * add edit on creation on rule * review * fix status icon color * fix filter label translation * [ML] Re-activate after method in transform test (#51815) * Move saved queries service + language switcher ⇒ NP (#51812) * Move saved queries service + language switcher to NP * test fixes * test fix * fix ts * mock search service * Allow routes to define some payload config values (#50783) * Allow routes to define some payload config values * Documentation typo * Move hapi `payload` config under `body` + additional validations * Update API docs * Amend explanation in API docs * Add stream and buffer types to @kbn/config-schema * Fixes based on PR feedback: - Add 'patch' and 'options' to valid RouteMethod - Add tests for all the new flags - Allow `stream` and `buffer` schema in the body validations (findings from tests) * API documentation update * Fix type definitions * Fix the NITs in the PR comments + better typing inheritance * API docs update * Fix APM-legacy wrapper's types * Fix KibanaRequest.from type exposure of hapi in API docs * Move RouterRoute interface back to private + Expose some public docs * Update @kbn/config-schema docs * [Dependencies]: upgrade react to latest v16.12.0 (#51145) * fixes timeline data providers tests (#51862) * [Discover] Improve Percy functional tests (#51699) * Implement new wait for chart rendered function * Add findByCssSelector to ensure the charts have been rendered * Fix error returned when creating an alert with ES security disabled (#51639) * Fix error returned when creating an alert with ES security disabled * Add test to ensure error gets thrown when inner function throws * Enable alerting and actions plugin by default (#51254) * Enable alerting and actions plugin by default * Fix test failure * Fix features test * Unify uiSettingsClient contracts (#51167) * introduce IUiSettingsClient on the client * switch uiSettings service to IUiSettingsClient * update uiSettings service tests * all plugins should use IUiSettingsClient * stop is not public methods anymore * remove savedobject attribute type references * regen docs * remove all references to UiSettingsClient class * regen docs * add migration example for uiSettings * update consumer types and tests * address comments * add eslint rule banning the core to import plugin code (#51563) * add eslint rule banning the core to import plugin code * Ban importing legacy plugin code in the сore * fix eslint errors * core cannot import xpack rule * regen docs * [ML] Anomaly Explorer: Deprecate explorer_controller (#51269) The main goal of this PR is to get rid of explorer_controller.js to unblock the migration to react-router. Previously we already used rxjs observables to migrate away from angular events. Observables were used to trigger actions to manage the react component's state as well as AppState. This PR builds upon this previous work. The actions already were done similar to redux, now the use of observables has been extended to use scan (see rxjs docs) which allows us to transform the actions into state updates. * [ML] Add Custom URLs and Calendars to the Job wizards (#51281) * [ML] editor to ts * [ML] custom_urls component to ts * [ML] custom urls in job creator, define CustomSettings interface * [ML] custom urls selection * [ML] custom urls description text * [ML] description for calendars * [ML] update layout to full width * [ML] fix i18n * [ML] adjust editor test * [ML] fix type for FormattedMessage * [ML] fix types * [ML] add modal edit mode to the custom urls editor * [ML] fix calendars saving * [ML] update snapshot * [ML] add test support * [ML] update form description text * [ML] adjust layout * [ML] set modal width * [ML] fix total value condition in getTestUrl * [ML] fix token replacement for string values * [ML] align calendar selection * [ML] manage calendars link * [ML] fix form description ids * [ML] styles for custom urls description * [ML] manage calendars as a link * [ML] refresh calendars button * [ML] remove redundant css rule * Revert "[ML] remove redundant css rule" This reverts commit 6b9a50e0 * [ML] update calendar service, move calendars out of the job config * [ML] fix translation key * [ML] fix i18n * [ML] alight calendars selection * [ML] calendar types * [ML] update calendars description * [ML] calendars cloning * [ML] align additional settings with advanced * [ML] fix typo, clean up calendars * [ML] persist created_by for custom urls * [ML] fix fetchCalendarsByIds * [ML] extend stashCombinedJob with calendars * [ML] remove redundant check and fallback for calendars * Add new eslint rules for TS (#51876) * [ML] Fixes loading of data visualizer with KQL saved search (#51882) * [Discover] shim with local application service (#49483) Shims discover app within local angular instance (or so called inner angular in the code) to enable migration to the new platform without getting rid of all Angular. Note that apart from Discover's main app, also context, doc view and embeddable are part of this shim. There a 2 sorts of local angular used, one for the main plugin, and a slimmer one for it's embeddable, that's needed to render the data table. Both are built on first demand. There are 2 changes of URLs inside /context/ becomes /discover/context /doc/ becomes /discover/doc Legacy url's are redirected, so they are still valid. * Migrate UiSettingsClient type to IUiSettingsClient (#51913) * [ML] Changing job message info icon (#51904) * [NP] Move IndexPatterns into the NP (#51199) * Make a copy of index_patterns module inside NP * Sync changes of index patterns * Remove UI component from service * Remove redundant exports from index_patterns * Sync old index_patters changes * Make common namespace of IndexPattern * Make re-exports of old IndexPatter dependencies * Remove utils and error of the old indexPattern * Plug InterPatternService in data plugin * Make shortenDotString in new data plugin * Remove field folder * Fix wrong merge * Remove index_pattern folder * Remove _LEGACY from service * Remove static code from service * Fix start contract * Remove services from the old plugin * Remove index_patterns_service * Fix filter.meta.index type * Fix reviews * Fix type error * Revert "[NP] Move IndexPatterns into the NP (#51199)" This reverts commit 3a0d5f4bf54f91129e70a82d7aeac6fa6ee2511a. * Newplatform/move index patterns to new plugins 2 (#51940) Re-merging a fixed merge PR by @Avinar-24 * Fix type error with jest tests (#51925) * [APM] Add script for creating roles for local dev use (#51824) * Remove usages of `@kbn-ui-framework` (#51696) * Replace StaticIndexPattern with IIndexPattern from NP (#51872) * replace SIP with IIP * import path * [Vis] TableVis uses local angular (#50759) * TableVis uses local angular * Clean up * Fix TS * Update angular_config * Fix export * Update render_app.ts * Cetralize ui deps * Fix loading KbnTableVisController in Dashboard * Fix graph * Rename const * Add table vis mocks * Fix kbn_top_nav * Add TS for test * Complete conversion paginated_table test to Jest * Convert table_vis_controller test to Jest * Convert table_vis_controller test to Jest * Create agg_table.test.ts * Fix mocha tests * Refactoring * Remove module dep * Remove LegacyDependenciesPlugin * Move file * Fix path * Fix path * Fix TS * Fix Jest test * [TSVB] Fix TSVB not saving correctly (#51887) * Fix TSVB not saving correctly * Address review * Add functional test * [ML] Adds functional tests for the index based data visualizer (#51832) * [ML] Adds functional tests for the index based data visualizer * [ML] Address comments from review on data viz tests * [ML] Skip data visualizer functional tests on Firefox * Fixed NaN comparison (#50000) * refactor: replace 'change' to arrow in index pattern selector (#51954) * move saved query UI components ⇒ NP (#51870) * move saved query UI components to NP * Fixed search bar test * Added service to siem mock * change import css * Http compression config (#50738) HTTP compression for responses may be disabled entirely, or it may be conditionally enabled based on a whitelist of trusted referrers. * [Maps] Move vector-style component to vector folder (#51779) * [Maps] fix too_many_buckets_exception for top hits (#51497) * [Maps] fix too_many_buckets_exception for top hits * spelling * do not change re-fetch behavior when zooming in * i18n cleanup * review feedback * Removed duplicate attribute (#50001) * Update kibana.yml explanation of elasticsearch.ssl (#51451) * [DOCS] Fixes xpack-ref links (#51936) * [APM] Add NP feature catalog registration (#51765) * Move register_feature.js to legacy_register_feature.js * Add feature catalog registration to NP plugin * Add plugin dependencies in kibana.json Fixes #51542. * [ML] Fixing info content detector field selection (#51914) * [ML] Fixing info content detector field selection * fixing test * fix tab url link (#51993) * Experimenting with delay in test startup (#51459) * Experimenting with delay in test startup * Move firefox tests sooner becuase they take a long time to run * Move functional test parallel process delay to a spot that should work better for flaky test runner * This app is an entirely different app on cloud. Disabling. (#51844) * [Canvas] New Platform shim Canvas Client Side (#49358) * New Platform shim Canvas * Fix typing * Use some new platform methods * Fix formatting issue * disable visual regression tests * Prevents xpack.task_manager.index being set to .tasks (#52002) A quick solution in preventing what was described in kibana#47716 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co> * [Maps] Refactor style editor to use style property class instead of style descriptor object (#52001) * [Maps] pass style property object instead of style descriptor to style editer components * pass symbol descriptor * cleanup from merge * [APM] getInjectedVars shim (#51635) * [APM] getInjectedVars shim Set up the APM public NP plugin to expose the config variables on its context, and replace use of getInjectedVars with that. Since we're not yet running as an NP plugin, we don't get passed a `pluginInitializerContext`, so we use a shim in the plugin setup that gets the config values from injected vars for the time being. Also: * Move toggle app link in nav shim to plugin setup * Replace the routes exported from Main/route_config with a function that takes a configuration object * [Doc]Replace outdated screenshots and change graphic format (#52017) * Replace outdated screenshots and change graphic format * Added screenshot role per review comments * Clear cache when updating indices via UI (#51767) * Clear cache when updating indices via UI * Removes .ts * Update App Arch CODEOWNERS with legacy plugins. (#51502) * [SIEM][Detection Engine] Switches actions/alerting on by default with SIEM (#51985) ## Summary * Removes environment variables to have alerting/actions be defaulted on within SIEM * Updates documentation * Removes unit tests for it ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ - [x] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process) * [APM] metadata.branch shim (#51770) Set up the APM public NP plugin to expose the stack version Since we're not yet running as an NP plugin, we don't get passed a pluginInitializerContext, so we use a shim in the plugin setup that gets the branch values from ui/metadata for the time being. Fixes #49327. * Migrate Capabilities to new platform (#51438) * create service skeleton * move registerCapabilitiesModifier to capabilities service and rename to registerCapabilitiesSwitcher * starts to move capabilities logic to CapabilitiesService * move capabilities route to service * add initial integration test for capabilities route * capabilitiesMixin now delegates to capability service * use server-side Capabilities import in server code * update generated doc * remove capabilities from injectedMetadatas * use applications sent from client instead of server-registered navLinks * disable authRequired for capabilities route * (temp) exposes two endpoints for capabilities * Add fetch-mock on capabilities call for karma tests * adapt xpack Capabilities test - first attempt * adapt x-pack ui_capabilities test * add '/status' to the list of anonymous pages * Add documentation on Capabilities APIs * move Capabilities to core/types * update generated docs * add service tests * protecting resolveCapabilities against added/removed capabilities * update generated docs * adapt mocks due to rebase * add forgotten exports * improve capabilities routes registering * name capabilities registering methods * resolve conflicts due to merge * address review issues * add comment about reason for exposing two routes * extract createHttpServer test helper * fix merge conflicts * improve documentation * remove `/status` anon registration as now done in NP status plugin * fix merge conflicts * [APM] Pagination of top 10 trace samples (#51911) * adding trace pagination * adding trace pagination * refactoring * refactoring * Move configuration validation to new platform (#51880) * move configuration validation to NP * update generated docs * fix integration test * ensure correct prototype for custom error * typealiases KibanaConfig to LegacyConfig * migrate test file to ts * move getUnusedConfigKeys to NP * method no longer async * getUnusedConfigKeys: switch to named arguments * introduce CriticalError error type * Add home tutorial for ActiveMQ filebeat module (#52034) * Unskip legacy editor test (#51907) * Move QueryStringInput ⇒ NP (#51873) * Move query string input and query bar top row to NP * Fix query string input imports * import * Remove duplicate implementation of getFromSavedObject * import getFromSavedObject * Fixed import * code review * Remove css color * Kibana app migration: Remove old apis (#50881) * Upgrade markdown-it to 10.0.0 (#51492) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> * fix linting errors in graph (#50907) * [APM] Ensure security is enabled before creating roles and users (#52065) * [ML] Functional tests - add retry to click the add detector button (#52057) This PR adds a retry to the `openCreateDetectorModal` method in order to increase test stability. * [ML] DF Analytics: use new _explain endpoint to estimate model memory limit (#51644) * update analytics job creation to use _explain api endpoint for modelMemoryLimit estimate * debounce modelMemoryLimit estimation fetch * check for undefined searchQuery (#52032) * Fix bad apply_filter_action_test (#52078) * rename actionType attribute 'description' to 'name' (#51661) resolves https://github.com/elastic/kibana/issues/49826 * [ML] Functional tests - increase stability of source selection (#51988) This PR adds a retry to the ML source selection service method for functional tests. * [ftr/visualize] continue to wait for save completion by default (#51413) * [ftr/visualize] continue to wait for save completion by default * capture the toast message when saving the visualization so that it can be inspected if desired * fix/flaky visualize listing tests (#33) * move waitForSaveModalToClose from dashboard_page to common_page * call common_page.waitForSaveModalToClose() * remove extra wait for toast * fix trailing spaces * improve error message of failing test * [APM] Address fallout from NP server migration (#51725) * [APM] Address fallout from NP server migration * Fix type issues in integration tests * Await creation of agent configuration index * filter out downstream failures with updated error message (#51623) * extend the test timeout along with the startup timeout (#52110) * [Reporting] Convert Export Type Definitions to Typescript (#51643) * simplify ts * fix generate_png + get_full_url * fix pdf execute job * fix pdf create job * fix decrypt job headers * fix generate pdf / generate png * remove log * export consts * move export type registration to ts * more export type registration to ts * ts generics * remove console.log * use generics * fix ts * fix ts * fix ts * fix ts * Multi-type handling readability fix * Support createJob's jobParams * i18n fixes * track down mysterious field * revisit ts-ignores * remove an any type in get_conditional_headers * ts fixes * typed export treatment for csv_from_savedobject#executeJob * refactor helper function plain bonkers signature * i18n merge fix * add error handling test * todo * fix .headers type def * Reduce number of loc change * remove unused params from generic signatures * Remove as/any * hoist out GenericWorkerFn for naming * remove unnecessary fields from JobDocPayloadPanelCsv * Introduce user defined type guard * Clean up ui/courier exports. (#52037) * Add PR comments to pipeline (#51504) * chore(NA): fix licenses task on grunt test task (#51444) * chore(NA): fix configuration on jest test environment (#50854) * chore(NA): fix configuration on jest test environment * chore(NA): disable corejs * chore(NA): remove mocks from module path ignore patterns * skip flaky test (#43938) * [APM] `await` promise in setup script * [State Management] Typescripify, jestify, simplify state_hashing and state_storage (#51835) The hashUrl and unhashUrl functions no longer rely on states being provided as an argument, therefore getUnhashableStates/getUnhashableStatesProvider have been removed. * [Reporting/Visualisation] DOM attributes for reporting integration are missing (#52109) * static createRepository factory function (#52056) * static createRepository factory function * Fix API docs after master merge * [ML] Fix search for Transforms and Analytics tables (#52163) * [ML] fix TransformTable init * [ML] fix Analytics table * [APM] Add support for basepath (#52162) * Add Kibana Security setup to readme * [APM] Improve formatting in readme * [APM] Make it possible to link directly to a trace with just the trace.id (#51450) * create new api to fetch root transaction by trace id * redirecting trace to transaction * redirecting trace to transaction * redirecting trace to transaction * testing external link * testing external link * testing external link * testing external link * changing route name * refactoring * refactoring * refactoring * fixing merge conflicts * adding rangeFrom and to, into the url query param * removing convertedValue from duration formatter * refactoring ES query and tracelink component * pr comments * refactoring link * [ML] Perform cardinality check on enabling the model plot (#51915) * [ML] add a callout * [ML] reactive validateCardinality$ * [ML] change check to analysis_config * [ML] change comment * [ML] WIP check cardinality in job validator * [ML] refactor to use jobValidatorUpdated * [ML] rename vars * [ML] rename config fields * [ML] improve stream to cache and compare only analysis_config * [ML] simplify jobCreator subject * [ML] remove condition from effect * [ML] PR remarks * [Lens] Remove unused datasource methods (#51840) * Remove unused datasource methods * Remove unused function * [Lens] Make Lens plugin registry signatures consistent (#51839) * [SR] Support for SLM on Cloud (#51000) * [APM] Fix failing ACM integration test (#52149) * Disabled actions (#51975) * feat: disable actions from SIEM by using `disabledActions` list * feat: filter out actions specified in `disabledActions` input prop * test: 💍 remove legacy test * chore: 🤖 remove unused import * test: 💍 add disabledActions prop tests * Instrument Kibana with Elastic APM (#43548) Instruments Kibana with Elastic APM by adding the Node.js agent to the source code. The agent is not turned on by default but can be enabled by setting the environment variable `ELASTIC_APM_ACTIVE=true` or by creating an apm config file called `config/apm.dev.js` and setting `active: true` inside of it. This implementation is not meant to be used by end-users of Kibana as it lacks integration with the regular Kibana config file. For now, this is meant as a useful internal tool for Elastic employees when developing Kibana. By default, it's pre-configured with a `serverUrl` pointing to an APM Server hosted on Elastic Cloud. The data is stored in an ES cluster accessible only by Elastic employees. These defaults can easily be overwritten using environment variables or via the custom config file. * Functional tests: elastic chart provider (#52085) * adding elastic-chart service * update visual test * [ML] DF Analytics: create classification jobs via the UI (#51619) * wip: classification job config in form * ability to create classification job in form + validation * ensure classification types are correct for validation * update reducer test to include jobType validity check * update analytics jobs help text * update newJobCapsService to support boolean fields * Clean up uses of deprecated API's in node core (#51431) Ensure no deprecated Node.js core API's are used in Kibana. This is achieved by throwing an error in either development mode or in CI if one of the deprecated API's is called, and as such, new PR's should no longer be able to be merged if they use deprecated API's. Some of these API's (like the `Buffer` constructor`) is a security risk. * Silence Axe's complaints about missing labels for Console's textarea elements, in order to allow our automated a11y tests to pass. (#52150) * Add a section on naming conventions for Start and Setup (#51676) * Whitelist email server in built-in email server action (#51489) resolves https://github.com/elastic/kibana/issues/50721 Uses the same whitelist config value / utilities that the webhook action already uses. Was already mentioned in the README doc that email uses this whitelist config value :-) Required a change to the functional tests to use a host already whitelisted in config, made for the the webhook action tests. Also realized some jest tests on email were bogus, so fixed those (was passing `user` in config, which is invalid, and masking the actual thing being tested). * [ML] Add functional tests for analytics UI (#52054) This PR adds functional UI tests for outlier detection and regression job creation. * add links to plugin manifest file format. add naming conventions (#51280) * add links to plugin manifest format * remove obsolete comment * address comments. add registry folder to file structure tree * revert back not confirmed conventions * [Maps] Add basic NP shim to Maps app and expose map config (#51683) * Move plugin init calls to new plugin file, update refs and bind to legacy var. Assume ref for license init * Bind server calls to __LEGACY var and pass to plugin setup * Expose map config externally * Pass in NP features and licensing plugins to replace some xpack_main functionality * Pass through legacy server config for routes dependency * Expose map config for legacy plugins * Remove unused legacy refs * Update license check to conform with latest NP license practices * Clean up * Revert to old license check. Handle NP license checking in separate PR * Add back the license check file * [SIEM] [Detection Engine] updates shape of mitre attack threats when creating / updating detection engine rules (#52200) * [Canvas] Fixes axisConfig position argument UI (#50717) * add validation for nested filter in find API (#51847) * Revert "Whitelist email server in built-in email server action (#51489)" This reverts commit 4ce703e4090f18907513160c168f8f9e5f517b43. * skip flaky suite (#40670) * Remove index pattern requirement when doing a nested query in KQL (#52023) * Move new Code Editor component to kibana_react so it can be used through Kibana (#45914) * Fixes outdated URLs (#52104) * [Metrics-UI] Port graphql query for snapshot and node details to hapijs (#50730) * Port graphql query for snapshot to hapijs * fix type * Remove graphql from node details. * Use boom for error handling. * Remove unused imports * Remove dead code. Merge types * Fix tests * Remove unused import * Rename types * Go all in on io-ts. Cleanup types * Fix typecheck * Whitelist email server in built-in email server action - second try (#52221) resolves https://github.com/elastic/kibana/issues/50721 note this branch was previously merged into master and then reverted: https://github.com/elastic/kibana/pull/51489 (prior PR made shape changes this one didn't take into account) Uses the same whitelist config value / utilities that the webhook action already uses. Was already mentioned in the README doc that email uses this whitelist config value :-) Required a change to the functional tests to use a host already whitelisted in config, made for the the webhook action tests. Also realized some jest tests on email were bogus, so fixed those (was passing `user` in config, which is invalid, and masking the actual thing being tested). * Don't allow duplicate saved views with the same name (#52040) * Don't allow duplicate saved views with the same name * Change logic to make it a little easier to reason about * Change error names * Updating accessibility guide and contributing readmes (#52038) * updating accessibility guide and contributing readmes * updating dev docs * [SIEM] Adds Signals Table and additional configuration options to StatefulEventsViewer (#52044) ## Summary This is `Part I` of `II` for adding the `Signals Table` to the main Detection Engine landing page ([meta issue](https://github.com/elastic/kibana/issues/50405)). Breaking into two parts as this contains additional configuration options to the `StatefulEventsViewer` which will be used as part of https://github.com/elastic/kibana/issues/51016. `Part I` includes: * `SignalsTable` component that displays signals from the default signals index `.siem-signals` * Refactors `StatefulEventsViewer` to use `useFetchIndexPatterns` hook instead of `WithSource` * Adds ability to specify `alias` to `ColumnHeader` when providing column names * Adds the following new props to `StatefulEventsViewer` * `defaultIndices?: string[]` -- for specifying a different index than `siemDefaultIndex` * `headerFilterGroup?: React.ReactNode` -- for providing a component to display in the top right of the table (e.g. filter buttons, select, etc.) * `timelineTypeContext?: TimelineTypeContextProps` -- config for when creating a new table * `documentType?: string` -- user string for type of records displayed (e.g. Signals) * `footerText?: string` -- custom footer text for given document type * `showCheckboxes: boolean` -- whether or not to show selection checkboxes * `showRowRenderers: boolean` -- whether or not to show row renderers * `timelineType: TimelineType` -- type of Timeline for setting default columns * `title?: string` -- optional custom title * `utilityBar?: (totalCount: number) => React.ReactNode` -- optional param for providing your own custom `UtilityBar` instead of using the default `Showing xxx events`. `Part II` will add support for selection and overflow/batch actions. <img width="1548" alt="Screen Shot 2019-12-02 at 19 59 34" src="https://user-images.githubusercontent.com/2946766/70016801-89aa0c80-153e-11ea-9dbf-b7b8648fb260.png"> ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. - [x] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) * Note: some placeholders were moved to their own files, and so some raw strings will still exist - [ ] ~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios - [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~ ### For maintainers - [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ - [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ * [NP] Expose global config to the plugins (#51478) * [NP] Expose global config to the plugins * globalConfig in Plugin context: expose read-only methods only * SharedGlobalConfig rework + Moving pkg, fromRoot & path utils from legacy to NP * Updated API docs * Fix test references to the moved utils * Replace zip with combineLatest * Change tests to describe/it + remove "(deprecated)" from the test description * Moving path files to a folder + exposing the config path in the contract * deepFreeze the globalConfig in the pluginContext * Fix types in tests with new path.config * Move fromRoot and package_json utils to core/server/utils * Rename globalConfig to legacy.globalConfig$ * path.config renamed to path.configDir (not renaming path.data because it might be a breaking change) * Change configDir in mocker as well * Fix test after config renamed to configDir * Fix API docs conflicts * Rename the path properties when exposing them * path.configDir removed from the path config-schema * Remove path.configDir. It is already in env.configs * Add Migration documentation and examples * Fix 'kibana/server' imports in the MIGRATION docs * [Uptime] Added eui spacers around callout in expanded row (#51843) * added eui spacers around callout * updated snaps * [State Management] Move HashedItemStore to kibana_utils plugin. Make it stateless on memory level. (#52172) HashedItemStore was also moved to the kibana_utils plugin * [Uptime] Fix/issue 116 fix monitor list pageination arrows (#51912) * fix pagination arrows for smaller screens * updated arrow color * make pagingationa arrow sizing consisten * [Uptime] Expanded list update most recent error timestamp (#51935) * update most recent error timestamo * update snaps * update snap * [Cloud] move cloud plugin to np (#51789) * move cloud dir to plugins from legacy * first iteration moving to NP * config as observable * finish plugin * migrate xpack.cloud.apm.* * self code review * remove unused lodash get * use lodash get to get configs * export SerializedDatatable * eslint fix * eslint fix * fix setup type * fix monitoring tests * fix test mock function * resolve ELASTIC_SUPPORT_LINK in core * running check_core_api_changes script * review feedback * mock setHelpSupportUrl * update snapshots * Add example apm.dev.js config file to CONTRIBUTING.md (#52206) * [APM] Quick fix for ACM to ensure more than 10 items are displayed (#52262) * [APM] Quick fix for ACM to ensure more than 10 items are displayed * Fix snapshot * ci(jenkins): enable APM UI e2e tests (#48109) * Fix elasticsearch.ssl.alwaysPresentCertificate default (#52242) * Fixing the default for elasticsearch.ssl.alwaysPresentCertificate * Updating snapshot * [failed_tests_cli] update reports with links to github issues (#52048) * [failed_tests_cli] update reports with links to github issues * reorder test report hooks so that published Junit includes modified reports * force failures and enable dry-run mode for debugging * auto-switch to --dry-run when running in non-tracked branches/prs * add --skip-junit-update flag to skip mutating the reports * remove comma after URL to support auto-linking in Jenkins * Revert "force failures and enable dry-run mode for debugging" This reverts commit ac0c287a3fadc25f51f0d6280858a6d8380a91a7. * fix method call * extend TestResult to include relevence flag rather than wrapping * fix createFailureIssue() tests * make report messages more consistent, append when not dry-run * rename module * update snapshots to not contain valid xml * don't send authorization header if no token defined * merge with master modified fixtures * [precommit hook] resolve relative file paths from cwd (#52157) * [skip-test] [Uptime] MostRecentError component renders properly with empty data 1 https://github.com/elastic/kibana/issues/52288 Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co> * [xpack/ftr/common] follow service provider conventions (#52236) * [xpack/ftr/common] follow conventions, expose a config file In order to make xpack/test/common ftr config easier to work with, this updates it to follow the conventions established by most other configs of exposing all services from the services module as an object that can be easily merges into local services modules. * common config file and FtrProviderConfig are unused and unnecessary * turns out FtrProviderContext was used... * do not use stdoutsocket for console logs on Windows (#52208) * [Uptime] Fixed failing unit test because of timestamp in snapshot (#52302) * fixed test * update snaps * moved security to OSS/common (#52322) * [APM] Expose and reorganize platform config (#52325) Add `exposeToBrowser` to the config object's we're using in the server config. Update our shim and usage of these config variables to match. * [ML] DF Analytics: add description field to job creation and display in jobs list (#52217) * add job description input to creation form * wip add description to expanded row * add description to analytics list table * update jest test * description input to 2 rows and update types * update type * [ci/reportFailures] --dry-run is overloaded, split it up (#52314) * [ci/reportFailures] --dry-run is overloaded, split it up * force some failures to verify the fix * Revert "force some failures to verify the fix" This reverts commit cf2a58e1398a27c57c453f1486d8f59b09ab958f. * update readme to mention new flags * remove unnecessary commas * [Maps] Load mapbox-gl with static worker (#51675) This instructs mapbox-gl to load its workers using a static script. It removes the corresponding child-src CSP-rule from the policy. It retains the worker-src rule as it is required by the ace-editor in Dev-tools. * [Coordinate map] Sanitize WMS attribution (#52309) Sanitize the WMS-attribution input before displaying it in the leaflet-attribution control. * Upgrade EUI to v16.1.0 (#52088) * eui 16.1.0 * snapshot udpates * more snapshot updates * enqueueStateChange mock, resulting snapshot updates * [Uptime] Migrate Uptime server routing to new platform (#51125) * Move a REST endpoint and the GQL endpoint to NP routing. * Delete obsolete REST endpoint. * Update remaining REST routes to work with NP router. * Remove obsolete code, update some unit tests. * Simplify route creation. * Remove tests of API decommissioned API endpoint. * Rename domain check. * Make return shape of index pattern endpoint correspond to required NP resp body. * Move validate to appropriate level of route definition object for monitor details endpoint. * Update snapshot count route. * Fix broken lint rule. * Remove usages of Boom. * Fix license router creation. * Add tutorial for using RBAC with Spaces (#52197) * add spaces with RBAC tutorial * add tutorial * removed extra dir * revert * init tutorial * link tutorial * fix images dir * fixed links * editing * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * show path * added path to crate user * rearrange * remove image * specify admin role * replace we with you * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * remove excess words * remove excess words * spelling * spelling * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * Update docs/user/security/rbac_tutorial.asciidoc Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com> * resolve reviewer comments * review updates * address review comments * removed monitor cluster priv * reviewer comments * skip flaky suite (#52246) * [SIEM][Detection Engine] Adds signal data index per spaces through index naming conventions (#52237) ## Summary Changes the signals output index to be based on the user's space * Adds the ability to create a space based index through `POST /api/detection_engine/index` * Adds the existence API for the index through `HEAD /api/detection_engine/index` * Adds an index check during the creation of a rule, `POST api/detection_engine/rules` that will return a status of 400 with an error message if the index does not exist * Adds a new optional key in kibana.dev.yml of `xpack.siem.signalsIndex` for developers working together who need to segregate signals indexes. * Splits apart the ECS mappings and the signal mappings into separate files for easier maintenance. * Deprecates the defaultSignalsIndex (will remove it once the UI is updated) * Updates the README.md to remove the SIGNALS_INDEX environment variable * Updates the existing unit tests * Adds more unit tests unit tests For people writing the UI: --- How do I check for the existence of a signals index? See [scripts/signal_index_exists.sh](https://github.com/elastic/kibana/blob/28937ebe00bfc90129cf7e3ca1a04755c6029331/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/signal_index_exists.sh) ```sh HEAD /api/detection_engine/index ``` How do I create a new signals index if my user has correct privileges? See [scripts/post_signal_index.sh](https://github.com/elastic/kibana/blob/28937ebe00bfc90129cf7e3ca1a04755c6029331/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/post_signal_index.sh) ```sh POST /api/detection_engine/index ``` How do I delete _everything_ of all signal indexes, policies, and templates for a particular space? See [scripts/delete_signal_index.sh](https://github.com/elastic/kibana/blob/28937ebe00bfc90129cf7e3ca1a04755c6029331/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/delete_signal_index.sh) ```sh DELETE /api/detection_engine/index ``` FAQ for people testing --- What is the name of the index, policy, etc... per space? If you're using the default space the index, policies, etc... will be: ```sh .siem-signals-default ``` If you're using a custom space such as `test-space` they will be: ```sh .siem-signals-test-space ``` If you set your `xpack.siem.signalsIndex` in your `kibana.dev.yml` to something such as: ```yml xpack.siem.signalsIndex: .siem-signals-frank-hassanabad ``` And use the default space it will be: ```sh .siem-signals-frank-hassanabad-default ``` And for a custom space such as `test-space` they will be: ```sh .siem-signals-frank-hassanabad-test-space ``` What is the policy that is being set? See: [signals_policy.json](https://github.com/elastic/kibana/blob/28937ebe00bfc90129cf7e3ca1a04755c6029331/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/signals_policy.json) ```json { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_size": "10gb", "max_age": "7d" } } } } } } ``` What is the boot strap index that is being set look like? See: [create_bootstrap_index.ts](https://github.com/elastic/kibana/blob/28937ebe00bfc90129cf7e3ca1a04755c6029331/x-pack/legacy/plugins/siem/server/lib/detection_engine/index/create_bootstrap_index.ts) You should see this when running: ```sh ./get_signal_index.sh | less ``` ```json ".siem-signals-default-000001": { "aliases": { ".siem-signals-default": { "is_write_index": true } }, ``` What is the template that is being set look like? See: [get_signals_template.ts](https://github.com/elastic/kibana/blob/28937ebe00bfc90129cf7e3ca1a04755c6029331/x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/index/get_signals_template.ts) You should see this at the bottom when running: ```sh ./get_signal_index.sh ``` ```json "settings": { "index": { "lifecycle": { "name": ".siem-signals-default", "rollover_alias": ".siem-signals-default" }, "number_of_shards": "1", "provided_name": ".siem-signals-default-000001", "creation_date": "1575502837772", "number_of_replicas": "1", "uuid": "GB0h3AYRQD6AWl8OfNonJA", "version": { "created": "8000099" } } } ``` For more in-depth of testing of spaces using dev tools of Kibana --- Different testing scenarios involving having spaces set in the URL, vs not having spaces set. Also different testing scenarios involving having a developer based `xpack.siem.signalsIndex` being set vs not having one set and gettin the default of `.siem-signals` With a default space and kibana.dev.yml setting of: * xpack.siem.signalsIndex: .siem-signals-frank-hassanabad You can use dev tools to check the results after doing a `./post_signal_index.sh` ``` sh GET /_template/.siem-signals-frank-hassanabad-default GET /.siem-signals-frank-hassanabad-default-000001 GET /_ilm/policy/.siem-signals-frank-hassanabad-default GET /_alias/.siem-signals-frank-hassanabad-default ``` With a default space and no `kibana.dev.yml` setting, you can use dev tools to check the results after doing a `./post_signal_index.sh` ```sh GET /.siem-signals-default GET /_template/.siem-signals-default GET /.siem-signals-default-000001 GET /_ilm/policy/.siem-signals-default GET /_alias/.siem-signals-default ``` Setting a space through: ```sh export SPACE_URL=/s/test-space ``` With a default space and `kibana.dev.yml` setting using a user name such as mine: * xpack.siem.signalsIndex: .siem-signals-frank-hassanabad You can use dev tools to check the results after doing a `./post_signal_index.sh` ``` GET /.siem-signals-frank-hassanabad-test-space GET /_template/.siem-signals-frank-hassanabad-test-space GET /.siem-signals-frank-hassanabad-test-space-000001 GET /_ilm/policy/.siem-signals-frank-hassanabad-test-space GET /_alias/.siem-signals-frank-hassanabad-test-space ``` With a default space and no `kibana.dev.yml` setting, you can use dev tools to check the results after doing a `./post_signal_index.sh` ``` GET /.siem-signals-test-space GET /_template/.siem-signals-test-space GET /.siem-signals-default-test-space-000001 GET /_ilm/policy/.siem-signals-test-space GET /_alias/.siem-signals-test-space ``` ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ - [x] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process) * add codeowners for legacy server folder (#52158) * make defaultRoute accessible in NP Config (#52308) * defaultRoute was not provided to the NP * improve defaultRoute validation * add test that defaultRoute is read from config * update tests * [State Management] Move url state_hashing utils to kibana_utils (#52280) Part of #44151, Continuation of #51835, Just moves existing state related url utils to kibana_utils plugin Also fixes small regression introduced in #51835, When sharing hashed url directly it should show error toast instead of full page fatal error * [SIEM] Remove placeholder from pinned event tooltips (#52361) ## [SIEM] Remove placeholder from pinned event tooltips Similar to signals, pinned timeline events should be copied from source indexes, which are subject to ILM, to separate (space-aware) indexes (with different ILM), such that pinned events can be viewed in a timeline after the events have aged out of the original indexes. The backend APIs and UI patterns in development now for signals can likely be reused to implement the above, but until then, the placeholder tooltip text for unpinned / pinned events, which mentions persistence, should be removed from the SIEM beta. - [x] Changed the _unpinned_ event tooltip text from (sic) `This is event is NOT persisted with the timeline` to `Unpinned event` - [x] Changed the pinned event tooltip text from `This event is persisted with the timeline` to `Pinned event` https://github.com/elastic/siem-team/issues/482 * [ML] Fetch the latest job messages and enable sorting by time (#52388) * [ML] add sorting support * [ML] change fetch sort to desc for anomaly detection jobs * [ML] rename param * [ML] Functional tests for Additional settings in the Job wizards (#52269) * [ML] test custom urls in multi-metric wizard * [ML] calendars test * [ML] tests for job cloning * [ML] single metric * [ML] advanced job * [ML] population job * [ML] update snapshot * [ML] ensure calendar deleted and created * [ML] improve custom urls assertation * [ML] update snapshot * [ML] update snapshot, fix data-test-subject * [ML] remove redundant functions * [ML] add ensureAdditionalSettingsSectionOpen check * [ML] remove assignCalendar method * [ML] ensure model window disappears after adding a custom url * [ML] create calendar logging, remove unused deleteCalendar method, parameterized saveCustomUrl * [Maps] use style metadata to calculate symbolization bands (#51713) * [Maps] use style metadata to calculate symbolization bands * only update style meta when fields change * load join source style meta * use style meta data request to populate range * apply source filter to style meta request * fix heatmap * only use style meta range if field supports field meta * add fieldMetaOptions to style prperty descriptor and add migration script * add UI for setting fieldMetaOptions.isEnabled * clean up * review feedback * fix can_skip_fetch tests * review feedback * only show field meta popover for fields that support field meta * avoid duplicate fields re-fetching style meta * clean up problems when first creating grid source * update text for enabling field meta toggle * provide UI for setting sigma * allow users to include global time in style meta request * update SIEM saved objects * add less than and greater than symbols when styling by field stats * fix functional tests * review feedback * add support for date fields * review feedback * only show less then and greater then in legend when values will be outside of std range * unnest VectorStyle._getFieldRange * remove unused function * only show style isTimeAware switch when style fields use field meta * Add pre-response http interceptor (#52366) * add onPreResponse interceptor * expose registerPreResponse to plugins * address comments * regen docs * chore(NA): add resolution to bump serialize-javascript (#52336) * Preserve currently loaded Saved Query in Discover when page reloads (#52323) * Fix import * Add test that would have failed with previous bug * Infra server NP shim + config/routing API adoption (#45299) * Basic cleanup before refactoring for shim work * shim WIP * Removes the configuration adapter * WIP more stuff * WIP refactoring of shimming work * WIP continues * Logging UI now runs on top of new platform shim * WIP continues * Removes unused imports and variables * Basic infra NP server shim in place * Reimplemented graphql http error handling for infra NP server shim * Adds new platform infra plugin to handle NP config for legacy server shim * Basic cleanup before refactoring for shim work * shim WIP * Removes the configuration adapter * WIP more stuff * WIP refactoring of shimming work * WIP continues * Logging UI now runs on top of new platform shim * WIP continues * Removes unused imports and variables * Basic infra NP server shim in place * Reimplemented graphql http error handling for infra NP server shim * Adds new platform infra plugin to handle NP config for legacy server shim * Adds comment about duplicating full config for NP config * Use New Platform features plugin to registerFeature() * Re-arranging and relying on request context as uch as possible * Refactors KibanaRequest for RequestHandlerContext * fixes types for callWithRequest * Moves callWithRequest method override types directly into class to get them working, need to fix this when we understand it better * Fixes callWithRequest framework types * Removes a few NP_TODO comments * Fix broken imports * Ensure GraphQL resolvers are actually passed requestContext and not the raw request, and switch to the savedObjects client via requestContext * Remove the legacy traces of the savedObjects plugin * Fixes TSVB access with NP raw requests and requestContext * Remove unused getUiSettingsService (moved to requestContext) * Migrate to new Spaces plugin * Fix calculateMetricInterval after merged changes * Reinstate and migrate the infrastructure metadata route * Fix various type check errors * Amend InfraSources lib unit tests Mock the savedObjects client differently * Amend MetricsExplorer API response Renaming of variable inadvertently broke the response * Remove GraphQLI references from feature controls tests * Remove other GraphiQL references * Fix security / access issue * Add a framework level registerRoute method which always adds access tags by default * *Temp* disable test * Migrate the log rate validation endpoint to the new platform Fully migrates the [Logs UI] log rate setup index validation #50008 PR to New Platform routing etc * Amend types * Example of how to expose APM get indices method in NP * Fix calls to TSVB bug caused by object mutation This is a temp fix as the TSVB NP migration will supercede this * Converts getApmIndices function to accept saved object client, implements usage in infra * Fix APM setup_request tests * Fixes some unused references for linting * Migrate all work from #50730 to NP * Remove duplicate declaration files for rison_node and add a single source of truth at x-pack/typings/rison_node.d.ts for x-pack uses * Moved type file back into infra plugin to bypass strange break * Updates apm indices method signature per feedback from @elastic/apm-ui * State containers (#52384) * feat: 🎸 add state containers * docs: ✏️ add state container demos * docs: ✏️ refrech state container docs * chore: 🤖 install default comparator * chore: 🤖 remove old state container implementation * feat: 🎸 add selectors * chore: 🤖 move Ensure tyep to type utils * fix: 🐛 fix useSelector() types and demo CLI command * test: 💍 add tests for state container demos * feat: 🎸 add ReacursiveReadonly to kbn-utility-types * feat: 🎸 shallow freeze state when not in production * test: 💍 fix Jest tests * refactor: 💡 remove .state and use BehaviourSubject * update columns (#51892) * [ci/pipeline/reportFailures] when aborted, run with --no-github-update (#52355) * [SIEM] Adds support for specifying default filters to StatefulEventsViewer (#52413) ## Summary Finishes plumbing through the `defaultFilters` prop on the `StatefuleEventsViewer` component so that your view will always be constrained by a specified filter. Also adds an example of doing so to the current WIP `SignalsTable`. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. - [ ] ~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~ - [ ] ~Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~ - [ ] ~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~ - [ ] ~[Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~ - [ ] ~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~ ### For maintainers - [ ] ~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ - [ ] ~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~ * Add Endpoint plugin and Resolver embeddable (#51994) * Add functional tests for plugins to x-pack (so we can do a functional test of the Resolver embeddable) * Add Endpoint plugin * Add Resolver embeddable * Test that Resolver embeddable can be rendered * Update default path linked on Kibana sidebar to avoid basename warning in browser. (#52008) * Deprecate recompose part 1 (#50806) * Update most-frequent.asciidoc (#52241) Backporting https://github.com/elastic/kibana/pull/52240 * [APM] add service name to jvm (#50830) * [APM] add service name to jvm * removes isLoading * [skip-ci][Maps] Connecting to Elastic Maps Service from an internal network documentation (#48927) * [skip-ci][Maps] Connecting to Elastic Maps Service from an internal network documenation * use ascii docs list format * review feedback * [SIEM] [Detection Engine] Set Signal status open closed REST API (#52356) * adds route to update status on a signal / query result of signals. This also adds some example curl commands in scripts/signals and updates the README with general instructions on how to use the scripts to interact with the new endpoint. * utilizes 'mget' to allow multiple id's to be passed in. Updates relevant example curl script. * utilizes updateByQuery instead of querying then updating individual results * adds tests for set signals status route. Adds tests for set signals route schema. Updates schema to make status a required field. * fixes type check failures, fixes missing import due to renaming, fixes naming of mocked elasticsearch plugin after rebasing * removes anys from mocked es plugin, updates example comments in signals scripts * removes undefined | null from signal status definition, updates put_signal_doc with the get_signals_index script, adds getIndex to open_close_signals_route, updates tests to mock getIndex fn, updates README with instructions to run signals scripts from inside scripts directory. * removes null/undefined check for callWithRequest * sets signal status query to object * Re-introducing docs on image url (#52458) * skip flaky suite (#48721) * fix data-shared-items-container on visualise page (#52370) Addresses #52109 (comment) Should fix #51309, Regression was introduced in: #46910 * NP licensing plugin improvements (#51818) * add onPreResponse interceptor * use onPreResponse interceptor to add license sign * expose registerPreResponse to plugins * refresh for license update get the most fresh license * license plugin injects own header for signature: 'kbn-license-sig' * add integration tests for license type and license header * switch config to duration * don't run interceptor on anon paths. add tests * add functional tests for licensing plugin * regen docs * fix test in security due to updated mocks; * update snapshots accoring to new mock implementation * migrate license expired banner to NP * add readme for the licensing plugin * remove outdated import. licensing has separate functional tests * add tag for test to run on CI * regen docs * Update x-pack/plugins/licensing/README.md Co-Authored-By: Josh Dover <me@joshdover.com> * update tests * Upgrade extend to 3.0.2. (#51902) * Cleanup: Replace IndexPatterns types (#52188) * IndexPatternsContract * types * Expose FieldList on indexPattern contract (as its stateful) Rename FieldListInterface to IFieldList Remove more exports from ui/index_patterns Fix Maps import Move data services initialization to top level of the plugin * Clean up mocks * ts * setFieldFormats from Legacy plugin * Remove FIeldList export * Fix checking of error type in management * Fix security import of indexpatterns * fix jest tests * Fix jest tests * Update snapshots * Fix mocha tests * Expose SavedObjectsStart from AppMountContext (#52059) * [ML] Fix table factory usage. (#52486) Moves calling the table factory out of components to avoid instantiation on every render call. Note all of this is a workaround for our own typescript version of EuiInMemoryTable. Once the original table has been migrated to TS, we should be able to remove the custom code. * fix flaky test (#52395) * License Management to New Platform (#51886) * License Management public -> NP * - Server to NP - Slight update to filepicker style (center it) * Fix snapshots and types * Server-side: separate new and legacy dependencies [skip ci] * Fix license upload route after refactor * Client side: separate new from legacy dependencies * xpackInfo -> xPackInfo * Fix types [skip ci] * Remove kbnUrl, autoLogout. Add history and update paths. * Update upload license test * Remove use of legacy chrome, remove use of k7breadcrumbs, replace some common strings with variable and use NP i18n * Move apply filters action to NP (#52156) * move action * attach action in start * Remove uiActions from core_plugins/data * Don't export apply filters popup * import DataPublicPlugin after all other deps have loaded * lint * Remove unused import * Graph: Stabilize functional test (#52417) * Upgrade EUI to v17.0.0 (#52342) * eui to 17.0.0 * Fix sass imports and mixin usages * Fix kbn ui-framework * snapshot updates * switch text * switch functional update * test subject * Fix timing issue with synchronizing the Kibana privileges to ES (#52214) * adding in updated links for feedback and ask (#52516) * ui/management -> new platform (#45747) Created management plugin under core_plugins. Filled the plugin with 2 services: IndexPatternManagementService and SavedObjectsManagementService. Removed related codes in ui/management and changed the paths. * [Lens] Show keyword fields for pre-7.3 index patterns (#52410) * [Canvas] Add simple visual test for fullscreen (#51234) * Add simple visual test for fullscreen workpads in Canvas * Adding canvas to the config * Adding sample data visual tests * Refactor to use new helper method * Reverting sample data visual test for now * Forgot to add the awaits * Upgrade typescript-eslint to 2.10.0 (#52528) * Flag nested fields as non-aggregatable (#51774) * Flag nested fields as non-aggregatable * Update tests * Support nested fields in existing filter types (#49537) * Add automatic support for nested fields in existing filter types * Index pattern could be undefined * add test for handleNestedFilter function * remove console.log * add tests for all "getFilterField" functions * update migrateFilters to work on full filter objects so that it doesn't have to wo…

watson force-pushed the apm branch from 1d9671e to 462ed94 Compare August 19, 2019 21:17