create kbn-legacy-logging package #77678

pgayvallet · 2020-09-16T20:58:29Z

Summary

Related to #76003 and #56205

Create the @kbn/legacy-logging package and move legacy logging implementation to it.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

…gging-package

elasticmachine · 2020-09-17T14:19:55Z

Pinging @elastic/kibana-platform (Team:Platform)

lukeelmers

App Arch updates LGTM

…gging-package

docs/development/plugins/data/public/kibana-plugin-plugins-data-public.querystringinput.md

mshustov · 2020-09-23T14:56:50Z

packages/kbn-legacy-logging/src/schema.ts

+    loggers: HANDLED_IN_KIBANA_PLATFORM,
+    root: HANDLED_IN_KIBANA_PLATFORM,
+
+    silent: Joi.boolean().default(false),


optional: is it possible to migrate this to config-schema to remove another dependency on joi?

I thought it was not possible, TIL: we actually got siblingRef in config-schema.

However the joi schema is still used by the legacy config: https://github.com/elastic/kibana/pull/77678/files#diff-30708bfadd0e62135e289c55c5d900be. Not sure we can convert it back to joi to register it to the legacy config if we migrate it. We could use the protected internalSchema property though I guess, but not sure this is a proper solution

kibana/packages/kbn-config-schema/src/types/type.ts

Lines 38 to 42 in c4ddd00

/**

* Internal "schema" backed by Joi.

* @type {Schema}

*/

protected readonly internalSchema: AnySchema;

From what I can see: legacy platform doesn't read config values, but pass them to @kbn/legacy-logging so it might be okay to move config validation to those @kbn/legacy-logging methods that accept config as a function parameter. The main downside of this approach is that the logging config will be validated at a different time than the other config parts.

Let's keep it that way until #81483 is resolved imho.

packages/kbn-legacy-logging/src/log_format.ts

mshustov · 2020-09-23T15:27:28Z

packages/kbn-legacy-logging/src/log_format_json.test.ts

-
-import KbnLoggerJsonFormat from './log_format_json';
+import { attachMetaData } from './metadata';
+import { createListStream, createPromiseFromStreams } from './test_utils';


another place to fix when we move stream utils to @kbn/std package

packages/kbn-legacy-logging/tsconfig.json

src/legacy/server/kbn_server.js

mshustov · 2020-09-23T15:38:21Z

packages/kbn-legacy-logging/package.json

+{
+  "name": "@kbn/legacy-logging",
+  "version": "1.0.0",
+  "private": true,


Could you add a Readme that this package is a temporary workaround until we migrate logging to the KP?

I was more planning to keep it there until we totally get rid of the legacy logging system in 8.0 tbh. It will be way easier to delete it with all the code being centralized in this package. Was that what you meant?

edit: oh, I guess you mean the server event logging part. But there is the same issue. We have to keep support of the legacy logging format, meaning that the module won't go away before 8.0.

mshustov · 2020-09-23T15:41:00Z

packages/kbn-legacy-logging/src/legacy_logging_server.ts


-    setupLogging(this, Config.withDefaultSchema(config));
+    setupLogging((this as unknown) as Server, loggingConfig, 2147483647);


So we expect LegacyLoggingServer won't be called when the package is imported from CLI scripts? How does the final dependency tree look like?

The only import of this package in core (soon in kbn/config instead) is from the legacy appender

kibana/src/core/server/legacy/logging/appenders/legacy_appender.ts

Lines 48 to 50 in ab92bbb

constructor(legacyLoggingConfig: Readonly<LegacyVars>) {

this.loggingServer = new LegacyLoggingServer(legacyLoggingConfig);

}

The idea was to change the file to use a dynamic require in the constructor to only load the module when needed. The cli / packages that will use the config service would default to KP logging (probably just defaulting to console appender), that way, the legacy logging (and hapi stuff) is not imported.

Note that the main goal to not load the legacy logging system / hapi when importing the config service was for the apm script instrumentation, which is no longer concerned by the problem, as it will have its own loader to ensure we are not importing any module we want to instrument at all (#77855)

…gging-package

mshustov · 2020-11-20T10:43:57Z

packages/kbn-legacy-logging/src/log_events.ts

+  data: string | Record<string, any>;
+}
+
+export type AnyEvent =


mshustov · 2020-11-20T10:47:38Z

packages/kbn-legacy-logging/src/log_interceptor.ts

    return downgradeIfErrorType('HPE_INVALID_METHOD', event);
  }

-  downgradeIfHTTPWhenHTTPS(event) {
+  downgradeIfHTTPWhenHTTPS(event: any) {


any --> unknown ?

even better. any -> AnyEvent a60d262

mshustov · 2020-11-20T10:53:57Z

packages/kbn-legacy-logging/src/test_utils/streams.ts

+ *  @param  {Array<any>} items - the list of items to provide
+ *  @return {Readable}
+ */
+export function createListStream<T = any>(items: T | T[] = []) {


we haven't move them to @kbn/std yet 😞 let's do in a follow-up

opened #84033

mshustov · 2020-11-20T10:55:00Z

packages/kbn-legacy-logging/tsconfig.json

+    "declarationMap": true,
+    "types": ["jest", "node"]
+  },
+  "include": ["./src/**/*.ts"]


optional: src/**/*

…gging-package

kibanamachine · 2020-11-22T17:01:55Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 389dfe9

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`enterpriseSearch`	731.7KB	731.8KB	+57.0B

Distributable file count

id	before	after	diff
`default`	42975	43023	+48
`oss`	22502	22550	+48

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`upgradeAssistant`	60.4KB	60.5KB	+57.0B

History

💚 Build #89226 succeeded dbbe2e5
💚 Build #88890 succeeded 0dd3b23
💚 Build #88870 succeeded 294a05a
💔 Build #88873 failed f2766e5
💔 Build #88868 failed 5f0471d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* create kbn-legacy-logging package and start to move things * fix rotator tests * fix logging system test mocks * move logging format to the package * move logging setup to package * adapt legacy logging server * remove usage of legacy config in the legacy logging server * move legacy logging server to package * remove `??` syntax from package * update generated doc * fix a few things due to month old merge * remove typings from project * move reconfigureLogging to package * add basic README file * update generated doc * remove old typings * add typing for legacy logging events * remove `??` from packages * fix / improve event types usages * remove suffix from tsconfig # Conflicts: # src/legacy/server/config/schema.js

* create kbn-legacy-logging package (#77678) * create kbn-legacy-logging package and start to move things * fix rotator tests * fix logging system test mocks * move logging format to the package * move logging setup to package * adapt legacy logging server * remove usage of legacy config in the legacy logging server * move legacy logging server to package * remove `??` syntax from package * update generated doc * fix a few things due to month old merge * remove typings from project * move reconfigureLogging to package * add basic README file * update generated doc * remove old typings * add typing for legacy logging events * remove `??` from packages * fix / improve event types usages * remove suffix from tsconfig # Conflicts: # src/legacy/server/config/schema.js * fix for 7.x branch

* master: (67 commits) [Observability] Load hasData call asynchronously (elastic#80644) Implement AnonymousAuthenticationProvider. (elastic#79985) Deprecate `visualization:colorMapping` advanced setting (elastic#83372) [TSVB] [Rollup] Table tab not working with rollup indexes (elastic#83635) Revert "[Search] Search batching using bfetch (elastic#83418)" (elastic#84037) skip flaky suite (elastic#83772) skip flaky suite (elastic#69849) create kbn-legacy-logging package (elastic#77678) [Search] Search batching using bfetch (elastic#83418) [Security Solution] Refactor Timeline flyout to take a full page (elastic#82033) Drop use of console-stamp (elastic#83922) skip flaky suite (elastic#84011 , elastic#84012) Fixed usage of `isReady` for usage collection of alerts and actions (elastic#83760) [maps] support URL drilldowns (elastic#83732) Revert "Added default dedupKey value as an {{alertInstanceId}} to provide grouping functionality for PagerDuty incidents. (elastic#83226)" [code coverage] Update jest config to collect more data (elastic#83804) Added default dedupKey value as an {{alertInstanceId}} to provide grouping functionality for PagerDuty incidents. (elastic#83226) [Security Solution] Give notice when endpoint policy is out of date (elastic#83469) [Security Solution] Sync url state on any changes to query string (elastic#83314) [CI] Initial TeamCity implementation (elastic#81043) ...

pgayvallet added 11 commits September 16, 2020 22:57

create kbn-legacy-logging package and start to move things

3ad4c93

Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…

73122e5

…gging-package

fix rotator tests

87ca167

fix logging system test mocks

f6dd86c

move logging format to the package

ba67eb7

move logging setup to package

e2e1615

adapt legacy logging server

bd0c68d

remove usage of legacy config in the legacy logging server

76f4f27

move legacy logging server to package

e54b7d9

remove ?? syntax from package

361deda

update generated doc

5d7b70a

pgayvallet marked this pull request as ready for review September 17, 2020 14:19

pgayvallet requested review from a team as code owners September 17, 2020 14:19

pgayvallet added Feature:Legacy Removal Issues related to removing legacy Kibana release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.10.0 labels Sep 17, 2020

lukeelmers approved these changes Sep 21, 2020

View reviewed changes

Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…

6bd16cf

…gging-package

mshustov reviewed Sep 23, 2020

View reviewed changes

LeeDr added v7.10.1 and removed v7.10.0 labels Nov 12, 2020

pgayvallet added v7.11.0 and removed v7.10.1 labels Nov 19, 2020

pgayvallet added 3 commits November 19, 2020 10:19

Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…

8ddc262

…gging-package

fix a few things due to month old merge

f6c67de

remove typings from project

8382a09

pgayvallet added 6 commits November 19, 2020 10:41

move reconfigureLogging to package

56e3ea1

add basic README file

4b8a661

update generated doc

5f0471d

remove old typings

294a05a

add typing for legacy logging events

f2766e5

remove ?? from packages

0dd3b23

mshustov mentioned this pull request Nov 20, 2020

[DX] Bump TS version to v4.1 #83397

Merged

mshustov approved these changes Nov 20, 2020

View reviewed changes

pgayvallet added 4 commits November 20, 2020 13:26

Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…

5a37913

…gging-package

fix / improve event types usages

a60d262

remove suffix from tsconfig

dbbe2e5

Merge remote-tracking branch 'upstream/master' into kbn-xxx-legacy-lo…

389dfe9

…gging-package

pgayvallet merged commit e176def into elastic:master Nov 22, 2020

pgayvallet mentioned this pull request Nov 22, 2020

[7.x] create kbn-legacy-logging package (#77678) #84034

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

create kbn-legacy-logging package #77678

create kbn-legacy-logging package #77678

pgayvallet commented Sep 16, 2020 •

edited

Loading

elasticmachine commented Sep 17, 2020

lukeelmers left a comment

mshustov Sep 23, 2020

pgayvallet Sep 23, 2020 •

edited

Loading

mshustov Sep 24, 2020

pgayvallet Nov 19, 2020

mshustov Sep 23, 2020

mshustov Sep 23, 2020

pgayvallet Sep 23, 2020 •

edited

Loading

mshustov Sep 23, 2020

pgayvallet Sep 23, 2020

mshustov Nov 20, 2020

mshustov Nov 20, 2020

pgayvallet Nov 20, 2020 •

edited

Loading

mshustov Nov 20, 2020

mshustov Nov 22, 2020

mshustov Nov 20, 2020

kibanamachine commented Nov 22, 2020

	/**
	* Internal "schema" backed by Joi.
	* @type {Schema}
	*/
	protected readonly internalSchema: AnySchema;


		setupLogging(this, Config.withDefaultSchema(config));
		setupLogging((this as unknown) as Server, loggingConfig, 2147483647);

	constructor(legacyLoggingConfig: Readonly<LegacyVars>) {
	this.loggingServer = new LegacyLoggingServer(legacyLoggingConfig);
	}

create kbn-legacy-logging package #77678

create kbn-legacy-logging package #77678

Conversation

pgayvallet commented Sep 16, 2020 • edited Loading

Summary

Checklist

elasticmachine commented Sep 17, 2020

lukeelmers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgayvallet Sep 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgayvallet Sep 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgayvallet Nov 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Nov 22, 2020

💚 Build Succeeded

Metrics [docs]

Async chunks

Distributable file count

Page load bundle

History

pgayvallet commented Sep 16, 2020 •

edited

Loading

pgayvallet Sep 23, 2020 •

edited

Loading

pgayvallet Sep 23, 2020 •

edited

Loading

pgayvallet Nov 20, 2020 •

edited

Loading