Merge branch 'master' into inventory-alerts-ux-fixes

docs/apm/metrics.asciidoc

@@ -11,8 +11,12 @@ For example, you might be able to correlate a high number of errors with a long
 [role="screenshot"]
 image::apm/images/apm-metrics.png[Example view of the Metrics overview in APM app in Kibana]
 
-If you're using the Java Agent, the metrics view focuses on JVMs.
-A detailed view of metrics per JVM makes it much easier to analyze the provided metrics:
+If you're using the Java Agent, you can view metrics for each JVM.
+
+[role="screenshot"]
+image::apm/images/jvm-metrics-overview.png[Example view of the Metrics overview for the Java Agent]
+
+Breaking down metrics by JVM makes it much easier to analyze the provided metrics:
 CPU usage, memory usage, heap or non-heap memory,
 thread count, garbage collection rate, and garbage collection time spent per minute.
 

docs/user/security/api-keys/index.asciidoc

@@ -3,18 +3,18 @@
 === API Keys
 
 
-API keys enable you to create secondary credentials so that you can send 
-requests on behalf of the user. Secondary credentials have 
-the same or lower access rights. 
+API keys enable you to create secondary credentials so that you can send
+requests on behalf of the user. Secondary credentials have
+the same or lower access rights.
 
 For example, if you extract data from an {es} cluster on a daily
-basis, you might create an API key tied to your credentials, 
-configure it with minimum access, 
+basis, you might create an API key tied to your credentials,
+configure it with minimum access,
 and then put the API credentials into a cron job.
-Or, you might create API keys to automate ingestion of new data from 
-remote sources, without a live user interaction. 
+Or, you might create API keys to automate ingestion of new data from
+remote sources, without a live user interaction.
 
-You can create API keys from the {kib} Console. To view and invalidate 
+You can create API keys from the {kib} Console. To view and invalidate
 API keys, use *Management > Security > API Keys*.
 
 [role="screenshot"]
@@ -24,63 +24,80 @@ image:user/security/api-keys/images/api-keys.png["API Keys UI"]
 [[api-keys-service]]
 === {es} API key service
 
-The {es} API key service is automatically enabled when you configure 
-{ref}/configuring-tls.html#tls-http[TLS on the HTTP interface]. 
+The {es} API key service is automatically enabled when you configure
+{ref}/configuring-tls.html#tls-http[TLS on the HTTP interface].
 This ensures that clients are unable to send API keys in clear-text.
 
-When HTTPS connections are not enabled between {kib} and {es}, 
+When HTTPS connections are not enabled between {kib} and {es},
 you cannot create or manage API keys, and you get an error message.
-For more information, see the 
-{ref}/security-api-create-api-key.html[{es} API key documentation], 
+For more information, see the
+{ref}/security-api-create-api-key.html[{es} API key documentation],
 or contact your system administrator.
 
 [float]
 [[api-keys-security-privileges]]
 === Security privileges
 
-You must have the `manage_security`, `manage_api_key`, or the `manage_own_api_key` 
-cluster privileges to use API keys in {kib}. You can manage roles in 
-*Management > Security > Roles*, or use the <<role-management-api, {kib} Role Management API>>. 
+You must have the `manage_security`, `manage_api_key`, or the `manage_own_api_key`
+cluster privileges to use API keys in {kib}. You can manage roles in
+*Management > Security > Roles*, or use the <<role-management-api, {kib} Role Management API>>.
 
 
 [float]
 [[create-api-key]]
 === Create an API key
-You can {ref}/security-api-create-api-key.html[create an API key] from 
-the Kibana Console. For example:
+You can {ref}/security-api-create-api-key.html[create an API key] from
+the {kib} Console. This example shows how to create an API key
+to authenticate to a <<api, Kibana API>>.
 
 [source,js]
 POST /_security/api_key
 {
- "name": "my_api_key",
- "expiration": "1d"
+ "name": "kibana_api_key",
 }
 
-This creates an API key with the name `my_api_key` that 
-expires after one day. API key names must be globally unique. 
-An expiration date is optional and follows {ref}/common-options.html#time-units[{es} time unit format]. 
+This creates an API key with the
+name `kibana_api_key`. API key
+names must be globally unique.
+An expiration date is optional and follows
+{ref}/common-options.html#time-units[{es} time unit format].
 When an expiration is not provided, the API key does not expire.
 
+The response should look something like this:
+
+[source,js]
+{
+ "id" : "XFcbCnIBnbwqt2o79G4q",
+ "name" : "kibana_api_key",
+ "api_key" : "FD6P5UA4QCWlZZQhYF3YGw"
+}
+
+Now, you can use the API key to request {kib} roles. You will need
+to base64-encode the `id` and `api_key` provided in the response
+and add it to your request as an authorization header. For example:
+
+[source,js]
+curl --location --request GET 'http://localhost:5601/api/security/role' \
+--header 'Content-Type: application/json;charset=UTF-8' \
+--header 'kbn-xsrf: true' \
+--header 'Authorization: ApiKey aVZlLUMzSUJuYndxdDJvN0k1bU46aGxlYUpNS2lTa2FKeVZua1FnY1VEdw==' \
+
 [float]
 [[view-api-keys]]
 === View and invalidate API keys
-The *API Keys* UI lists your API keys, including the name, date created, 
+The *API Keys* feature in Kibana lists your API keys, including the name, date created,
 and expiration date. If an API key expires, its status changes from `Active` to `Expired`.
 
-If you have `manage_security` or `manage_api_key` permissions, 
-you can view the API keys of all users, and see which API key was 
+If you have `manage_security` or `manage_api_key` permissions,
+you can view the API keys of all users, and see which API key was
 created by which user in which realm.
 If you have only the `manage_own_api_key` permission, you see only a list of your own keys.
 
-You can invalidate API keys individually or in bulk. 
+You can invalidate API keys individually or in bulk.
 Invalidated keys are deleted in batch after seven days.
 
 [role="screenshot"]
 image:user/security/api-keys/images/api-key-invalidate.png["API Keys invalidate"]
 
-You cannot modify an API key. If you need additional privileges, 
+You cannot modify an API key. If you need additional privileges,
 you must create a new key with the desired configuration and invalidate the old key.
-
-
-
-

packages/kbn-es/src/utils/native_realm.js

@@ -76,10 +76,6 @@ exports.NativeRealm = class NativeRealm {
  }
 
  const reservedUsers = await this.getReservedUsers();
- if (!reservedUsers || reservedUsers.length < 1) {
- throw new Error('no reserved users found, unable to set native realm passwords');
- }
-
  await Promise.all(
  reservedUsers.map(async user => {
  await this.setPassword(user, options[`password.${user}`]);
@@ -88,16 +84,18 @@ exports.NativeRealm = class NativeRealm {
  }
 
  async getReservedUsers() {
- const users = await this._autoRetry(async () => {
- return await this._client.security.getUser();
- });
+ return await this._autoRetry(async () => {
+ const resp = await this._client.security.getUser();
+ const usernames = Object.keys(resp.body).filter(
+ user => resp.body[user].metadata._reserved === true
+ );
 
- return Object.keys(users.body).reduce((acc, user) => {
- if (users.body[user].metadata._reserved === true) {
- acc.push(user);
+ if (!usernames?.length) {
+ throw new Error('no reserved users found, unable to set native realm passwords');
  }
- return acc;
- }, []);
+
+ return usernames;
+ });
  }
 
  async isSecurityEnabled() {
@@ -125,10 +123,9 @@ exports.NativeRealm = class NativeRealm {
  throw error;
  }
 
- this._log.warning(
- 'assuming [elastic] user not available yet, waiting 1.5 seconds and trying again'
- );
- await new Promise(resolve => setTimeout(resolve, 1500));
+ const sec = 1.5 * attempt;
+ this._log.warning(`assuming ES isn't initialized completely, trying again in ${sec} seconds`);
+ await new Promise(resolve => setTimeout(resolve, sec * 1000));
  return await this._autoRetry(fn, attempt + 1);
  }
  }

src/plugins/bfetch/server/index.ts

@@ -21,6 +21,7 @@ import { PluginInitializerContext } from '../../../core/server';
 import { BfetchServerPlugin } from './plugin';
 
 export { BfetchServerSetup, BfetchServerStart, BatchProcessingRouteParams } from './plugin';
+export { StreamingRequestHandler } from './types';
 
 export function plugin(initializerContext: PluginInitializerContext) {
  return new BfetchServerPlugin(initializerContext);

src/plugins/bfetch/server/mocks.ts

@@ -28,6 +28,7 @@ const createSetupContract = (): Setup => {
  const setupContract: Setup = {
  addBatchProcessingRoute: jest.fn(),
  addStreamingResponseRoute: jest.fn(),
+ createStreamingRequestHandler: jest.fn(),
  };
  return setupContract;
 };

src/plugins/bfetch/server/plugin.ts

@@ -24,6 +24,8 @@ import {
  Plugin,
  Logger,
  KibanaRequest,
+ RouteMethod,
+ RequestHandler,
 } from 'src/core/server';
 import { schema } from '@kbn/config-schema';
 import { Subject } from 'rxjs';
@@ -35,6 +37,7 @@ import {
  removeLeadingSlash,
  normalizeError,
 } from '../common';
+import { StreamingRequestHandler } from './types';
 import { createNDJSONStream } from './streaming';
 
 // eslint-disable-next-line
@@ -47,6 +50,7 @@ export interface BatchProcessingRouteParams<BatchItemData, BatchItemResult> {
  onBatchItem: (data: BatchItemData) => Promise<BatchItemResult>;
 }
 
+/** @public */
 export interface BfetchServerSetup {
  addBatchProcessingRoute: <BatchItemData extends object, BatchItemResult extends object>(
  path: string,
@@ -56,11 +60,48 @@ export interface BfetchServerSetup {
  path: string,
  params: (request: KibanaRequest) => StreamingResponseHandler<Payload, Response>
  ) => void;
+ /**
+ * Create a streaming request handler to be able to use an Observable to return chunked content to the client.
+ * This is meant to be used with the `fetchStreaming` API of the `bfetch` client-side plugin.
+ *
+ * @example
+ * ```ts
+ * setup({ http }: CoreStart, { bfetch }: SetupDeps) {
+ * const router = http.createRouter();
+ * router.post(
+ * {
+ * path: '/api/my-plugin/stream-endpoint,
+ * validate: {
+ * body: schema.object({
+ * term: schema.string(),
+ * }),
+ * }
+ * },
+ * bfetch.createStreamingResponseHandler(async (ctx, req) => {
+ * const { term } = req.body;
+ * const results$ = await myApi.getResults$(term);
+ * return results$;
+ * })
+ * )}
+ *
+ * ```
+ *
+ * @param streamHandler
+ */
+ createStreamingRequestHandler: <Response, P, Q, B, Method extends RouteMethod = any>(
+ streamHandler: StreamingRequestHandler<Response, P, Q, B, Method>
+ ) => RequestHandler<P, Q, B, Method>;
 }
 
 // eslint-disable-next-line
 export interface BfetchServerStart {}
 
+const streamingHeaders = {
+ 'Content-Type': 'application/x-ndjson',
+ Connection: 'keep-alive',
+ 'Transfer-Encoding': 'chunked',
+};
+
 export class BfetchServerPlugin
  implements
  Plugin<
@@ -76,10 +117,12 @@ export class BfetchServerPlugin
  const router = core.http.createRouter();
  const addStreamingResponseRoute = this.addStreamingResponseRoute({ router, logger });
  const addBatchProcessingRoute = this.addBatchProcessingRoute(addStreamingResponseRoute);
+ const createStreamingRequestHandler = this.createStreamingRequestHandler({ logger });
 
  return {
  addBatchProcessingRoute,
  addStreamingResponseRoute,
+ createStreamingRequestHandler,
  };
  }
 
@@ -106,19 +149,30 @@ export class BfetchServerPlugin
  async (context, request, response) => {
  const handlerInstance = handler(request);
  const data = request.body;
- const headers = {
- 'Content-Type': 'application/x-ndjson',
- Connection: 'keep-alive',
- 'Transfer-Encoding': 'chunked',
- };
  return response.ok({
- headers,
- body: createNDJSONStream(data, handlerInstance, logger),
+ headers: streamingHeaders,
+ body: createNDJSONStream(handlerInstance.getResponseStream(data), logger),
  });
  }
  );
  };
 
+ private createStreamingRequestHandler = ({
+ logger,
+ }: {
+ logger: Logger;
+ }): BfetchServerSetup['createStreamingRequestHandler'] => streamHandler => async (
+ context,
+ request,
+ response
+ ) => {
+ const response$ = await streamHandler(context, request);
+ return response.ok({
+ headers: streamingHeaders,
+ body: createNDJSONStream(response$, logger),
+ });
+ };
+
  private addBatchProcessingRoute = (
  addStreamingResponseRoute: BfetchServerSetup['addStreamingResponseRoute']
  ): BfetchServerSetup['addBatchProcessingRoute'] => <

src/plugins/bfetch/server/streaming/create_ndjson_stream.ts

@@ -17,19 +17,17 @@
  * under the License.
  */
 
+import { Observable } from 'rxjs';
 import { Logger } from 'src/core/server';
 import { Stream, PassThrough } from 'stream';
-import { StreamingResponseHandler } from '../../common/types';
 
 const delimiter = '\n';
 
-export const createNDJSONStream = <Payload, Response>(
- payload: Payload,
- handler: StreamingResponseHandler<Payload, Response>,
+export const createNDJSONStream = <Response>(
+ results: Observable<Response>,
  logger: Logger
 ): Stream => {
  const stream = new PassThrough();
- const results = handler.getResponseStream(payload);
 
  results.subscribe({
  next: (message: Response) => {