[BUG][2.11] Home fails to load if OS Security plugin installed and OSD Security plugin _not_ installed

[pjfitzgibbons]

Describe the bug

A clear and concise description of what the bug is.
When OS Security plugin is installed, and OSD Security plugin is not installed,
OSD homepage (and any page) fails to load
Browser displays :
{"statusCode":401,"error":"Unauthorized","message":"Response Error"}

OSD Server log includes :

'SavedObjectsClient/notAuthorized'
}
server   error  [18:29:34.929]  Error: Internal Server Error

(full error in comment below)

To Reproduce
Steps to reproduce the behavior:

1. Run OS with Security plugin enabled (for instance, from .tar.gz release or docker image)
2. Run OSD without Security plugin installed (for instance, from git repo pull)
3. Run OS by normal means
4. Run OSD with yarn start
5. See error

Expected behavior
Homepage and any page should load without error

OpenSearch Version
OS 2.11

Dashboards Version
OSD 2.11

Plugins

Please list all plugins currently enabled.

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Browser and version [e.g. 22]

Additional context

Add any other context about the problem here.

[pjfitzgibbons]

Full error on OSD Server log :

server    log   [18:29:35.073] [error][data][opensearch] [ResponseError]: Response Error
server    log   [18:29:35.082] [error][http] ResponseError: Response Error
    at onBody ($DEV_ENV/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)
    at IncomingMessage.onEnd ($DEV_ENV/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)
    at IncomingMessage.emit (node:events:525:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  meta: {
    body: '',
    statusCode: 401,
    headers: {
      'x-opaque-id': '37c4f987-8f92-4dd4-b4dd-d492907935b2',
      'www-authenticate': 'Basic realm="OpenSearch Security"',
      'content-type': 'text/plain; charset=UTF-8',
      'content-length': '0'
    },
    meta: {
      context: null,
      request: [Object],
      name: 'opensearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    }
  },
  isBoom: true,
  isServer: false,
  data: null,
  output: {
    statusCode: 401,
    payload: {
      statusCode: 401,
      error: 'Unauthorized',
      message: 'Response Error'
    },
    headers: {}
  },
  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/notAuthorized'
}
server   error  [18:29:34.929]  Error: Internal Server Error
    at HapiResponseAdapter.toInternalError ($DEV_ENV/src/core/server/http/router/response_adapter.ts:71:19)
    at Router.handle ($DEV_ENV/src/core/server/http/router/router.ts:304:34)
    

[cwperks]

@pjfitzgibbons This is as intended unless you disable security or run in SSL only mode. Saved objects are stored in a .kibana* index and to search that index w/ the security plugin installed requires credentials. Without the security-dashboards-plugin installed its sending requests without credentials and getting an unauthorized response which is expected.

See demo config in the developer guide for security-dashboards-plugin for setting up the security-dashboards-plugin here.

[pjfitzgibbons]

OSD config/opensearch_dashboards.yml was :

opensearch.hosts: ['https://localhost:9200/']
opensearch.username: 'admin'
opensearch.password: 'admin'
opensearch.ssl.verificationMode: none

even with this config, if you want to replicate this error, verify the config is working with OS/OSD security plugins installed, then remove the OSD security plugin.
This "used to" work in 2.10 and below.

[pjfitzgibbons]

Also this change in OS Security plugin 2.11 may have effected this
opensearch-project/security#2753

This behavour may indeed be "as intended". My fear is that this change is "new" for 2.11, is a significant behavioral change for Opensearch 2.11, and the community has not been widely informed of the change.

[cwperks]

@pjfitzgibbons I figured out what fixes the issue, but not sure I quite fully understand yet why the browser isn't challenging to the basic auth creds.

The difference between 2.11.0 and 2.10.0 is that the body of challenge response contained Unauthorized in 2.10.0 and contains an empty body in 2.11.0.

Line where the challenge response is created in 2.11.0 (note the empty string param at the end): https://github.com/opensearch-project/security/blob/2.11/src/main/java/org/opensearch/security/http/HTTPBasicAuthenticator.java#L68-L73

Same line in 2.10.0: https://github.com/opensearch-project/security/blob/2.10/src/main/java/org/opensearch/security/http/HTTPBasicAuthenticator.java#L69

The challenge works when going to OpenSearch directly. Its only an issue with OSD.

Looks like hapi is having issues with the empty body:

server   error  [23:24:27.525]  Error: Internal Server Error
    at HapiResponseAdapter.toInternalError (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/src/core/server/http/router/response_adapter.ts:71:19)
    at Router.handle (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/src/core/server/http/router/router.ts:304:34)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at handler (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/src/core/server/http/router/router.ts:241:11)
    at exports.Manager.execute (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/request.js:371:32)
    at Request._execute (/Users/cwperx/Projects/opensearch/OpenSearch-Dashboards/node_modules/@hapi/hapi/lib/request.js:281:9)