[BUG] github action service fails with "Error: Could not find or load main class Cannot"

[mrkamel]

Describe the bug

In our github action which start an opensearch instance we often (every 20-30th run) see the opensearch container fail with 2.11, 2.12 and also 2.13. Excerpt from our action config:

    services:
      opensearch:
        image: opensearchproject/opensearch:2.13.0
        env:
          discovery.type: single-node
          plugins.security.disabled: "true"
          ES_JAVA_OPTS: "-Xms512m -Xmx512m"
          OPENSEARCH_INITIAL_ADMIN_PASSWORD: "..."
        options: >-
          --health-cmd "curl http://localhost:9200/_cluster/health"
          --health-interval 10s 
          --health-timeout 5s
          --health-retries 10

Pipeline output:

Service container opensearch failed.
  /usr/bin/docker logs --details 963fd41d9172b63f2402f48bd758b9d5b31bba6b30c4201e88c89496fc73b27c
   Enabling OpenSearch Security Plugin
   WARNING: java.io.tmpdir directory does not exist
   Error: Could not find or load main class Cannot
   Caused by: java.lang.ClassNotFoundException: Cannot
   Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin 
   OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user. 
   Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string. 
   If a password is not provided, the setup will quit. 
    For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
   ### OpenSearch Security Demo Installer
   ### ** Warning: Do not use on production or public reachable systems **
   OpenSearch install type: rpm/deb on Linux 6.1.79-99.167.amzn2023.x86_64 amd64
   OpenSearch config dir: /usr/share/opensearch/config/
   OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
   OpenSearch bin dir: /usr/share/opensearch/bin/
   OpenSearch plugins dir: /usr/share/opensearch/plugins/
   OpenSearch lib dir: /usr/share/opensearch/lib/
   Detected OpenSearch Version: 2.13.0
   Detected OpenSearch Security Version: 2.13.0.0
   Admin password set successfully.
   ### Success
   ### Execute this script now on all your nodes and then start all nodes
   ### OpenSearch Security will be automatically initialized.
   ### If you like to change the runtime configuration 
   ### change the files in ../../../config/opensearch-security and execute: 
   sudo "/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh" -cd "/usr/share/opensearch/config/opensearch-security" -icl -key "/usr/share/opensearch/config/kirk-key.pem" -cert "/usr/share/opensearch/config/kirk.pem" -cacert "/usr/share/opensearch/config/root-ca.pem" -nhnv
   ### or run ./securityadmin_demo.sh
   ### To use the Security Plugin ConfigurationGUI
   ### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/<your-custom-admin-password>.
   ### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
   Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
  Error: Failed to initialize container opensearchproject/opensearch:2.13.0
Error: One or more containers failed to start.

Related component

Search

To Reproduce

1. Have a github action starting opensearch
2. run the action often times
3. See error

Expected behavior

The container never fails to start

Additional Details

No response

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8 9 10]
@mrkamel Thanks for creating this issue; however we are rejecting this issue because it isn't clear how to reproduce this error state. Could you please link to the github action workflow / action with failure after recreating this issue?

[dblock]

The ClassNotFoundException is odd. These are identical containers?

[mrkamel]

@dblock not 100% sure what you mean exactly, but i assume: yes - the containers are unchanged and the same with every run

[dblock]

I've moved this to opensearch-devops and will reopen it.

There's visibly something not working, but it's likely not OpenSearch itself if I were to guess. Since this is intermittent without changes, could it be a file system failure or something like that? Is there a way to narrow this down further?

[mrkamel]

@dblock so far we've unfortunately not been able to narrow it down further. It's running on private runners, but using detault settings and due to everything being ephemeral it is hard to debug. On github's own runners i've not been able to reproduce it, though.

[prudhvigodithi]

[Triage]
Hey @mrkamel you can disable the security plugin if required using DISABLE_SECURITY_PLUGIN env value, please check
https://github.com/opensearch-project/opensearch-build/blob/main/docker/release/README.md#disable-security-plugin-security-dashboards-plugin-security-demo-configurations-and-related-configurations

https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/#sample-docker-compose-file-for-development

Thank you

[mrkamel]

thank you @prudhvigodithi I indeed can not reproduce it so far with 2.13.0 and this env being set

[prudhvigodithi]

Thanks for the update @mrkamel, I will go ahead and close this issue, please feel free to re-open if required.
@getsaurabh02