[BUG] OpenSearch Dashboards is NOT Deployed after the cluster(master 3, data2) configuration finished. #853
Comments
YeonghyeonKO
added
bug
YeonghyeonKO
changed the title
|
I have observed the exact same behavior. I had deployed this in a staging environment 6 months ago and the issue was not present. When preparing to bring the service to production this week, I observed that I can repeatedly redeploy opensearch and the dashboard will silently never come up. This is my live configuration: The live environment looks like this: Similar to @YeonghyeonKO's reported issue I never get the dashboard to deploy. Because this worked 6 months ago I tried altering the configuration quite a bit with no change in behavior. I did 10+ redeploys with minimal variation always with the same end result. 6 months ago when deploying with the operator version My deployment is also facilitated via argocd and is very similar, with the exact same issue. Please let me know if I can provide any specific diagnostic data to help get this issue addressed. What is your host/environment? |
|
@YeonghyeonKO I did more debugging and actually got the dashboard to come up. In my case, my pdb settings were wrong; I had a log line similar to one of yours that gave me the hint In your case, if you explicitly set It is odd that this setting completely stops the deploy component of the dashboards in the operator but this might also be your root cause |
|
@jd4883 Hi, To tell you the conclusion, migration of Kubernetes Cluster from 1.20.1 to 1.25.6 does work. By changing just version of k8s regardless whether expliciting the below yaml, nodePools:
...
pdb:
enable: false
# enable: true
# minAvailable: 1
...
Also, with @jd4883 I'm just curious about your working environment. But you are trying to inject trusted-cas.pem using Secret (kind).
|
|
@YeonghyeonKO thanks for the detailed response. Glad you got yours behaving now too. A few things that might make this make more sense:
The deployment I have is still getting some fine tuning so it's as close to perfect when it comes time to deploy it to production. I am pre-injecting all the configuration files into a secret which I've done by extending the chart as such (this is an additional template file for the chart to render): That secret grabs all the files which I have been working out optimal configurations for. Previously, this was problematic because what I had ready to go from 6 months back was not spinning up the dashboard, but now that I've dug into the issue I have a working baseline in my branch to work with and am figuring out how to improve it further. At this point I am not functionally blocked anywhere but hopefully this explains a bit as to why my configuration may be a bit different with regards to ingress support. Kind regards, Jacob |
This error log told me that a PDB Api resource cannot be deployed |
|
I think what made the error harder to catch is the error does not come up frequently and if you miss it its hard to spot. If there is an error mid deployment, maybe a functional improvement to the chart is to be more verbose about a specific error that keeps occurring so if you start sifting through logs to debug, the error is easier to find. That is my 2c though. Glad to know why it was a problem and how to fix it for the moment. |
|
@jd4883 I agree with your opinion, the problem above OpenSearch itself, (for example Kubernetes Cluster, this time) is hard to find why something is wrong or not working. |
getsaurabh02
moved this from 🆕 New
to Planned (Next Quarter)
in Engineering Effectiveness Board
|
I seem to be unable to get it working also with EKS 1.29 + OS 2.11.1, even though its status is green. |
Remove this monitoring and redeploy, Opensearch Dashboards will get redeployed. I think Opensearch Operator is not designed to use by providing this configuration. This apprears to be a bug |
|
Remove this monitoring and redeploy, Opensearch Dashboards will get redeployed. I think Opensearch Operator is not designed to use by providing this configuration. This apprears to be a bug |
What is the bug?
securityconfig-updateis complete so the.opendistro_securityindex exists. Also, abootstrappod had been terminated normally after leader election between Master nodes.This is the logs from ArgoCD.
What is your host/environment?
The text was updated successfully, but these errors were encountered: