Probes test failure: index out of range

[vqmarkman]

Happened in PR: https://github.com/sundeck-io/OpsCenter/actions/runs/6047474866/job/16413995185
During initial loading of the page.

probes.spec.cy.js.mp4

cc @rymurr

[vqmarkman]

My conclusion is that this is OpsCenter bug and I can't reproduce it locally :(
The worst part about it is that video does not show last (most important part) of the stack trace.

@doronrosenberg , @susannahmarcus - is there any other way to debug this type of failure? Where should I be looking for the logs?

[susannahmarcus]

I can't say much since I haven't worked on this, but I can weigh in on the viewport. For future instances like this, we could make it so that it isn't cut off. Additionally, we might be able to add a check for the type of an error message and then have it spit that into the cypress log, so we can get more information.

If you'd like to work on that together, feel free to reach out and we can try some stuff out locally 😊 👍

[joshelser]

The worst part about it is that video does not show last (most important part) of the stack trace.

Well, I think we know about where it is failing. It's inside the _call(..) method, so we can assume it's stuck on the snippet in probes.py

_ = self.snowflake.call(
            "INTERNAL.REPORT_ACTION",
            "probes",
            "list",
        )

self.snowflake here is a snowflake-snowpark-python Session instance, which, I'm guessing we're on the line https://github.com/snowflakedb/snowpark-python/blob/v1.7.0/src/snowflake/snowpark/session.py#L2447 inside of the Snowpark library.

I'd guess that we didn't get any rows back (for whatever reason) like #35 which caused this. Maybe we need to try to figure out why this Session seems to be returning bogus results sometimes?

[vqmarkman]

@joshelser, the "interesting" part about this thing is ... of course this particular case (probes test) is failing in automation and is not failing locally for me. I'm not sure what the next steps here are ...

[joshelser]

I wonder if this is related to a2f5b67

https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/api/snowflake.snowpark.Session mentions that Session is not threadsafe. My thinking is that adding the extra chart on the index page has caused general de-stabilization of the rest of the tests (since they start by going to / and then finding the next thing to click on).

I don't know for sure how the native app framework works, but given what we're seeing, I'd believe that it falls into the bucket of "not thread safe"-ness of the Session. As a quick band-aid, could we wait for warehouse heatmap to fully load before we click past it? Maybe that would help stabilize.

[vqmarkman]

@joshelser, here is more of what we've discovered:

• we were able to reproduce this locally by clicking on the tabs on the left side bar
• it does look like all sorts of random things are failing (not only on probes pages) when other pages are supposedly still being rendered.

• I'm going to put a temporary bandaid (hardcoded sleep for each tab) just to eliminate random failures.
  (until I find out how to correctly wait in Cypress for Streamlit page to finish loading)
• This is NOT going to fix OpsCenter, users can still get random failures while clicking on pages manually