Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random CI failure in master #5724

Closed
alasarr opened this issue May 5, 2021 · 6 comments
Closed

Random CI failure in master #5724

alasarr opened this issue May 5, 2021 · 6 comments
Assignees
Labels

Comments

@alasarr
Copy link
Contributor

alasarr commented May 5, 2021

Description

After merge MVT triangulation for binary data it looks the test are failing:

https://github.com/visgl/deck.gl/runs/2501602495#step:5:6607

Expected Behavior

CI should pass

@alasarr
Copy link
Contributor Author

alasarr commented May 5, 2021

@Pessimistress reported to me that MVT render tests are failing randomly, would be great @felixpalmer if you could take a look

@felixpalmer
Copy link
Collaborator

I think this is related to what @Pessimistress reported, I re-ran the tests and they came through fine: https://github.com/visgl/deck.gl/runs/2507689263. I'm looking into what is the cause though

@felixpalmer
Copy link
Collaborator

I've done some investigation and cannot pin down a cause. For reference here is a failing run and a successful run. I've compared the raw logs using a diff viewer and can't find any clue as to why it is failing.

I've run the tests locally and the issue is not reproduced. Also re-running the tests on the CI system results in success.

It is true that I submitted a PR that touches the MVT layer, but we should note that this was pulling in a version of loaders.gl (3.0.0-alpha.6) on the CI system that didn't have the corresponding changes and thus should be a no-op as far as the CI is concerned.

Thus my conclusion here is that this is a spurious error, and I'm not inclined to investigate further. @Pessimistress if you have some useful background or ideas I'm all ears but right now I feel I've reached a point where it is too time-consuming to continue.

@ibgreen
Copy link
Collaborator

ibgreen commented May 6, 2021

@felixpalmer It is a render test that fails. The MVTLayer is rendered and compared against a "golden image" but the matching pixels are only 22%.

Since the render delta is so big, and sporadic, it makes me think that it could be a timing issue - sometimes the tiles are loaded, sometimes not.

I though we had synchronization in place but maybe there is some issue with the MVT tile layer. We should probably look into that.

The surprising thing is that your changes should make things load faster, so if there is indeed a sync issue it stands to reason that we should see less problems, not more.

Not sure if this gives you any actionable leads.

If the CI failure is very rare we might be able to land this. @Pessimistress will have to make the call.

@Pessimistress
Copy link
Collaborator

The failure is not caused by this particular PR. It started showing up about a week ago.

@Pessimistress Pessimistress changed the title Test are failing in master after merge #512 Random CI failure in master May 7, 2021
@Pessimistress
Copy link
Collaborator

Seems to be resolved by #5780

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants