Disable failing ci builders #7206
Conversation
Pull Request Test Coverage Report for Build kirill.burtsev_116
💛 - Coveralls |
|
Apparently, ubuntu arm64 is even less ready for sanitizers... What is the list of distros we have for evergreen runners? Is there ubuntu2204-arm64-large for example? |
| - name: ubuntu2204-arm64-asan | ||
| display_name: "Ubuntu 22.04 ARM64 (ASAN)" | ||
| run_on: ubuntu2204-arm64-large | ||
| expansions: | ||
| cmake_url: "https://s3.amazonaws.com/static.realm.io/evergreen-assets/cmake-3.20.3-linux-aarch64.tar.gz" | ||
| cmake_bindir: "./cmake_binaries/bin" | ||
| python3: "/opt/mongodbtoolchain/v3/bin/python3" | ||
| use_system_openssl: On | ||
| fetch_missing_dependencies: On | ||
| cmake_build_type: Debug | ||
| enable_asan: On | ||
| tasks: | ||
| - name: compile_test | ||
|
|
There was a problem hiding this comment.
So you got ubuntu2204-arm64-asan to pass by suppressing the messages around adjtime - nice!
There was a problem hiding this comment.
not quite, this is asan, not tsan. The problem with adjtime was a new one after #6911. It also fails builds from time to time. TSAN on ubuntu on arm64 still needs a lot of fixes. The test build on ubuntu2204-arm64-large showed a few hundred issues, i didn't yet look into them apart from linked open ones.
| # FIXME: tsan is not stable on arm64, fails often with internal errors | ||
| # - name: compile_test |
There was a problem hiding this comment.
Was this for macOS as well? I thought it was only for ubuntu2204...
There was a problem hiding this comment.
this is for #7185 - this failure is presumably core_tests and sync_tests specific only on macos 11 arm64. I haven't seen this failures myself on macos 13 arm64, so i hope if we move this builder on macos1300-arm64 then we can reenable whole test suite.
ubuntu on arm64 reports different issues but consistently and for every test suite we have from what i can tell
| - name: compile_test | ||
| # FIXME: tsan is not stable on arm64, fails often with internal errors | ||
| # - name: compile_test | ||
| - name: compile_test_object_store |
There was a problem hiding this comment.
Do you know if the sync-tests pass on this platform?
There was a problem hiding this comment.
they do, but way too often they fail #7185
There was a problem hiding this comment.
@kiburtse you are disabling the extra builders for TSAN only for evergreen as far as I can see. In relation to my comment for this issue, #7185 ... we could potentially get away, since we are releasing our binaries via jenkins (for now). If we are not touching Jenkins, we should be fine.
Other than this, it looks good to me.
well, tsan on macos arm64 was a new builder, apparently it's not very stable. Even compiler on this macos1100-arm64 is not the latest one from AppleClang 13, which should be 13.0.1. I see no point in fighting it and tolerating this failures. It was well over 10s of failures in the last month along. Our ci is constantly red, it's hard to keep track of all these failures. So i'd rather disable partially for now, and resume with macos1300 - it should be better i hope. |
What, How & Why?
☑️ ToDos
📝 Changelog update🚦 Tests (or not relevant)C-API, if public C++ API changedbindgen/spec.yml, if public C++ API changed