ModuleNotFoundError: No module named 'pkg_resources' #4756
Comments
|
we're tracking work on this in dotnet/runtime#4751. we found the problem another way and noticed it seems to be specific to ArmArch Linux Docker containers just yesterday. hoping to move away from the deprecated and, now, apparently sometimes unavailable package in time to include the fix in our next rollout am I correct the containers failing in your build are quite infrequently used❓ |
|
This is failing on many PRs in dotnet/runtime. The affected containers are used by default dotnet/runtime CI configuration. |
You can see it in the stats in the top post. |
This affects number of Linux and macOS variants. For example, here is a log from macOS x64: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/Microsoft.Extensions.Configuration.Tests/1/console.47b8555d.log?helixlogtype=result |
|
I think the common factor is having Python 3.12, e.g. if you look at https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/ComInterfaceGenerator.Tests/1/console.aa96da09.log?helixlogtype=result from the same job as Jan posted above it worked, because the dci-mac-build-133 macOS machine is not using Python 3.12 (I guess because it wasn't updated yet) |
I agree. it also seems like the problem is only partially under our direct control. we use unless we missed some older builds that failed, problems started just after our rollout began updating queues yesterday. the earliest error I saw had timestamp |
|
Python 3.12.8 released on 2024-12-03. it contains a bunch of fixes and dependency updates though nothing obviously linked to I checked the |
|
we're reverting yesterday's rollout due to the problems discussed in this issue. queues are getting updated as I type this note |
|
revert is now complete but I don't see a clear signal that this particular problem has been resolved. please let us know |
|
Seems to be working again. Though I noticed that e.g. a build on dci-mac-build-108 which failed before is still printing |
that's very interesting. I can only guess why an exception turned into a simple message |
|
@jkotas with the rollout revert we performed, it appears that the issue has been mitigated (per the telemetry above, no new build failures have been detected in over a week). Is there any reason to continue to keep this issue open? |
|
I agree - this can be closed. |
|
We're using this issue to track a proper fix to the deployments. (Though of course I may have missed a conversation, so let me know if this was resolved elsewhere.) |
totally agree. the revert seemed to avoid the problem but this issue tracks making sure our next rollout doesn't break Python scenarios again. we're close but not done |
|
if you care about the details, our Helix wrapping code for Linux machines hit some issues using
|
|
|
|
that machine was unreachable. switched to using |
|
not quite done w/ testing but I'm pretty sure the remaining overlapping this is the fact our to avoid such messages about |
|
ugh, I was wrong. reporter/run.py sometimes shows up in other Helix console logs |
|
builds are expiring and getting deleted. here's an example of the original osx.1200.amd64 failure: reporter/run.py and xharness-event-processor.py come from dotnet/arcade. run.py executes unconditionally as the last command in I suspect problems causing the user Python environment to be empty (lacking any of our usual dependencies) e.g., Python packages are installed per-version and a version bump can break things. but I searched through all of the logs above to find the macOS failures among the builds that remain. could only find three such logs, all running on just two machines — I'm trying to work w/ |
- see dotnet/dnceng#4756 - with this, executing `python3` should not result in `pkg_resources not found` error - note direct Python dependencies in dotnet/arcade include only default modules - change in Microsoft.DotNet.Build.Tasks.Installers likely less important
|
sorry for not reporting back here. we had a total of three problems leading to the the reason for the problems w/ Python 3.12+ was PEP 668 enforcement, which is particularly stringent in macOS. messing with the system or user Python environments was strongly discouraged and failed unless you removed some files that really shouldn't be touched ( you'll notice these issues are all long-standing problems that went unnoticed. it's as if something introduced |
|
@dougbu These failures started hitting dotnet/runtime CI heavily again earlier today. dotnet/runtime#112083 has the details. Could you please take a look? |
|
new issue likely has a similar root cause to this problem i.e., something I can't find switching regardless, I believe we (DNCEng) need to revert our rollout again 😦 |
|
using this issue to track the problems addressed to date. will track the latest problems in dotnet/runtime#112083 note we need another small PR to include the macOS configuration changes (adding the |
Build
https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=910866
Build leg reported
ComInterfaceGenerator.Tests.WorkItemExecution
Pull Request
dotnet/runtime#110558
Known issue core information
Fill out the known issue JSON section by following the step by step documentation on how to create a known issue
{ "ErrorMessage" : "ModuleNotFoundError: No module named 'pkg_resources'", "BuildRetry": false, "ErrorPattern": "", "ExcludeConsoleLog": false }@dotnet/dnceng
Release Note Category
Release Note Description
Additional information about the issue reported
No response
Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=910866
Error message validated:
[ModuleNotFoundError: No module named 'pkg_resources']Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/8/2025 10:20:43 PM UTC
Report
Summary
The text was updated successfully, but these errors were encountered: