OS resource issues with recent changes (0.4.9)

[james-s-w-clark]

Summary:

millw (since ~ 1 day ago) appears to:

• leave too many files open
• use too many native threads

---

I'm currently experimenting in CI to try and fix things, but I believe the issue is a recent change in millw.

This works fine:

# Download mill
RUN curl --fail -L -o /usr/local/bin/mill https://github.com/lihaoyi/mill/releases/download/0.10.12/0.10.12 && \
    chmod 755 /usr/local/bin/mill

...

a bunch of compilation + more Mill tasks
...

This has been having issues, since yesterday:

# Download millw
RUN curl --fail -L -o /usr/local/bin/mill https://raw.githubusercontent.com/lefou/millw/main/millw && \
    chmod 755 /usr/local/bin/mill
...
mill --version
mill __.compile

Even just invoking --version gets issues. The issues when using millw are like:

/usr/local/bin/mill: fork: retrty: Resource temporarily unavailable
/usr/local/bin/mill: fork: retrty: No child processes 
Caused by: java.io.IOException: Cannot run program "/usr/lib/jvm/java-17-amazon-corretto/bin/java": error=11, Resource temporarily unavailable
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "/usr/lib/jvm/java-17-amazon-corretto/bin/java": error=11, Resource temporarily unavailable

-       15:20:03  Error: Could not create the Java Virtual Machine.
-       15:20:03  Error: A fatal exception has occurred. Program will exit.
-       15:20:03  Could not flush log: stdout (Broken pipe (32))
-       15:20:03  Error: A JNI error has occurred, please check your installation and try again
-       15:20:03  Exception in thread "main" java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

On some runs, too many files are left open - which causes issues for future Jenkins jobs running on the same swarm agents. They can fail with some OS tools, python scripts, whatever - anything that wants to read files:

10:54:07  ImportError: libk5crypto.so.3: cannot open shared object file: Too many open files in system
10:54:07  Traceback (most recent call last):
10:54:07    File "/usr/libexec/urlgrabber-ext-down", line 22, in <module>
...
10:54:07  ImportError: librt.so.1: cannot open shared object file: Too many open files in system

Mostly I've been blowing away the EC2 instance each time to clear things (trying to avoid blowing up my own PC's resources)

[james-s-w-clark]

I see readme instructions have changed, will look into that. Will try 0.4.9 and 0.4.8.

Edit: pinning to 0.4.8 of the wrapper I don't get the issue.

[james-s-w-clark]

I tried to make a reproducible example.

https://github.com/IdiosApps/millw-tester/actions/runs/6341629848/job/17225692316. It gets stuck in a loop:

#6 [3/4] RUN curl -L -o mill https://raw.githubusercontent.com/lefou/millw/0.4.9/millw &&     chmod +x mill &&     mv mill /usr/local/bin/
#6 0.697   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
#6 0.699                                  Dload  Upload   Total   Spent    Left  Speed
#6 0.702 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  7747  100  7747    0     0  96603      0 --:--:-- --:--:-- --:--:-- 96837
#6 DONE 0.8s

#7 [4/4] RUN mill --version
#7 0.078 No mill version specified.
#7 0.078 You should provide a version via '.mill-version' file or --mill-version option.
#7 0.083 find: ‘/root/.cache/mill/download/.latest’: No such file or directory
#7 0.084 Retrieving latest mill version ...
#7 0.187 Using mill version 0.11.4
#7 0.199 No mill version specified.
#7 0.199 You should provide a version via '.mill-version' file or --mill-version option.
#7 0.206 Using mill version 0.11.4
#7 0.216 No mill version specified.
#7 0.216 You should provide a version via '.mill-version' file or --mill-version option.
#7 0.222 Using mill version 0.11.4
#7 0.233 No mill version specified.
#7 0.233 You should provide a version via '.mill-version' file or --mill-version option.
#7 0.239 Using mill version 0.11.4
#7 0.249 No mill version specified.
#7 0.249 You should provide a version via '.mill-version' file or --mill-version option.
#7 0.256 Using mill version 0.11.4
#7 0.265 No mill version specified.

I'll try again with the version specified.

---

Same log spam with .mill-version in root: https://github.com/IdiosApps/millw-tester/actions/runs/6341650799/job/17225760045

---

Same, with --mill-version: https://github.com/IdiosApps/millw-tester/actions/runs/6341684086/job/17225865431

---

Using 0.4.8, there is no infinite logs: https://github.com/IdiosApps/millw-tester/actions/runs/6341719164/job/17225972414

---

The lesson for me is to pin to a specific version (of course!).
The new install instructions in the readme are indeed version specific in the curl, so that's a great addition 👍

I hope these help resolve the issue :)

[lefou]

Thank you for this report. I already thought about such an situation just after I've merges PR #51, but later forgot about that "potential issue". The issue is, that we try to use a system-installed mill executable, if installed, but if that executable itself is a millw script trying to use the system-installed mill executable, we have a never ending story.

@Flowdalic Looks like we need some checks that the "system" executable we want to ask for it's version is not us, the millw script itself.

[lefou]

I downgraded the 0.4.9 release to a pre-release and added a note to the changelog. I don't have the bandwidth to fix this issue immediately. Any help would be appreciated.

[Flowdalic]

@IdiosApps Could you check if the PR fixed the issue for you?

[james-s-w-clark]

@Flowdalic Looks good :)
https://github.com/IdiosApps/millw-tester/actions/runs/6350352017/job/17249900431