Too deeply nested script executions lead to not found NuGet packages

[UweKeim]

I migrated a larger project of mine from using CS-Script for .NET Framework to the "new" CS-Script for .NET Core.

Errors

While most of the scripts run successfully I experience errors like these:

error CS0246: Der Typ- oder Namespacename "ICSharpCode" wurde nicht gefunden (möglicherweise fehlt eine using-Direktive oder ein Assemblyverweis).

Translated to English this should read:

error CS0246: The type or namespace name 'ICSharpCode' could not be found (are you missing a using directive or an assembly reference?).

This happens only when the script is executed from within other scripts.

My script "architecture"

Basically I'm doing a chain of something like this.

1. start "a.cmd" which internally calls "css a.cs".
2. "a.cs" does some processing and later uses Process.Start to start "b.cmd".
3. "b.cmd" calls "css b.cs".
4. "b.cs" does some processing and later uses Process.Start to start "c.cmd".
5. "c.cmd" calls "css c.cs".
6. "c.cs" does some processing and later uses Process.Start to start "d.cmd".
7. etc.

I'm not quite sure but I do think the maximum nesting level is about 3 or 4.

In addition, I sometimes execute child scripts in a loop and this also seems to fail after some loops.

Assumption

This works for some nesting depth but suddenly leads to above errors when the nesting depth is too deep.

(At least this is my strong assumption)

Discoveries

I do log a lot of stuff, beside others the Path environment variable.

To my surprise for every call to my scripts, it seems to get larger and something like this is appended:

C:\P\zp-develop\ZetaProducerContent\Development\Build\Build;C:\Users\ukeim.dotnet\tools.store\cs-script.cli\4.8.12\cs-script.cli\4.8.12\tools\net8.0\any;C:\Users\ukeim.nuget\packages\sharpziplib\1.4.2\lib\net6.0;C:\Users\ukeim.nuget\packages\jetbrains.annotations\2023.3.0\lib\netstandard2.0;C:\Users\ukeim.nuget\packages\morelinq\4.1.0\lib\net8.0;C:\Users\ukeim.nuget\packages\newtonsoft.json\13.0.3\lib\net6.0;C:\Users\ukeim.nuget\packages\restsharp\110.2.0\lib\net7.0;C:\Users\ukeim.nuget\packages\zetadeployer.runtimeuploader\1.0.86\lib\net8.0;C:\Users\ukeim.nuget\packages\zetashortpaths\2.0.35\lib\net7.0;C:\Users\ukeim.dotnet\tools.store\cs-script.cli\4.8.12\cs-script.cli\4.8.12\tools\net8.0\any\lib;C:\ProgramData\cs-script\commands;C:\ProgramData\cs-script\inc;C:\Users\ukeim\AppData\Local\Temp\csscript.core\cache\449437209

and later it gets bigger:

C:\P\zp-develop\ZetaProducerContent\Development\Build\Build;C:\Users\ukeim.dotnet\tools.store\cs-script.cli\4.8.12\cs-script.cli\4.8.12\tools\net8.0\any;C:\Users\ukeim.nuget\packages\sharpziplib\1.4.2\lib\net6.0;C:\Users\ukeim.nuget\packages\jetbrains.annotations\2023.3.0\lib\netstandard2.0;C:\Users\ukeim.nuget\packages\morelinq\4.1.0\lib\net8.0;C:\Users\ukeim.nuget\packages\newtonsoft.json\13.0.3\lib\net6.0;C:\Users\ukeim.nuget\packages\restsharp\110.2.0\lib\net7.0;C:\Users\ukeim.nuget\packages\zetadeployer.runtimeuploader\1.0.86\lib\net8.0;C:\Users\ukeim.nuget\packages\zetashortpaths\2.0.35\lib\net7.0;C:\Users\ukeim.dotnet\tools.store\cs-script.cli\4.8.12\cs-script.cli\4.8.12\tools\net8.0\any\lib;C:\ProgramData\cs-script\commands;C:\ProgramData\cs-script\inc;C:\Users\ukeim\AppData\Local\Temp\csscript.core\cache\449437209;C:\P\zp-develop\ZetaProducerContent\Development\Build\Build;C:\Users\ukeim.dotnet\tools.store\cs-script.cli\4.8.12\cs-script.cli\4.8.12\tools\net8.0\any;C:\Users\ukeim.nuget\packages\sharpziplib\1.4.2\lib\net6.0;C:\Users\ukeim.nuget\packages\jetbrains.annotations\2023.3.0\lib\netstandard2.0;C:\Users\ukeim.nuget\packages\morelinq\4.1.0\lib\net8.0;C:\Users\ukeim.nuget\packages\newtonsoft.json\13.0.3\lib\net6.0;C:\Users\ukeim.nuget\packages\restsharp\110.2.0\lib\net7.0;C:\Users\ukeim.nuget\packages\zetadeployer.runtimeuploader\1.0.86\lib\net8.0;C:\Users\ukeim.nuget\packages\zetashortpaths\2.0.35\lib\net7.0;C:\Users\ukeim.dotnet\tools.store\cs-script.cli\4.8.12\cs-script.cli\4.8.12\tools\net8.0\any\lib;C:\ProgramData\cs-script\commands;C:\ProgramData\cs-script\inc;C:\Users\ukeim\AppData\Local\Temp\csscript.core\cache\449437209

(My largest Path value was 3421 characters long in total)

My assumption is that the Path variable gets larger for every nested call to the a script and then it is too large and it cannot find the paths to the NuGet packages anymore and thus results in the above error.

Request for help

Does the above makes some sense?

Is there a way to not completely rewrite/redesign my nested calling structure and still fix this issue?

[oleg-shilo]

Uwe, just letting you know... I am looking into it.
Give me another day or so.

[oleg-shilo]

Uwe, everytime a script is executed it adds user search dirs to the path:

But it is done once per process. Thus it should not accumulate as I assume you are talking about script execution in new processes.
....
Though I think when you fork a chile process it add inherits all search dirs of its parent. This can actually lead to accumulation.

[oleg-shilo]

Uwe, I just changed the implementation of that algorithm (a368d5b)

Can you check if it addresses the problem of growing PATH?
It is a prerelease: https://github.com/oleg-shilo/cs-script/releases/tag/v4.8.14.1

Please note that PATH can also be updated by any other apps sunning in the same parent-child process branch.

[UweKeim]

I've tried the preview version directly to run on my project.

It now immediately fails on some scripts with the original error message of missing references, even without any nested calls.

I try to reproduce this in an isolated minimal script but currently fail. Will continue trying.

What I discovered is that running

"C:\foo\cs-script.win.v4.8.14.1\cscs.exe" /dbg "my-script.cs"

always seems to not generate those *.cs.nuget.cs files anymore whereas running

css /dbg "my-script.cs"

with the stable version installed as a .NET Tool actually does create them.

---

Update 1

I still fail to create an isolated set of example scripts but now even other projects that succeed with the stable version fail with the new preview version.

Update 2

I've tried with v4.8.14.0 and this version also does not create those *.cs.nuget.cs files.

Update 3

I've tried with v4.8.12.0, i.e. the very same version that I also have installed as a global .NET Tool.

Running

"C:\foo\cs-script.win.v4.8.12.0\cscs.exe" /dbg "my-script.cs"

does not create those *.cs.nuget.cs files whereas running

css /dbg "my-script.cs"

does.

I'm very confused now.

What might I do wrong?

[oleg-shilo]

The preview was a throw-away build just to quickly test the immediate idea in your environment. Most likely I have made some mistake there. Please remove that preview and I will rebuild it after I troubleshoot it. The change itself is valid. There is also another possibility to oversaturate envars via storing assembly path. I have fixed it by allowing switching off in CSScriptLib.dll but I did not do it for exe. I did not anticipate an intensive parent-child process forking. I just overlooked it.

I am looking into it.

What will also help, is a hello-world test case that shows the problem for the v4.8.14. Will need it if I cannot get anywhere on my own.

[UweKeim]

Thanks, Oleg. As I added in my above comment, even 4.8.12, when not running as a .NET Tool, no *.cs.nuget.cs files are being created.

I cannot get my head around what is different (on my system) to have such a behavior.

[oleg-shilo]

I am also a little puzzled.
I am looking at the code and "*.cs.nuget.cs" is only being processed if you have legacyNugetSupport disabled. But in case of prerelease, it is enabled because you have not disabled it explicitly as you did for the globally installed version.

So it explains.

This will do the trick.

.\css -config:set:LegacyNugetSupport=false
# ensure you are not executing 
css -config:set:LegacyNugetSupport=false

But... I do not see any code that actually creates this file regardless of the settings. CS-Script seems to read it but not to create it. It does not make sense but I am trying to see what is going on.

Any ideas?

[UweKeim]

I see, thanks a lot, Oleg. In fact I did call css -config:set:LegacyNugetSupport=false for the global .NET Tool but not for the local test download versions.

Will try now whether this makes any difference.

[oleg-shilo]

Yep. This will create that *.nuget.cs file.
I have misinterpreted the code. It was right in front of me:

creation of this file is part of the new Nuget support. It contains all nuget assemblies that dotnet aggregates on the dotnet restore and then dotnet publish.

Thus your problem with pre-release will be solved by disabling the LegacyNugetSupport algorithm. I probably should now disable it by default.

After that you can see if the change in the path makes any difference.

I also check in your scripts if Environment.GetEnvironmentVariables is growing as you go to the children scripts. Probably it is

[oleg-shilo]

I have also added a more detailed description about the way nuget support works: https://github.com/oleg-shilo/cs-script/wiki/NuGet-Support#how-does-it-work

[oleg-shilo]

With your pre-release binaries, I expect your PATH variable to sto growing.

Though for aver script execution there will be a single new variable:

If you have a huge amount of nested execution it can exhaust your envar buffer.
If it's what is happening there I can create a mechanism for disabling the storing location of the script assembly in the envars.

[UweKeim]

Thanks again, Oleg!

I'm just in the middle of letting my scripts run with v4.8.14.1 and until now it seems to run way better.

I'll post here an update when if succeeded or failed.

[oleg-shilo]

Not a prob. When you have time.

[UweKeim]

A first, shorter run through the scripts succeeded. 🥰

Will try another more thorough run now.

[oleg-shilo]

I also disabled that location-envar by default. But it will only be available with the next pre- or a proper release.

[UweKeim]

Running my full build scripts with v4.8.14.1 results again in these errors

error CS0246: The type or namespace name 'ICSharpCode' could not be found (are you missing a using directive or an assembly reference?).

I'll simply stick to legacy CS-Script for now and when the new version is released (no need to hurry!) I'll try again with this version.

If you want me to test other previews simply drop me a line, I'm happy to do so 😊

[oleg-shilo]

OK, then let's put it on pause until it is ready.
In the meantime, I will do a formal prerelease so you can try it in isolation. But most likely I will need a testcase (hello-world) that lets me troubleshoot and catch it in the action.

[UweKeim]

Thanks. I've tried several hours to create a test case without success 😭

[oleg-shilo]

:)
then indeed you need a break.
I will try to create one again but maybe programmatically.
Give me the names of the packages you are using there.

[UweKeim]

The names of the packages seem to be arbitrary. One of them is e.g. SharpZipLib.

[oleg-shilo]

I have attached the automated test that is testing deep to 25 levels of nesting.
So far I have not triggered the problem. Runs fine.
If you want to run it you will need to update the rootDir available in the script. And ensure NugetLegacySupport is disabled. :)

cs-script_#364.zip

[UweKeim]

Thank you!

I've let your script run for several hours now and I even added looping.

Even with v4.8.12 I could not generate any errors.

Drives me crazy!

[oleg-shilo]

I know the feeling..... :o(