errors don't propagate correctly on windows

[h-vetinari]

While CI in conda-forge/zlib-feedstock#83 is pretty broken due to various issues, I noticed that win-arm64 unexpectedly passed (in particular because even when built with conda-build it currently runs into conda-forge/vc-feedstock#83).

Looking at the logs, I see

 │ │ (base) C:\Program Files\Microsoft Visual Studio\2022\Enterprise>CALL "VC\Auxiliary\Build\vcvars64.bat" -vcvars_ver=14.41 10.0.22621.0 
 │ │ **********************************************************************
 │ │ ** Visual Studio 2022 Developer Command Prompt v17.10.5
 │ │ ** Copyright (c) 2022 Microsoft Corporation
 │ │ **********************************************************************
 │ │ [ERROR:vcvars.bat] Toolset directory for version '14.41' was not found.
 │ │ [ERROR:VsDevCmd.bat] *** VsDevCmd.bat encountered errors. Environment may be incomplete and/or incorrect. ***
 │ │ [ERROR:VsDevCmd.bat] In an uninitialized command prompt, please 'set VSCMD_DEBUG=[value]' and then re-run
 │ │ [ERROR:VsDevCmd.bat] vsdevcmd.bat [args] for additional details.
 │ │ [ERROR:VsDevCmd.bat] Where [value] is:
 │ │ [ERROR:VsDevCmd.bat]    1 : basic debug logging
 │ │ [ERROR:VsDevCmd.bat]    2 : detailed debug logging
 │ │ [ERROR:VsDevCmd.bat]    3 : trace level logging. Redirection of output to a file when using this level is recommended.
 │ │ [ERROR:VsDevCmd.bat] Example: set VSCMD_DEBUG=3
 │ │ [ERROR:VsDevCmd.bat]          vsdevcmd.bat > vsdevcmd.trace.txt 2>&1
 │ │
 │ ╰─────────────────── (took 2 seconds)

however, the build simply continues, and due to the lack of tests in cross-compilation, the build "passes".

Speaking of windows, there's a bunch of mojibake in all the requirements tables:

 │ │ │ ╭──────────────┬──────────────┬─────────────┬─────────────┬────────────╮
 │ │ │ │ Package      ┆ Version      ┆ Build       ┆ Channel     ┆ Size       │
 │ │ │ ╞�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd╪�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd╪�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd╪�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd╪�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd�\ufffd╡
 │ │ │ │ libzlib      ┆ 1.3.1        ┆ h2466b09_2  ┆ .tmpgatR6E  ┆ 8.90 KiB   │
 │ │ │ │ ucrt         ┆ 10.0.22621.0 ┆ h57928b3_0  ┆ conda-forge ┆ 1.22 MiB   │
 │ │ │ │ vc           ┆ 14.3         ┆ h8a93ad2_20 ┆ conda-forge ┆ 16.98 KiB  │
 │ │ │ │ vc14_runtime ┆ 14.40.33810  ┆ hcc2c482_20 ┆ conda-forge ┆ 733.43 KiB │
 │ │ │ ╰──────────────┴──────────────┴─────────────┴─────────────┴────────────╯

[h-vetinari]

Ah, and on win-64, I basically have no logs w.r.t. what's going wrong:

 │ │ ╰─────────────────── (took 1 second)
 │ │
 │ ╰─────────────────── (took 1 second)
 │
 ╰─────────────────── (took 29 seconds)
 × error Error building package: failed to run test: Script failed with status Some(1).
 × error Work directory: "C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\.tmpll7rzR"
 × error To debug the build, run it manually in the work directory (execute the `./conda_build.sh` or `conda_build.bat` script)
Error:   × failed to run test: Script failed with status Some(1).
  │ Work directory: "C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\.tmpll7rzR"
  │ To debug the build, run it manually in the work directory (execute the `./
  │ conda_build.sh` or `conda_build.bat` script)


ERROR conda.cli.main_run:execute(125): `conda run rattler-build build --recipe recipe -m .ci_support\win_64_.yaml --extra-meta flow_run_id=azure_20240909.2.1 --extra-meta remote_url=*** --extra-meta sha=2508762d40532a9e6e823597a818f823c7380697 --experimental` failed. (See above for error)
##[error]Cmd.exe exited with code '1'.

Presumably it also runs into #1051 though

[h-vetinari]

For win-64, the test failures are completely invisible still:

 │ │ │  Successfully updated the environment
 │ │ │ Testing commands:
 │ │ │ (base) C:\Users\VSSADM~1\AppData\Local\Temp\.tmpLlvOMg>IF "" == "" (call C:\Users\VSSADM~1\AppData\Local\Temp\.tmpLlvOMg\build_env.bat ) 
 │ │ │
 │ │ ╰─────────────────── (took 1 second)
 │ │
 │ ╰─────────────────── (took 2 seconds)
 │
 ╰─────────────────── (took 79 seconds)
 × error Error building package: failed to run test: Script failed with status Some(1).
 × error Work directory: "C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\.tmpLlvOMg"
 × error To debug the build, run it manually in the work directory (execute the `./conda_build.sh` or `conda_build.bat` script)
Error:   × failed to run test: Script failed with status Some(1).
  │ Work directory: "C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\.tmpLlvOMg"
  │ To debug the build, run it manually in the work directory (execute the `./
  │ conda_build.sh` or `conda_build.bat` script)

Tests are specified as follows:

    tests:
      - script:
        - if: unix
          then:
            - test ! -f ${PREFIX}/lib/libz.a
            - test ! -f ${PREFIX}/lib/libz${SHLIB_EXT}
            - test ! -f ${PREFIX}/include/zlib.h
          else:
            - if not exist %LIBRARY_BIN%\zlib.dll exit 1
            - if not exist %PREFIX%\zlib.dll exit 1

and I can see that the respective files are actually there:

 │ │ Files in package:
 │ │   - Library\bin\zlib.dll
 │ │   - zlib.dll

logs

[wolfv]

Do you know if this is different from conda-build? We might miss some CLI arg when calling cmd.exe but apart from that idk what's different.

Windows has always been pretty weird with the if %exitcode% exit 1 stuff.

[h-vetinari]

Windows has always been pretty weird with the if %exitcode% exit 1 stuff.

You should use the %-less option. I also ran into problems with this recently (c.f. conda-forge/sentencepiece-feedstock@2169c56); more details here (and dispersed over looooots of SO questions).

[wolfv]

Sure, but we don't actually do anything with error level. We might need a different flag for cmd.exe /XXX though.

[wolfv]

Just FYI there is one line from the script execution in what you pasted:

 │ │ │ Testing commands:
 │ │ │ (base) C:\Users\VSSADM~1\AppData\Local\Temp\.tmpLlvOMg>IF "" == "" (call C:\Users\VSSADM~1\AppData\Local\Temp\.tmpLlvOMg\build_env.bat ) 

Do you have an idea on how to fix this?

Also just an aside, you can also use the new package content test, e.g.:

tests:
   - package_content:
        lib: [zlib]

[wolfv]

I took a look just now, and both conda-build and rattler-build are calling cmd.exe /d /c ..., so there is no difference.

I will try to come up with a short test script to see if I can reproduce or find a better way of dealing with things on Windows. If you have ideas for a good reproducer, please let me know :)

[wolfv]

I made a bunch of changes just now and this is what the error looks like now. With @echo on we see the line that was last executed and we print the error right underneath it now.

[h-vetinari]

Perfect, that's something actually to tackle at least!

[wolfv]

Moji-bake fixed: conda-forge/conda-smithy#2091