Windows Docker image for conda-forge development

[xhochy]

I have some Windows systems (also includes throwaway VMs using cloud providers) that I use to develop and debug conda-forge recipe failures on Windows. Sadly I have never had a Windows system that was working with all conda-forge packages. Most often I have to resort to developing against the CI. The documentation we have is okish, improving it would help but having a Windows docker image that replicates our CI setup would greatly help me to enable packages for Windows.

Would it be possible to create such an image or Dockerfile? (we might have to use the latter as we may not be able to redistribute some SDKs?)

[mariusvniekerk]

I think a viable solution is to have an ansible playbook somewhere that can provision a windows machine.

[xhochy]

Meanwhile I have a basic Dockerfile that works for building packages with native code on Windows:

FROM mcr.microsoft.com/windows/servercore:ltsc2019

WORKDIR C:/Users/Administrator

# Be patient, this takes quite a while and you see no action.
RUN powershell -Command " \
    $url = \"http://go.microsoft.com/fwlink/?LinkId=691126\"; \
    $client = new-object System.Net.WebClient; \
    $client.DownloadFile( $url, \"msvc_build_tools.exe\"); \
    ./msvc_build_tools.exe /full /q | Write-Output; \
    Start-Sleep -s 20; \
    del msvc_build_tools.exe; \
    "

RUN powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))"
RUN chocolatey feature enable -n allowGlobalConfirmation
RUN choco install neovim

RUN powershell -Command " \
    $url = \"https://repo.continuum.io/miniconda/Miniconda3-latest-Windows-x86_64.exe\"; \
    $client = new-object System.Net.WebClient; \
    $client.DownloadFile( $url, \"miniconda3.exe\"); \
    ./miniconda3.exe /S /D=C:\Users\Administrator\miniconda3 | Write-Output; \
    del miniconda3.exe; \
    ./miniconda3/Scripts/conda clean --all -y; \
    "

RUN miniconda3\Scripts\activate.bat && \
    conda config --add channels conda-forge && \
    conda install -y conda-build git && \
    conda config --set always_yes true && \
    conda config --set anaconda_upload false

Only works reliably if you disable any Virus scanner.

[jjhelmus]

@xhochy This is really neat. Do you know what version of Visual Studio/Visual C++ is installed by the first RUN step?

[jakirkham]

Do you know why virus scanners need to be disabled?

[xhochy]

Do you know why virus scanners need to be disabled?

I have often had the problem that conda was unable to delete files due to permission problems. Normally this is an indicator that another process also has an open handle to that file. These problems vanished completely after disabling Windows Defender.

[xhochy]

@xhochy This is really neat. Do you know what version of Visual Studio/Visual C++ is installed by the first RUN step?

VS Build Tools 2015. With the latest changes on conda-forge, I need to find the 2017 ones. Currently I only see links for 2015 and 2019.

Meanwhile I found that on chochlatey also the tools are packages and I will try to use them instead of the direct download link as that should make the version more obvious and hopefully the installation less mysterious.

[jakirkham]

I wonder if Conda is not closing files properly then.

[mingwandroid]

No @jakirkham, Windows File IO does not do file locking well. That is the problem here.

[jakirkham]

cc @jaimergp (for reference)

[carterbox]

I updated @xhochy's recipe to use the VS build tools installer. This installer allows you to provide a config file, so you know exactly what tools are being installed.

FROM mcr.microsoft.com/windows/servercore:ltsc2019

SHELL ["cmd", "/S", "/C"]

WORKDIR C:/Users/Administrator

# Exported from Visual Studio Installer GUI, this config file is used to
# explicitly list out which MSVC components are installed. Component names are
# also documented here:
# https://docs.microsoft.com/en-us/visualstudio/install/workload-component-id-vs-build-tools
ADD .vsconfig-2017 "C:\\TEMP\\.vsconfig"

# Be patient, this takes quite a while and you see no action.
RUN powershell -Command " \
    $url = \"https://aka.ms/vs/17/release/vs_buildtools.exe\"; \
    $client = new-object System.Net.WebClient; \
    $client.DownloadFile( $url, \"msvc_build_tools.exe\"); \
    ./msvc_build_tools.exe --quiet --wait --norestart --nocache --config C:\TEMP\.vsconfig | Write-Output; \
    Start-Sleep -s 20; \
    del msvc_build_tools.exe; \
    "

RUN powershell -Command " \
    $url = \"https://repo.continuum.io/miniconda/Miniconda3-latest-Windows-x86_64.exe\"; \
    $client = new-object System.Net.WebClient; \
    $client.DownloadFile( $url, \"miniconda3.exe\"); \
    ./miniconda3.exe /S /D=C:\Users\Administrator\miniconda3 | Write-Output; \
    del miniconda3.exe; \
    ./miniconda3/Scripts/conda clean --all -y; \
    "

RUN miniconda3\Scripts\activate.bat && \
    conda config --add channels conda-forge && \
    conda install -y conda-build git && \
    conda config --set always_yes true && \
    conda config --set anaconda_upload false

CMD cmd.exe /k miniconda3\Scripts\activate.bat

.vsconfig-2017

{
  "version": "1.0",
  "components": [
    "Microsoft.VisualStudio.Component.Roslyn.Compiler",
    "Microsoft.Component.MSBuild",
    "Microsoft.VisualStudio.Component.CoreBuildTools",
    "Microsoft.VisualStudio.Workload.MSBuildTools",
    "Microsoft.VisualStudio.Component.Windows10SDK",
    "Microsoft.VisualStudio.Component.VC.CoreBuildTools",
    "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
    "Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
    "Microsoft.VisualStudio.Component.Windows10SDK.19041",
    "Microsoft.VisualStudio.Component.VC.CMake.Project",
    "Microsoft.VisualStudio.Component.TestTools.BuildTools",
    "Microsoft.VisualStudio.Component.VC.ASAN",
    "Microsoft.VisualStudio.Component.TextTemplating",
    "Microsoft.VisualStudio.Component.VC.CoreIde",
    "Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
    "Microsoft.VisualStudio.Component.VC.v141.x86.x64",
    "Microsoft.VisualStudio.Workload.VCTools"
  ]
}

This one is 7GB larger for a total of 15GB! It includes additional things such as Windows CMake tools and the Windows 10 SDK, but I believe these things are necessary for CMake to function correctly.