Installation on Julia >= v1.1 broken on Windows

[JuergenWiemers]

As mentioned here installation on Windows starting from Julia v1.1 seems to be broken right now. This is still an issue on Julia v1.5. When trying to install DSGE.jl in a clean environment I get:

ERROR: Error when installing package DSGE:
AssertionError: length(dirs) == 1
Stacktrace:
 [1] install_archive(::Array{Pair{String,Bool},1}, ::Base.SHA1, ::String) at D:\buildbot\worker\ppackage_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Operations.jl:562
 [2] macro expansion at D:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\srackage_win64\build\usr\share\julia\stdlib\v1.rc\Operations.jl:727 [inlined]
 [3] (::Pkg.Operations.var"#58#61"{Bool,Pkg.Types.Context,Dict{Base.UUID,Array{String,1}},Channelc\Operations.jl:727 [inlined]l{Any},Channel{Any}})() at .\task.jl:356  

The error occurs in Pkg.jl's Operations.jl in the function install_archive. The function checks whether the temporary folder, which includes the extracted DSGE.jl (or any other) package, contains exactly one path - excluding a possible spurious file called pax_global_header, which seemingly 7z might create on Windows:

            dirs = readdir(dir)
            # 7z on Win might create this spurious file
            filter!(x -> x != "pax_global_header", dirs)
            @assert length(dirs) == 1
            unpacked = joinpath(dir, dirs[1])

However, for DSGE.jl the contents of the temporary folder that is created during the installation of DSGE.jl look like this:

julia> tp = "C:/Users/Juergen/AppData/Local/Temp/84mXPHT7jq6E/" # that's the random temp path
julia> readdir(tp)
13-element Array{String,1}:
 "4e6ee4544e19ad2bafb35de9c647ce2e1d3f74c7.data"
 "4e6ee4544e19ad2bafb35de9c647ce2e1d3f74c7.paxheader"
 "5b8fa286e6a8358c29532fad1b593b644169585e.data"
 "5b8fa286e6a8358c29532fad1b593b644169585e.paxheader"
 "634244007a6e3cbe50fba432cce3285b48c42ee2.data"
 "634244007a6e3cbe50fba432cce3285b48c42ee2.paxheader"
 "FRBNY-DSGE-DSGE.jl-a824dcd"
 "bc56c6b6969d5a0662d910e703089c2786d69168.data"
 "bc56c6b6969d5a0662d910e703089c2786d69168.paxheader"
 "e17f8432420fc31d1ddcca8003fba9b91968c096.data"
 "e17f8432420fc31d1ddcca8003fba9b91968c096.paxheader"
 "eef350acd564c108036f8adf58f321418d003fa1.data"
 "eef350acd564c108036f8adf58f321418d003fa1.paxheader"

Thus, filtering pax_global_header doesn't help because it isn't even there. Instead there is a bunch of other .data and .paxheader files, which are not filtered out such that @assert length(dirs) == 1 errors.

I guess this would be easy to fix in Julia. Simpliy replacing

filter!(x -> x != "pax_global_header", dirs)

with

filter!(x -> (x != "pax_global_header") &
            !endswith(x, ".data") &
            !endswith(x, ".paxheader") , dirs)

should do the trick (have submitted the issue to Julia here). However, I thought it might be useful to submit the issue here as well, since I never had this issue with any other package (and I installed quite a lot...). What's so special about the DSGE.jl package that 7z creates these weird "spurious" files on Windows?

Version info:

Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA = "C:\Users\Juergen\AppData\Local\Programs\Julia 1.5.0\bin\"
  JULIA_EDITOR = "C:\Users\Juergen\AppData\Local\Programs\Microsoft VS Code\Code.exe"
  JULIA_NUM_THREADS = 8

[JuergenWiemers]

Just to confirm, after applying my fix above DSGE.jl can be installed on Julia v1.5 for Windows.

[JuergenWiemers]

Executing run_default.jl seems to work as well (I am not sure whether the missing values in the data are to be expected, though...)

Creating dataset...
API key loaded.
Fetching FRED series GDP...
Fetching FRED series CNP16OV...
Fetching FRED series GDPDEF...
Fetching FRED series AWHNONAG...
Fetching FRED series CE16OV...
Fetching FRED series COMPNFB...
Fetching FRED series PCEPILFE...
Fetching FRED series DFF...
Fetching FRED series PCE...
Fetching FRED series FPI...
Fetching FRED series BAA...
Fetching FRED series BAMLC8A0C15PYEY...
Fetching FRED series GS10...
Fetching FRED series GDI...
Updated data from FRED written to C:\Users\Juergen\.julia\packages\DSGE\tJSkz\save\input_data\raw\fred_181115.csv.
┌ Warning: C:\Users\Juergen\.julia\packages\DSGE\tJSkz\save\input_data\raw\dlx_181115.csv was not found; missings used
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:242
┌ Warning: C:\Users\Juergen\.julia\packages\DSGE\tJSkz\save\input_data\raw\ois_181115.csv was not found; missings used
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:242
┌ Warning: df[obs_longinflation] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_longrate] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_tfp] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_nominalrate1] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_nominalrate2] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_nominalrate3] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_nominalrate4] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_nominalrate5] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
┌ Warning: df[obs_nominalrate6] is all missing.
└ @ DSGE C:\Users\Juergen\.julia\packages\DSGE\tJSkz\src\data\load_data.jl:428
dataset creation successful
Reoptimizing...
Iter     Function value   Gradient norm 
     0     6.826539e+03     6.003667e+03
     1     1.569033e+03     5.943879e+02
     2     1.461981e+03     7.444913e+02
     3     1.236580e+03     3.203990e+02
     4     1.163819e+03     2.254286e+02

[chenwilliam77]

Thanks for figuring out the problem for us, or at least a fix to whatever is causing the problem. I'll update the README to point to this post.

With regard to the data, run_default uses Model1002, which uses observables not available from FRED (e.g. we use overnight index swaps to proxy for interest rate expectations during ZLB periods). We get this data through vendors and sources that cannot be released publicly, hence the missing observables.

[JuergenWiemers]

William, can you help @StefanKarpinski (see here) regarding the TAR archive of DSGE.jl?

[StefanKarpinski]

The fix here as detailed in JuliaLang/Pkg.jl#1943 is to not include such long paths in the package. There are paths in this package that are 195 bytes long with file names as long as 109 bytes. That has multiple problems:

1. It's too long to be included in a TAR file without extended headers (max file name: 100 bytes), which the 7z tar extractor does not understand, which is why these bogus .data and .paxheader files are created.

2. Too long for the standard Windows path length limit of 260 UTF-16 characters. The length of these paths inside of packages leaves very little room for the user's home directory.

The error handling can be improved but if you want this package to be installable on Windows, shorten the paths.

[chenwilliam77]

@JuergenWiemers Can you try dev'ing DSGE? I removed the problematic test files generated by the plot test script (although maybe there are other files that are problematic). Unfortunately, I don't have access to a Windows machine, or else I would test it myself.

Alternatively, you can check here: https://api.github.com/repos/FRBNY-DSGE/DSGE.jl/tarball/c6a269cab44459bc1747c3316a9b6bee1ac61887

[JuergenWiemers]

@chenwilliam77 Well, this is kind of embarassing. Before dev'ing DSGE, I first tried to reproduce the error. So I started from a clean state (I even completely deleted the .julia folder) and switched off the long pathname support in the Windows registry. DSGE 1.1.6 installed without problems. Then I tried the same thing on a second Windows machine, which didn't even have Julia installed before. Again, no problems with installing DSGE on that machine. It goes without saying that the current master of DSGE works as well.

I'm not sure what to make of that. Some weird Heisenbug?

[chenwilliam77]

Hrmmmmm, well, whatever happened, I'm glad it works now. Fingers crossed that my changes to the plotting tests are all that's required in the future!

Thanks for all the help in getting this sorted out.