Problems with zlib in a haskell project

[commandodev]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I'm trying to deploy a fairly straightforward haskell (servant project).

I've specified zlib and zlib.dev in --pkgs but I'm still running into:

#12 119.8
#12 119.8 --  While building package zlib-0.6.2.1 (scroll up to its section to see the error) using:
 
#12 119.8       /root/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1 configure --with-ghc=/root/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.5/bin/ghc-8.6.5 --with-ghc-pkg=/root/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.5/bin/ghc-pkg-8.6.5 --user --package-db=clear --package-db=global --package-db=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/pkgdb --libdir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/lib --bindir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/bin --datadir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/share --libexecdir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/libexec --sysconfdir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/etc --docdir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/doc/zlib-0.6.2.1 --htmldir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/doc/zlib-0.6.2.1 --haddockdir=/root/.stack/snapshots/x86_64-linux-tinfo6/1d8d3d2d1848c85724a4602c53666fa3959874eda6c0f7830bd85ea082eb5f39/8.6.5/doc/zlib-0.6.2.1 --dependency=base=base-4.12.0.0 --dependency=bytestring=bytestring-0.10.8.2 -f-non-blocking-ffi -f-pkg-config --exact-configuration --ghc-option=-fhide-source-paths
#12 119.8     Process exited with code: ExitFailure 1
 
#12 ERROR: executor failed running [/bin/bash -ol pipefail -c stack build]: exit code: 1
-----
> [8/9] RUN  stack build:
-----
executor failed running [/bin/bash -ol pipefail -c stack build]: exit code: 1
 
Error: Docker build failed

To reproduce

I would guess this should reproduce with most haskell project (zlib is a common pain).

I will try to repro and fix in vanilla nix locally.

Expected behavior

I would have expected the stack build to succeed.

Environment

Local:

Ubuntu

❯ nixpacks --version
nixpacks 0.0.17

Railway project ID is 042095d5-6077-432c-ac60-351f0dffc072/service/267f19e6-9a03-428f-ab79-2f2beace373f

[coffee-cup]

Thanks for the bug report @commandodev !

Can you try building with NIXPACKS_APT_PKG=pkg-config.

Also, do you have a minimal repro that can be used to test with (and later add to the examples directory that we can add a test against).

[commandodev]

I've created the bare minimum servant project with stack new <name> servant here: https://github.com/cdodev/railway-servant-haskell

Just deploying now and will see if the same happens (I'll also try NIXPACKS_APT_PKG=pkg-config)

[commandodev]

Confirmed that that one fails out of the box, as well as with NIXPACKS_APT_PKG=pkg-config added to the environment

[coffee-cup]

Thanks for the test repo. It looks like there is a lot of discussion around zlib with Nix here commercialhaskell/stack#2975. I'm going to try a few of the suggestions and will report back.

[commandodev]

Yeah funny enough, I've just tried adding a nix section to my stack file and overriding the build command. Let's see.

zlib, haskell/stack and nix has been a pain the rear forever!

[coffee-cup]

I was able to get it to build successfully with this config. I'm not exactly what did it though 😆 Going to narrow it down.

As a side note, building takes a bit of time. Do you know a good directory that we can cache to speed up builds? Keeping in mind that the directory will not show up in the final image

[commandodev]

.stack-work would be a good candidate.

I suspect that it's the addition of zlibg-dev into apt pkgs that's done it here. This means that stack isn't really using nix for its packages properly, this has also been a pain basically forever 🤣.

That being said, a win is a win!

[commandodev]

What's the distinction between _LIBS and _PKGS in those last two env vars @coffee-cup?

[coffee-cup]

What's the distinction between _LIBS and _PKGS in those last two env vars @coffee-cup?

_LIBS uses the Nix makeLibraryPath function and appends the result to LD_LIBRARY_PATH. The relevant nix code is

                APPEND_LIBRARY_PATH = \"${{lib.makeLibraryPath [ {libraries} ] }}\";
                myLibraries = writeText \"libraries\" ''
                  export LD_LIBRARY_PATH=\"${{APPEND_LIBRARY_PATH}}:$LD_LIBRARY_PATH\"
                '';

[commandodev]

Confirming that adding zlib1g-dev worked for me too (on the simple repro).

AFIAR (and it's been a while since I was using nix in anger) the library thing is more for packing software up than using existing packages... It would probs be better to stick with packages if possible (I also didn't have _LIBS in by build that worked)

[coffee-cup]

Confirming that adding zlib1g-dev worked for me too (on the simple repro).

Awesome! Do you think that this is common enough to always install with the Haskell provider?

[commandodev]

Awesome! Do you think that this is common enough to always install with the Haskell provider?

Yes I would say so

[commandodev]

@coffee-cup - happy to test out any speed improvements you want to try. I guess I could set NIXPACKS_INSTALL_CACHE_DIRS myself right?

[coffee-cup]

Yes I would say so

I can make this change then 😄

I guess I could set NIXPACKS_INSTALL_CACHE_DIRS myself right?

Ya for sure! NIXPACKS_BUILD_CACHE_DIRS can also be used for build specific caches.

[commandodev]

OK, giving that a try.

One last thing - my original build is working now, but my overrridden command is having some problems:

stack exec server -- -p $PORT is telling me that the executable can't be found. It's working on the example repo with no override (which I think just defaults to the first exe name).

Any idea?

[commandodev]

Ya for sure! NIXPACKS_BUILD_CACHE_DIRS can also be used for build specific caches.

Hmmm, not sure adding .stack-work made any difference - what should I be looking for in the logs?

[commandodev]

Adding ~/.stack would also be a good idea I think. Is it possible to add directories that aren't in $CWD?

[coffee-cup]

OK, giving that a try.

One last thing - my original build is working now, but my overrridden command is having some problems:

stack exec server -- -p $PORT is telling me that the executable can't be found. It's working on the example repo with no override (which I think just defaults to the first exe name).

Any idea?

What executable is not available? stack ? Something might need to be added to the PATH ?

[coffee-cup]

Ya for sure! NIXPACKS_BUILD_CACHE_DIRS can also be used for build specific caches.

Hmmm, not sure adding .stack-work made any difference - what should I be looking for in the logs?

It will only make a difference on subsequent builds. In the logs you should see that the cache=type=mount prefixes the relevant install/build command. But the main indicator will be that the total build time has decreased.

Adding ~/.stack would also be a good idea I think. Is it possible to add directories that aren't in $CWD?

Yes anything can be added. The cache dir in this case will be /root/.stack

[commandodev]

Hmmm 12 0.436 /bin/bash: line 1: .stack-work,target=/root/.stack: No such file or directory

Something not quite right.

[coffee-cup]

The cache directory is only available for that specific phase (install/bundle). If that directory is needed for both phases then make sure to set both cache env vars. If that directory is needed for runtime that we can't the nixpacks caching mechanism since. However, often there is a subdirectory (e.g. /root/.stack/cache) that can be cached.

[coffee-cup]

I've closed the issue since the original bug is fixed. However, if you figure out the best directories to cache please lmk (or make a PR). Still interested in improving the Haskell build times 😄