How to vendor Dream?

[benjamin-thomas]

Hello!

I'd like to vendor dream into my project.

Even though I ran opam install --deps-only ./dream.opam --with-test insider the vendored dream directory, I get these compile errors at my project root.

Entering directory '/home/benjamin/code/github.com/paurkedal/caqti-study'
File "06-lwt-web-app/lib/web/server/vendor/dream/src/vendor/dune", line 38, characters 11-26:
38 |   (modules gluten_lwt_unix tls_io ssl_io)))
                ^^^^^^^^^^^^^^^
Error: Module Gluten_lwt_unix doesn't exist.
File "06-lwt-web-app/lib/web/server/vendor/dream/src/vendor/dune", line 110, characters 9-20:
110 |   (names gen_huffman gen_static)
               ^^^^^^^^^^^
Error: Module "Gen_huffman" doesn't exist.
File "06-lwt-web-app/lib/web/server/vendor/dream/src/http/shared/websocket.ml", line 8, characters 21-50:
8 | module Websocketaf = Dream_websocketaf.Websocketaf
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error: Unbound module Dream_websocketaf.Websocketaf
File "06-lwt-web-app/lib/web/server/vendor/dream/src/vendor/dune", line 170, characters 11-14:
170 |   (modules paf)
                 ^^^
Error: Module Paf doesn't exist.
File "06-lwt-web-app/lib/web/server/vendor/dream/src/vendor/dune", line 178, characters 11-15:
178 |   (modules alpn)
                 ^^^^
Error: Module Alpn doesn't exist.
File "06-lwt-web-app/lib/web/server/vendor/dream/src/vendor/dune", line 189, characters 11-21:
189 |   (modules paf_mirage)
                 ^^^^^^^^^^
Error: Module Paf_mirage doesn't exist.

I still get complaints about a missing paf module, even after issuing opam install paf.

So it seems I can't manage the vendored libraries of my vendored library.

I found a workaround, by changing Dream's dune file at the root of its vendor folder here. Instead of containing this:

(data_only_dirs *)

(subdir
 gluten/lib
 (library
  (name dream_gluten)
  (public_name dream-httpaf.dream-gluten)
  (libraries bigstringaf faraday ke)))

(subdir
 gluten/lwt
 (library
  (name dream_gluten_lwt)
  (public_name dream-httpaf.dream-gluten-lwt)
  (libraries dream-httpaf.dream-gluten lwt)))
...

It contains just this:

(vendored_dirs *)

And that seems to satisfy my dune instance, not to go further. And my project compiles fine, I can modify Dream's code alright.

Could this dune file be simplified as I did? If not, could you please advise how to proceed?

Thanks

[aantron]

Sorry for the delay! Did you clone Dream with --recursive when vendoring?

[benjamin-thomas]

Hello @aantron, no worries.

So I tried again and this time, it (almost) works

Here's a simple project structure:

$ tree -L 2 server/
server/
├── dune
├── server.ml
└── vendor
    └── dream

I initialized the project like such:

git init .
git submodule add https://github.com/aantron/dream.git ./server/vendor/dream/
git submodule update --init --recursive
cd ./server/vendor/dream/
opam install --deps-only ./dream.opam --with-test

Now the only remaining difficulty is this example which doesn't compile with dune.

$ dune build
File "server/vendor/dream/example/z-playground/runtime/playground.eml.ml", line 56, characters 4-17:
Error: Unbound module Examples

It's unclear to me where Examples should come from.

Also, it looks like this example is old and maybe not relevant anymore? http://dream.as does not respond + the npm run start returns the error Missing script: "bundle"

[benjamin-thomas]

Just an additional info: using the vendored_dirs stanza, dune won't try to compile this problematic project so it's all good!

The number of files to compile is about 1/4 the original size so the compilation is faster too.

(library
 (name server)
 (libraries dream))

(vendored_dirs vendor)

I'd be interested to hear your response about the z-playground before closing.

[aantron]

The playground needs to be fixed (#222). I'm glad the build working. Let's leave this open, as a reminder to me to take a look at why Examples is not found. It's probably because it is generated programmatically on the deploy server. Maybe I'll just add a dummy/placeholder locally so that there is no error during vendoring.