[RFC] Foreign libraries

[snowleopard]

While thinking about extending support for C/C++ libraries (see #2644), we realised that the proposed approach could be simplified and at the same time made more general by switching from c_library to a generic foreign_library stanza:

(foreign_library
  (language c)
  (archive_name my_lib)

  (names file1 file2 ...)
  (flags flag1 flag2 ...)
  (include_dirs dir1 %{lib:another_lib} ...)
  (extra_deps file1 file2 ...))

Here is the semantics:

• The language field selects the compiler. At first, we will support only c and cxx settings, but in future other languages/compilers could be supported, e.g. Rust and Clang.
• The compiler is called on each file matching names in a sandbox. Duplicate name matches are disallowed as in the current implementation to avoid conflicting object names in the archive.
• We pass flags to every invocation of the compiler.
• The resulting objects are placed into the archive_name.a file. To support mixing different languages in the same foreign library, we can allow the same archive_name appear in different foreign_library stanzas, as long as there are no conflicting object names.
• The header directories from include_dirs will be available and tracked as dependencies. They will also appear in the command line in -I flags. Note that you can refer to public headers of other (OCaml) libraries using the %{lib:another_lib} syntax.
• Additionally, extra_deps will be available and tracked to support ad hoc include dependencies.

We can also use the same stanza as a part of an OCaml library definition via foreign_stubs:

(library 

...

  (foreign_stubs (language ... ) ...)

...
)

Any comments/suggestions are welcome!

[rgrinberg]

Additionally, extra_deps will be available and tracked to support ad hoc include dependencies.

Which stage of the build pipeline would this field be used? preprocessing, compilation, or linking?

Are there header files in rust? If not, I feel like we're going to language specific fields. In which case perhaps it makes sense to have a representation like:

(foreign_library
 (language
  (name c)
  (header foo bar)
 ..)

Overall, I like this feature however. I don't see a reason why we can't support all C like languages in one go. This would be usable with Fortran, D, etc as well.

[snowleopard]

Which stage of the build pipeline would this field be used? preprocessing, compilation, or linking?

@rgrinberg I think it will be used in the preprocessing & compilation stage.

Are there header files in rust? If not, I feel like we're going to language specific fields.

Yes, and I think we could just raise an error "unsupported field X for language Y" in this case.

Overall, I like this feature however. I don't see a reason why we can't support all C like languages in one go. This would be usable with Fortran, D, etc as well.

Indeed 🎉

[rgrinberg]

I just had a brief chat with @avsm and this proposal is something that we should take a closer look at. I'll take this opportunity to jot down some notes from our discussion while the context is still fresh in my mind.

I'll start by pointing out that one issue that is specific to foreign_libraries as opposed OCaml libraries is that they're quite often provided externally. Consider ssl - this library can be provided in three different ways on OSX:

• Builtin version that comes with MacOS

• Openssl as provided by homebrew

• A version that is built with foreign_library and is possibly vendored in some ocaml-ssl bindings library.

This might not be the place to address all these concerns, but it's quite important to have a mechanism to consistently resolve such conflicts.

At minimum, I think that we need the following capabilities:

• A way to define external libraries without building them from scratch. At most, the build procedures should be just defining some configuration options and copying some archive files from the system. This is necessary for external libraries where we can't realistically define a dune port.

• A way to state that a single external library may have multiple "sources". This is similar to virtual libraries and implementations in the sense that we may link only one implementation at a time. However, implementations are really just dumb scripts that discover the correct flags, header files, and archives to use these libraries.

A simple proposal in this direction that is modelled after virtual libraries:

(foreign_library
 (virtual)
 (public_name ssl)
 ;; the implementations will need to produce these targets.
 ;; Unlike in OCaml, we don't actually care to make sure the header files are consistent in the "implementations"
 (header ssl.h)
 (archives ssl.a)
 (name ssl)
 ;; :best is basically a way for dune to arbitrarily make a decision that can be 
 ;; overriden by a user via some configuration
 (default_implementation homebrew_ssl))

(foreign_library
 (name homebrew_ssl)
 (implements ssl))

;; rule to "build" the header/so/a files for homebrew_ssl
;; ...

(foreign_library
 (name macos_ssl)
 (implements ssl)
 (headers ssl.h))

;; rule to "build" the header/so/a files for macos_ssl
;; ..

Given the above, it's not strictly necessary to propagate c_flags as I discussed initially with Anil. It's possible to just have rules that will discover where the header files/archives live, and simply copy them to the build dir.

As with virtual libraries, this doesn't solve the issue with configuring large workspaces. At least it provides something that is quite similar to what we already have and solves some concrete problems.

[ghost]

@rgrinberg this proposal seems interesting, could you put it in a separate issue?

[snowleopard]

Implemented in #2659.

Here is an overview:

https://dune.readthedocs.io/en/latest/concepts.html#foreign-sources-and-archives