Handling of no-entrypoint feature breaks with multiple programs in one workspace

[ruuda]

Problem

Suppose you have a Cargo workspace with two Solana BPF program crates in it, foo and bar, where bar depends on foo with features = ["no-entrypoint"]. There is also a third crate testlib with some helper code for tests. It depends on both foo and bar (with no features), and then foo and bar have a dev-dependency on testlib. Now suppose foo has solana_program_test tests, that set up a test context, and add the foo program (and not bar).

If you now run cargo test-bpf, this will fail like this:

[2021-10-18T14:37:36.279334239Z INFO  solana_program_test] "foo" BPF program from /home/ruud/repos/bpf-repro/target/deploy/foo.so, modified 13 seconds, 272 ms, 319 µs and 472 ns ago
[2021-10-18T14:37:36.603637144Z DEBUG solana_runtime::message_processor] ELF error: Multiple text sections, consider removing llc option: -function-sections
test tests::test_foo::test_foo ... FAILED

(This error message is incorrect. There aren’t multiple text sections, there are zero.)

Now remove the dependency of bar on foo, and the tests will pass just fine, even though bar is not used in the tests.

The error outcome can also be achieved by adding features = ["no-entrypoint"] to the foo = { path = "../foo" } dependency in testlib. @enriquefynn prepared a minimal example here: https://github.com/enriquefynn/solana-test

One thing that stands out is that in the error case, foo.so both in target/bpfel-unknown-unknown and in target/deploy is small (only 1 kB in deploy), but in the success case, it is much larger (44.9 kB in deploy). With objdump --all-headers, we can confirm that in the bad case, the .so file in target/deploy indeed contains no text section:

Idx Name          Size      VMA               LMA               File off  Algn
  0 .dynsym       00000018  00000000000000e8  00000000000000e8  000000e8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .gnu.hash     0000001c  0000000000000100  0000000000000100  00000100  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .hash         00000010  000000000000011c  000000000000011c  0000011c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .dynstr       00000001  000000000000012c  000000000000012c  0000012c  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .eh_frame     0000001c  0000000000000130  0000000000000130  00000130  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .dynamic      00000090  0000000000000150  0000000000000150  00000150  2**3
                  CONTENTS, ALLOC, LOAD, DATA

But in the success case, it does contain a text section:

Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         000086a8  00000000000000e8  00000000000000e8  000000e8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .dynsym       00000078  0000000000008790  0000000000008790  00008790  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .gnu.hash     00000024  0000000000008808  0000000000008808  00008808  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .hash         00000030  000000000000882c  000000000000882c  0000882c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .dynstr       00000028  000000000000885c  000000000000885c  0000885c  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .rel.dyn      00000f90  0000000000008888  0000000000008888  00008888  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .rodata       00000e11  0000000000009818  0000000000009818  00009818  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .eh_frame     000000bc  000000000000a630  000000000000a630  0000a630  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  8 .data.rel.ro  00000448  000000000000a6f0  000000000000a6f0  0000a6f0  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  9 .dynamic      000000d0  000000000000ab38  000000000000ab38  0000ab38  2**3
                  CONTENTS, ALLOC, LOAD, DATA

The text section is also absent in target/bpfel-unknown-unknown, and it is not only the .so file that is smaller in the bad case, the rlib is smaller as well:

target/bpfel-unknown-unknown (bad):
drwxr-xr-x 1 ruud ruud   414 Oct 18 16:23 build
drwxr-xr-x 1 ruud ruud  5310 Oct 18 16:23 deps
drwxr-xr-x 1 ruud ruud     0 Oct 18 16:23 examples
drwxr-xr-x 1 ruud ruud     0 Oct 18 16:23 incremental
-rw-r--r-- 1 ruud ruud   301 Oct 18 16:23 bar.d
-rwxr-xr-x 1 ruud ruud 86424 Oct 18 16:23 bar.so
-rw-r--r-- 1 ruud ruud   162 Oct 18 16:23 foo.d
-rwxr-xr-x 1 ruud ruud  3736 Oct 18 16:23 foo.so
-rw-r--r-- 1 ruud ruud   306 Oct 18 16:23 libbar.d
-rw-r--r-- 1 ruud ruud 30932 Oct 18 16:23 libbar.rlib
-rw-r--r-- 1 ruud ruud   167 Oct 18 16:23 libfoo.d
-rw-r--r-- 1 ruud ruud  5578 Oct 18 16:23 libfoo.rlib

target/bpfel-unknown-unknown (good):
drwxr-xr-x 1 ruud ruud   414 Oct 18 16:23 build
drwxr-xr-x 1 ruud ruud  5310 Oct 18 16:23 deps
drwxr-xr-x 1 ruud ruud     0 Oct 18 16:23 examples
drwxr-xr-x 1 ruud ruud     0 Oct 18 16:23 incremental
-rw-r--r-- 1 ruud ruud   211 Oct 18 16:23 bar.d
-rwxr-xr-x 1 ruud ruud 86424 Oct 18 16:23 bar.so
-rw-r--r-- 1 ruud ruud   211 Oct 18 16:23 foo.d
-rwxr-xr-x 1 ruud ruud 86400 Oct 18 16:23 foo.so
-rw-r--r-- 1 ruud ruud   216 Oct 18 16:23 libbar.d
-rw-r--r-- 1 ruud ruud 30932 Oct 18 16:23 libbar.rlib
-rw-r--r-- 1 ruud ruud   216 Oct 18 16:23 libfoo.d
-rw-r--r-- 1 ruud ruud 31024 Oct 18 16:23 libfoo.rlib

Proposed Solution

I’m not sure how to go about this. I suspect that due to the use of a workspace, the build artifacts for the no-entrypoint configuration overwrite the artifacts for the configuration with entrypoint somehow. If the dependencies are between crates that are not in a workspace, this is not a problem, because they get their own target directories. Probably cargo build-bpf will need to be more careful about where it stores its build artifacts. Avoiding the same name for different build configurations (with and without entry point) may help.

A workaround for now is to remove the bar crate from the workspace.

Might be related to #16329.

[mvines]

Does opting in to use resolver 2 help? https://doc.rust-lang.org/cargo/reference/resolver.html#feature-resolver-version-2

[enriquefynn]

That actually worked! 🖤

[mvines]

Cool, this looks like just a known limitation of the older Cargo feature resolver. Thankfully Rust 2021 is coming to stable soon and will default to resolver 2