escriptize broken when an included app is in _checkouts

[jadeallenx]

Environment

mallen-rMBP:basho_bench mallen$ rebar3 report
Rebar3 report
 version 3.3.1
 generated at 2017-05-26T21:14:40+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: 
Entered as:
  
-----------------
Operating System: x86_64-apple-darwin16.4.0
ERTS: Erlang/OTP 19 [erts-8.2.2] [source] [64-bit] [smp:8:8] [async-threads:0] [kernel-poll:false] [dtrace]
Root Directory: /Users/mallen/erlangs/19.2.3
Library directory: /Users/mallen/erlangs/19.2.3/lib
-----------------
Loaded Applications:
bbmustache: 1.0.4
certifi: 0.4.0
cf: 0.2.1
common_test: 1.13
compiler: 7.0.3
crypto: 3.7.2
cth_readable: 1.2.3
dialyzer: 3.0.3
edoc: 0.8.1
erlware_commons: 0.21.0
eunit: 2.3.2
eunit_formatters: 0.3.1
getopt: 0.8.2
inets: 6.3.5
kernel: 5.1.1
providers: 1.6.0
public_key: 1.3
relx: 3.21.0
sasl: 3.0.2
snmp: 5.2.4
ssl_verify_fun: 1.1.1
stdlib: 3.2
syntax_tools: 2.1.1
tools: 2.9

-----------------
Escript path: /Users/mallen/bin/rebar3
Providers:
  app_discovery as clean compile compile config cover ct cut deps dialyzer do docs edoc escriptize eunit help info install install_deps key list lock new owner path pkgs publish release relup report search shell state tar tree unlock update upgrade upgrade upgrade user version xref 

Current behaviour

Make sure that some project in _checkouts is in the escript_incl_apps list in rebar.config and then:

$ mkdir _checkouts
$ ln -s ../simple_project _checkouts/simple_project
$ rebar3 compile escriptize

===> Building escript...
===> Creating escript file /Users/mallen/github/basho_bench/_build/default/bin/basho_bench
===> processing <<"basho_bench">>
===> new deps of <<"basho_bench">> found to be []
===> processing <<"bear">>
===> new deps of <<"bear">> found to be []
===> processing <<"folsom">>
===> new deps of <<"folsom">> found to be []
===> processing <<"gestalt">>
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,error}
===> Stack trace to the error location:
[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,234}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,72}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,125}]},
 {rebar3,main,1,
         [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar3.erl"},
          {line,56}]},
 {escript,run,2,[{file,"escript.erl"},{line,757}]},
 {escript,start,1,[{file,"escript.erl"},{line,277}]}]

escriptize works fine when everything is inside of the _build directory via deps or something.

Expected behaviour

Everything is copacetic.

[ferd]

@mrallen1 didn't manage to reproduce in 3.4.1. Think you could try again on a newer version?
Also rebar3 compile escriptize is not really a valid command but I assume this is a typo.

[jadeallenx]

OK, I will update my rebar3 and try again. Thanks!

[ferd]

@mrallen1 were you able to reproduce or get a fix from a newer version? Wondering if this is still worth investigating.

[jadeallenx]

I'll give it a go with a new version tomorrow

[ttyerl]

I'm having a similar error running "rebar3 bare compile --paths "/Users/tty/Project/testapp/_build/dev/lib/*/ebin"" on cuttlefish

========
Error: {badmatch,error}
[{rebar_prv_escriptize,find_deps_of_deps,3,
[{file,"/Users/tty/.cache/compile/rebar3/_build/prod/lib/rebar/src/rebar_prv_escriptize.erl"},
{line,247}]},
{rebar_prv_escriptize,find_deps,2,
[{file,"/Users/tty/.cache/compile/rebar3/_build/prod/lib/rebar/src/rebar_prv_escriptize.erl"},
{line,241}]},
{rebar_prv_escriptize,escriptize,2,
[{file,"/Users/tty/.cache/compile/rebar3/_build/prod/lib/rebar/src/rebar_prv_escriptize.erl"},
{line,104}]},
{rebar_prv_escriptize,do,1,
[{file,"/Users/tty/.cache/compile/rebar3/_build/prod/lib/rebar/src/rebar_prv_escriptize.erl"},
{line,80}]},
{rebar_core,do,2,
[{file,"/Users/tty/.cache/compile/rebar3/build/prod/lib/rebar/src/rebar_core.erl"},
{line,154}]},
{rebar_hooks,run_provider_hooks,6,
[{file,"/Users/tty/.cache/compile/rebar3/_build/prod/lib/rebar/src/rebar_hooks.erl"},
{line,49}]},
{rebar_hooks,run_all_hooks,6,
[{file,"/Users/tty/.cache/compile/rebar3/_build/prod/lib/rebar/src/rebar_hooks.erl"},
{line,16}]},
{rebar_prv_compile,compile,3,
[{file,"/Users/tty/.cache/compile/rebar3/_build/prod/lib/rebar/src/rebar_prv_compile.erl"},
{line,153}]}]

========
rebar3 version:
rebar 3.5.3+build.4100.ref60364fc6 on Erlang/OTP 19 Erts 8.3.5.3

MacOSX 10.13.5

[ferd]

It is not surprising that bare compilation fails if it must call escriptize, since the escriptize task does not work with a standalone task; it expects a regular rebar3 project. I don't think we can support this easily.

[ferd]

@mrallen1 was this resolved for you?

[ttyerl]

I tried the previous version of rebar3: rebar 3.2.0 on Erlang/OTP 19 Erts 8.3.5.3
and the issue does not appear with that version.

[jadeallenx]

@ferd I can't remember. AFAIC, it's cool to close this ticket. I'll open a new one if I need to. Thanks.

[ferd]

3.2.0 is a 2 years old version that was broken and there may have been half-a-dozen patches in there. It's unclear to me if the standalone compiler worked with it by accident and got broken by a fix for Erlang users, or if something in rebar3 that is unrelated broke and caused problems. If you can narrow down versions a little bit, it might help.

However, the error you have appears to be on line here: https://github.com/erlang/rebar3/blob/master/src/rebar_prv_escriptize.erl#L247 which would just point to the app not being able to be found anymore.

That code was added in 2016, a few days after 3.2.0, here: 6bc8cce#diff-9d4027415800d49d0a03022f3cdb65eb and was to fix issues where escripts would accidentally always contain all applications, which generated files that are too large for what they needed.

The error you see happens because the app you're trying to build (or one of its dependencies) mentions an OTP application in its .app file that cannot be found in the current rebar3 state. My guess is just that escriptize depends on a compilation job having been run to populate the state, and the compilation job usually depends on deps having been installed locally within the project, which depends on a local app discovery phase. When running the bare compiler, only the compile job is run, and the rebar3 state is not being pre-populated for escriptize to run properly, explaining the crash.

I don't know that there is any fix for this possible. maybe @tsloughter has an idea, but to me it sounds like this task implicitly relies on the rebar3 workflow, and the usage of the bare compiler purposefully undoes that workflow, and so those are just two incompatible goals.

[tsloughter]

Did the bare compile provider exist in 3.2.0?

[ferd]

it started in 2015 (https://github.com/erlang/rebar3/commits/master/src/rebar_prv_bare_compile.erl) so it probably existed, but incidentally did not support the wildcard paths specified by the bug report either, so it might have been all accidental.