Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

escriptize failing with elixir dependencies in rebar3_elixir_compile (find_deps_of_deps) #1568

Closed
PragTob opened this issue Jun 5, 2017 · 9 comments
Labels
mix/elixir use case that specifically hits elixir users through mix

Comments

@PragTob
Copy link

PragTob commented Jun 5, 2017

I want to run an escript with elixir dependencies (well, basically I just wanna run a script in erlang with elixir dependencies and figured escript would be the way to go). Repo is here: https://github.com/PragTob/benchee_erlang_try (calling the elixir code from an eshell ession works)

Worth noting that I already filed an issue with the rebar3 plugin for using elixir dependencies but haven't received an answer yet and am also not sure what/where it goes wrong barrel-db/rebar3_elixir_compile#21

Thanks a lot in advance :)

Environment

tobi@speedy ~/github/benchee_erlang $ rebar3 report escriptize
Rebar3 report
 version 3.3.6
 generated at 2017-06-05T20:21:08+00:00
=================
Please submit this along with your issue at https://github.com/erlang/rebar3/issues (and feel free to edit out private information, if any)
-----------------
Task: escriptize
Entered as:
  escriptize
-----------------
Operating System: x86_64-unknown-linux-gnu
ERTS: Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]
Root Directory: /home/tobi/.asdf/installs/erlang/19.1/lib/erlang
Library directory: /home/tobi/.asdf/installs/erlang/19.1/lib/erlang/lib
-----------------
Loaded Applications:
bbmustache: 1.3.0
certifi: 0.4.0
cf: 0.2.2
common_test: 1.12.3
compiler: 7.0.2
crypto: 3.7.1
cth_readable: 1.2.4
dialyzer: 3.0.2
edoc: 0.8
erlware_commons: 1.0.0
eunit: 2.3.1
eunit_formatters: 0.3.1
getopt: 0.8.2
inets: 6.3.3
kernel: 5.1
providers: 1.6.0
public_key: 1.2
relx: 3.22.3
sasl: 3.0.1
snmp: 5.2.4
ssl_verify_fun: 1.1.1
stdlib: 3.1
syntax_tools: 2.1
tools: 2.8.6

-----------------
Escript path: /home/tobi/dev/bin/rebar3
Providers:
  app_discovery as clean compile compile compile config cover ct cut deps dialyzer do docs edoc escriptize eunit get-deps help info install install_deps key list lock new owner path pkgs publish release relup report search shell state tar tree unlock update upgrade upgrade upgrade user version xref 

...

Current behaviour

Escriptizing fails around here somewhat similar to #1560 as in the compiled things might be in _elixir_build and not _build although I see the dependencies in _build as well.

Error occurs during escriptizing.

tobi@speedy ~/github/benchee_erlang $ DEBUG=1 rebar3 escriptize
===> Load global config file /home/tobi/.config/rebar3/rebar.config
===> Not adding provider hex user from module rebar3_hex_user because it already exists from module rebar3_hex_user
===> Not adding provider hex cut from module rebar3_hex_cut because it already exists from module rebar3_hex_cut
===> Not adding provider hex config from module rebar3_hex_config because it already exists from module rebar3_hex_config
===> Not adding provider hex key from module rebar3_hex_key because it already exists from module rebar3_hex_key
===> Not adding provider hex info from module rebar3_hex_info because it already exists from module rebar3_hex_info
===> Not adding provider hex owner from module rebar3_hex_owner because it already exists from module rebar3_hex_owner
===> Not adding provider hex docs from module rebar3_hex_docs because it already exists from module rebar3_hex_docs
===> Not adding provider hex search from module rebar3_hex_search because it already exists from module rebar3_hex_search
===> Not adding provider hex publish from module rebar3_hex_pkg because it already exists from module rebar3_hex_pkg
===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'

===> 	opts: []

===> Port Cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'
Port Opts: [exit_status,{line,16384},use_stdio,stderr_to_stdout,hide,eof]

===> Expanded command sequence to be run: [{default,app_discovery},
                                                  {default,install_deps},
                                                  {default,lock},
                                                  {default,compile},
                                                  {default,escriptize}]
===> Not adding provider hex user from module rebar3_hex_user because it already exists from module rebar3_hex_user
===> Not adding provider hex cut from module rebar3_hex_cut because it already exists from module rebar3_hex_cut
===> Not adding provider hex config from module rebar3_hex_config because it already exists from module rebar3_hex_config
===> Not adding provider hex key from module rebar3_hex_key because it already exists from module rebar3_hex_key
===> Not adding provider hex info from module rebar3_hex_info because it already exists from module rebar3_hex_info
===> Not adding provider hex owner from module rebar3_hex_owner because it already exists from module rebar3_hex_owner
===> Not adding provider hex docs from module rebar3_hex_docs because it already exists from module rebar3_hex_docs
===> Not adding provider hex search from module rebar3_hex_search because it already exists from module rebar3_hex_search
===> Not adding provider hex publish from module rebar3_hex_pkg because it already exists from module rebar3_hex_pkg
===> Not adding provider ex compile from module rebar3_elixir_compile_prv_ex_compiler because it already exists from module rebar3_elixir_compile_prv_ex_compiler
===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'

===> 	opts: []

===> Port Cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'
Port Opts: [exit_status,{line,16384},use_stdio,stderr_to_stdout,hide,eof]

===> Not adding provider hex user from module rebar3_hex_user because it already exists from module rebar3_hex_user
===> Not adding provider hex cut from module rebar3_hex_cut because it already exists from module rebar3_hex_cut
===> Not adding provider hex config from module rebar3_hex_config because it already exists from module rebar3_hex_config
===> Not adding provider hex key from module rebar3_hex_key because it already exists from module rebar3_hex_key
===> Not adding provider hex info from module rebar3_hex_info because it already exists from module rebar3_hex_info
===> Not adding provider hex owner from module rebar3_hex_owner because it already exists from module rebar3_hex_owner
===> Not adding provider hex docs from module rebar3_hex_docs because it already exists from module rebar3_hex_docs
===> Not adding provider hex search from module rebar3_hex_search because it already exists from module rebar3_hex_search
===> Not adding provider hex publish from module rebar3_hex_pkg because it already exists from module rebar3_hex_pkg
===> Not adding provider ex compile from module rebar3_elixir_compile_prv_ex_compiler because it already exists from module rebar3_elixir_compile_prv_ex_compiler
===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'

===> 	opts: []

===> Port Cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'
Port Opts: [exit_status,{line,16384},use_stdio,stderr_to_stdout,hide,eof]

===> Verifying dependencies...
===> Upgrading benchee ({elixir,"benchee","0.8.0"})
Dependency benchee already exists

===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'

===> 	opts: []

===> Port Cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'
Port Opts: [exit_status,{line,16384},use_stdio,stderr_to_stdout,hide,eof]

===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: /home/tobi/.asdf/shims/mix deps.get

===> 	opts: [{cd,"/home/tobi/github/benchee_erlang/_elixir_build/benchee"},
                       {use_stdout,true}]

===> Port Cmd: /home/tobi/.asdf/shims/mix deps.get
Port Opts: [{cd,"/home/tobi/github/benchee_erlang/_elixir_build/benchee"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof]

Running dependency resolution...
All dependencies up to date
===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: /home/tobi/.asdf/shims/mix compile

===> 	opts: [{cd,"/home/tobi/github/benchee_erlang/_elixir_build/benchee"},
                       {use_stdout,true}]

===> Port Cmd: /home/tobi/.asdf/shims/mix compile
Port Opts: [{cd,"/home/tobi/github/benchee_erlang/_elixir_build/benchee"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide,eof]

===> Moving checkout "/tmp/.tmp_dir308357801541" to "/home/tobi/github/benchee_erlang/_build/default/lib/benchee"
===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: mv /tmp/.tmp_dir308357801541 /home/tobi/github/benchee_erlang/_build/default/lib/benchee

===> 	opts: [{use_stdout,false},abort_on_error]

===> Port Cmd: mv /tmp/.tmp_dir308357801541 /home/tobi/github/benchee_erlang/_build/default/lib/benchee
Port Opts: [exit_status,{line,16384},use_stdio,stderr_to_stdout,hide,eof]

===> App bunt is no longer needed and can be deleted.
===> App credo is no longer needed and can be deleted.
===> App deep_merge is no longer needed and can be deleted.
===> App dialyxir is no longer needed and can be deleted.
===> App earmark is no longer needed and can be deleted.
===> App ex_doc is no longer needed and can be deleted.
===> App fs is no longer needed and can be deleted.
===> App mix_test_watch is no longer needed and can be deleted.
===> run_hooks("/home/tobi/github/benchee_erlang", pre_hooks, compile) -> no hooks defined

===> Compiling benchee_erlang
===> Adding Elixir Libs
===> sh info:
	cwd: "/home/tobi/github/benchee_erlang"
	cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'

===> 	opts: []

===> Port Cmd: elixir -e 'IO.puts :code.lib_dir(:elixir)'
Port Opts: [exit_status,{line,16384},use_stdio,stderr_to_stdout,hide,eof]

===> run_hooks("/home/tobi/github/benchee_erlang", pre_hooks, compile) -> no hooks defined

===> run_hooks("/home/tobi/github/benchee_erlang", pre_hooks, erlc_compile) -> no hooks defined

===> erlopts []
===> files to compile ["/home/tobi/github/benchee_erlang/_build/default/lib/benchee_erlang/src/benchee_erlang.erl"]
===> run_hooks("/home/tobi/github/benchee_erlang", post_hooks, erlc_compile) -> no hooks defined

===> run_hooks("/home/tobi/github/benchee_erlang", pre_hooks, app_compile) -> no hooks defined

===> run_hooks("/home/tobi/github/benchee_erlang", post_hooks, app_compile) -> no hooks defined

===> run_hooks("/home/tobi/github/benchee_erlang", post_hooks, compile) -> no hooks defined

===> run_hooks("/home/tobi/github/benchee_erlang", post_hooks, compile) -> no hooks defined

===> run_hooks("/home/tobi/github/benchee_erlang", pre_hooks, escriptize) -> no hooks defined

===> run_hooks("/home/tobi/github/benchee_erlang", pre_hooks, escriptize) -> no hooks defined

===> Building escript...
===> Creating escript file /home/tobi/github/benchee_erlang/_build/default/bin/benchee_erlang
===> processing <<"benchee_erlang">>
===> new deps of <<"benchee_erlang">> found to be [<<"benchee">>]
===> processing <<"benchee">>
===> new deps of <<"benchee">> found to be []
===> processing <<"deep_merge">>
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace or consult rebar3.crashdump
===> Uncaught error: {badmatch,error}
===> Stack trace to the error location:
[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,246}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,80}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar3,main,1,
         [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar3.erl"},
          {line,66}]},
 {escript,run,2,[{file,"escript.erl"},{line,757}]},
 {escript,start,1,[{file,"escript.erl"},{line,277}]}]

Expected behaviour

I would like my escript to be build so that I can run benchee benchmarks from erlang in a compiled escript seamlessly :) Any help is gladly welcome, granted, I'm not much of an Erlang/rebar3 specialist :)

Thanks for all the great work 🚀

@ferd ferd added the mix/elixir use case that specifically hits elixir users through mix label Jun 9, 2017
@ferd
Copy link
Collaborator

ferd commented Jun 9, 2017

Hrm. This is a tricky one, and since I don't use Elixir (don't have it installed) this is tricky to debug.

My guess is that this happens because the dependency is declared in the .app file generated for it, but the deep_merge dependency is not locatable by rebar3 within the apps it has in memory (see https://github.com/erlang/rebar3/blob/master/src/rebar_app_utils.erl#L49-L51). This probably has to do with the fact that escriptize depends on compile, but without the elixir-specific hook being part of it, and so there is no state for the escript aspect of it being visible.

One thing to try would be to add the elixir plugin's compile hooks as a pre-hook to escriptize, or to at least run the compile job once before building the escript. This is not guaranteed to work since hooks cannot inject state into their related apps, but it's worth a try since it could make things be in the right paths to be discovered.

If it doesn't work, then the elixir plugin may have to provide its own functionality that can encompass both. I don't know that we can easily or reasonably make an escriptize version that does not trust its own compile dependency and re-runs a discovery for third-party plugin usage.

@uwiger
Copy link
Contributor

uwiger commented Sep 15, 2017

I have an issue in the setup repos related to this, and think I've found a perverse workaround.
uwiger/setup#41

The solution in rebar2 was to invoke make escriptize in the post_hooks.

{post_hooks, [{compile, "${MAKE:-make} escriptize"}]}.

For rebar3 (checked in rebar.config.script), we remove that post_hook and insert a provider_hooks entry.

case erlang:function_exported(rebar3, main, 1)
    true ->
        lists:keydelete(post_hooks, 1, CONFIG1)
            ++ [{provider_hooks, [{post, [{compile, escriptize}]}]}];
    _ ->
        CONFIG1
end.

While this works for rebar3, it doesn't seem to work when mix invokes rebar3 bare compile. Checking for this and leaving the rebar2 solution in place seems to work. This means that the make command will actually use rebar2 for the escriptize operation (perhaps not necessary, but that's how it works in the current Makefile.)

Thus:

IsRebar3Bare = (nomatch =/= re:run(init:get_plain_arguments(), "rebar3bare")).
...
%% It seems `rebar3 bare compile` (mix) fails if the following subst is made.
case erlang:function_exported(rebar3, main, 1) andalso (not IsRebar3Bare) of
    true ->
        lists:keydelete(post_hooks, 1, CONFIG1)
            ++ [{provider_hooks, [{post, [{compile, escriptize}]}]}];
    _ ->
        CONFIG1
end.

Hardly a generic, or particularly satisfactory, solution, but perhaps it can inspire something better?

@tsloughter
Copy link
Collaborator

tsloughter commented Sep 15, 2017 via email

@tsloughter
Copy link
Collaborator

Argh, I completely mis read this in my morning haze and now when rereading it I realize what I said was completely unrelated.

It still may be an issue with mix and project plugins but it is not related to what is going on here.

@gpad
Copy link

gpad commented Dec 27, 2017

I have a similar issue compile a project that contains cuttlefish.
This is the stack trace:

[{rebar_prv_escriptize,find_deps_of_deps,3,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,246}]},
 {rebar_prv_escriptize,find_deps,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,240}]},
 {rebar_prv_escriptize,escriptize,2,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,104}]},
 {rebar_prv_escriptize,do,1,
                       [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_escriptize.erl"},
                        {line,80}]},
 {rebar_core,do,2,
             [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
              {line,153}]},
 {rebar_hooks,run_provider_hooks_,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,49}]},
 {rebar_hooks,run_all_hooks,6,
              [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_hooks.erl"},
               {line,16}]},
 {rebar_prv_compile,compile,3,
                    [{file,"/home/tristan/Devel/rebar3/_build/default/lib/rebar/src/rebar_prv_compile.erl"},
                     {line,129}]}]

It's an elixir project and the problem is related to rebar3 bare compile --paths I would add that with and old version of rebar 3.2 I don't have any issue ...

@bitwalker
Copy link

I think I have some idea of what is going on here, as I just ran into this with edump and decided to explore a bit.

In my case, I hit the same stack trace in find_deps_of_deps as mentioned previously, but only when compiled with rebar3 bare compile --paths <path> not with rebar3 compile. From what I can gather, the escriptize task is not looking in the paths provided via --paths for applications, it is instead looking in _build/default (in my case, but the point is that it is not looking in the paths passed via the flag). To replicate, follow the steps below:

  • git clone https://github.com/bitwalker/edump_issue
  • cd edump_issue && mix do deps.get, compile
  • At this point you will see that Mix failed to compile the project, moving on..
  • cd deps/edump
  • DEBUG=1 rebar3 bare compile --paths ../../_build/dev/lib/*/ebin You'll notice that it failed to build with the badmatch error mentioned previously
  • DEBUG=1 rebar3 compile works fine

Based on what rebar3 is doing when it fails (looking up an application in a list of applications), the only thing that could possibly be different is if the way it's looking up applications is affected by --paths somehow, in such a way that it either can't find the application at all, or fetching information about the application fails. Strictly speaking, I don't believe this is an Elixir/Mix issue, but rather something in either the way rebar3 bare compile works, or in the escriptize task specifically.

/cc @ericmj

linduxed added a commit to esl/ex_cuttlefish that referenced this issue Aug 1, 2019
When building the cuttlefish library as a dependency through `mix`, the
build fails when attempting to run the `escriptize` hook.

Comment the hook out to make building with `mix` work.

The problem has been mentioned on the [Elixir forum][1], and technical
discussion can be found [on Github][2].

  [1]: https://elixirforum.com/t/erlang-dependency-doesnt-compile-with-mix/7043
  [2]: erlang/rebar3#1568
marianoguerra added a commit to marianoguerra/uniendo that referenced this issue Sep 8, 2019
@marianoguerra
Copy link
Contributor

Solution, in rebar.config:

{overrides, [{override, cuttlefish, [{provider_hooks, []}]}]}.

@tsloughter
Copy link
Collaborator

The way to use elixir deps is with https://github.com/Supersonido/rebar_mix these days so I'm consdiering closing this and requiring it be reopened if there is still an issue when using the new plugin.

@PragTob
Copy link
Author

PragTob commented Jul 5, 2020

💯 agreed!

@PragTob PragTob closed this as completed Jul 5, 2020
picaoao added a commit to picaoao/cuttlefish that referenced this issue Aug 11, 2020
Without commenting the line, it would fail to compile when compiling as a dep in mix.exs.
erlang/rebar3#1568
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mix/elixir use case that specifically hits elixir users through mix
Projects
None yet
Development

No branches or pull requests

7 participants