Deployment with Docker

[yanshiyason]

Hi, I would like to start this thread in order to document what is the best way to deploy elixir apps that use meeseeks with docker. Once an answer is found, I would like to make a PR to the
deployment guide

I did not yet succeed in using Meeseeks in a docker container successfully.

My 1st attempt, was so add the same commands found in the alpine-3.11 rust Dockerfile do make rust available in my build image.

which failed with this output

error: cannot produce dylib for `meeseeks_html5ever_nif v0.12.1 (/app/deps/meeseeks_html5ever/native/meeseeks_html5ever_nif)` as the target `x86_64-unknown-linux-musl` does not support these crate types
could not compile dependency :meeseeks_html5ever, "mix compile" failed. You can recompile this dependency with "mix deps.compile meeseeks_html5ever", update it with "mix deps.update meeseeks_html5ever" or clean it with "mix deps.clean meeseeks_html5ever"
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
    lib/mix/tasks/compile.rustler.ex:68: Mix.Tasks.Compile.Rustler.compile_crate/1
    (elixir 1.10.3) lib/enum.ex:1396: Enum."-map/2-lists^map/1-0-"/2
    lib/mix/tasks/compile.rustler.ex:15: Mix.Tasks.Compile.Rustler.run/1
    (mix 1.10.3) lib/mix/task.ex:330: Mix.Task.run_task/3
    (mix 1.10.3) lib/mix/tasks/compile.all.ex:76: Mix.Tasks.Compile.All.run_compiler/2
    (mix 1.10.3) lib/mix/tasks/compile.all.ex:56: Mix.Tasks.Compile.All.do_compile/4
    (mix 1.10.3) lib/mix/tasks/compile.all.ex:27: anonymous fn/2 in Mix.Tasks.Compile.All.run/1
    (mix 1.10.3) lib/mix/tasks/compile.all.ex:43: Mix.Tasks.Compile.All.with_logger_app/2

A little bit of google searching and someone proposed to add this environment variable:

export RUSTFLAGS="-C target-feature=-crt-static"

another run with that environment variable set, and the build succeeds, but when I try to start my application, I get this runtime error: (formatted as best as I could)

{supervisor,
  {local,kernel_sup}}
 ,{errorContext,start_error},
    {reason,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'}},
     {offender,[{pid,undefined},{id,kernel_safe_sup},
      {mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}}, 

    ...
#{label=>{proc_lib,crash},report=>[
  [{initial_call,{supervisor,kernel,['Argument__1']}},
  {pid,<0.1459.0>},{registered_name,[]},{error_info,
          {exit,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'}, 
           [{init,run_on_load_handlers,0,[]},
            {kernel,init,1,[{file,"kernel.erl"},
                {line,189}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},

Do you have any idea what the problem may be?

[yanshiyason]

I am making a demo repository to test, I will share in a bit!

[yanshiyason]

https://github.com/yanshiyason/meeseeks_docker_example

OK, so this repo reproduces the current problem. Please have a look.

[yanshiyason]

I noticed that inside the _build folder, there is a rustler_crate folder, but I am not including it in the final image.. it is only inside the build image.. I wonder if I need to bring that over and specify the path somehow..

[yanshiyason]

the rust binary seems to be contained within the build artifact

meeseeks_docker_example/lib/meeseeks_html5ever-0.12.1/priv/native/libmeeseeks_html5ever_nif.so

Maybe it's not related to the rustler_crate folder?

[mischov]

This help? rusterlium/rustler#296

[yanshiyason]

Hi @mischove, Thank you for looking into it. So the solution from the thread above seems to be to add this export this environment variable:

export RUSTFLAGS="-C target-feature=-crt-static"

Which does fix the compilation step. But it does not fix the runtime problem that occurs when trying to start the app..

The error seems to stem from this module: Elixir.MeeseeksHtml5ever.Native

stacktrace:

/app $ prod/rel/meeseeks_docker_example/bin/meeseeks_docker_example start
2020-06-01 04:21:48.255412 supervisor_report   #{label=>{supervisor,start_error},report=>[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'}},{offender,[{pid,undefined},{id,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
2020-06-01 04:21:48.255594 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.724.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'},[{init,run_on_load_handlers,0,[]},{kernel,init,1,[{file,"kernel.erl"},{line,189}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[kernel_sup,<0.699.0>]},{message_queue_len,0},{messages,[]},{links,[<0.701.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,265}],[]]}
2020-06-01 04:21:49.297450 crash_report        #{label=>{proc_lib,crash},report=>[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.698.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[<0.697.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.699.0>,normal}]},{links,[<0.697.0>,<0.696.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,193}],[]]}
2020-06-01 04:21:49.313488 std_info            #{label=>{application_controller,exit},report=>[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'}}},{kernel,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,'Elixir.MeeseeksHtml5ever.Native'}}},{kerne

[mischov]

To conclude, building on alpine works when build-base is installed and the target-feature=-crt-static > flag is used:

RUSTFLAGS='--codegen target-feature=-crt-static'

Is build-base installed?

Edit: looking at the dockerfile it looks like it is.

[yanshiyason]

Yes, that's right, it's installed.

RUN apk add --no-cache build-base git python \
        ca-certificates \
        gcc

I am now looking into meeseeks_html5ever to see if I can find any references to the native rust binary.

Edit: All I can find is the crate option

and it seems to be located here inside the _build folder here after compiling:

prod/rel/meeseeks_docker_example/lib/meeseeks_html5ever-0.12.1/priv/native/libmeeseeks_html5ever_nif.so

[yanshiyason]

running this command inside the container yielded interesting output

/app $ ./prod/rel/meeseeks_docker_example/bin/meeseeks_docker_example eval 'IO.puts(Meeseeks.parse("<div>hello</div>"))'

** (UndefinedFunctionError) function MeeseeksHtml5ever.Native.parse_html/1 is undefined (module MeeseeksHtml5ever.Native is not available)
    MeeseeksHtml5ever.Native.parse_html("<div>hello</div>")
    lib/meeseeks_html5ever.ex:12: MeeseeksHtml5ever.parse_html/1
    lib/meeseeks/parser.ex:15: Meeseeks.Parser.parse/1
    (stdlib 3.12.1) erl_eval.erl:680: :erl_eval.do_apply/6
    (stdlib 3.12.1) erl_eval.erl:888: :erl_eval.expr_list/6
    (stdlib 3.12.1) erl_eval.erl:411: :erl_eval.expr/5

It seems like the meeseeks dependencies are not installed..

Edit: Ohh, and running this command printed the reason!!

/app $ ./prod/rel/meeseeks_docker_example/bin/meeseeks_docker_example eval 'IO.puts(MeeseeksHtml5ever.Native.parse_html("<div>hello</div>"))'

04:52:57.873 [warn]  The on_load function for module Elixir.MeeseeksHtml5ever.Native returned:
{:error, {:load_failed, 'Failed to load NIF library: \'Error loading shared library libgcc_s.so.1: No such file or directory (needed by /app/prod/rel/meeseeks_docker_example/lib/meeseeks_html5ever-0.12.1/priv/native/libmeeseeks_html5ever_nif.so)\''}}

Something about libgcc_s.so ...

Edit:

I think this is related: rust-lang/rustup#2213

[mischov]

Nice catch.

[mischov]

Updating the dockerfile to

RUN apk add --no-cache openssl ncurses-libs libgcc

Solves for me.

Edit:

How frustrating, I even found that issue earlier when I searched for "rust x86_64-unknown-linux-musl".

[yanshiyason]

Oh!!! I went off on another goose chase because I didn't realize you were talking about the release image! (😆 )

I was trying to use that command in the build phase!! I will make a PR for instructions on deploying with Docker! I think it will help a lot other devs trying to adopt the library!

Edit: Yup! Fixed it for me as well!! Thank you so much @mischov !

[yanshiyason]

#99
I made a PR to document my findings! Please have a look.

[mischov]

Oh!!! I went off on another goose chase because I didn't realize you were talking about the release image!

Sorry about that, I should have been more explicit.