DistributedAtomic.Worker implements sub_get as a call

[philipgiuliani]

We have a distributed environment in which we are starting Pads on different nodes. We experienced this issue multiple times in this cluster. Not sure how this issue didn't come up yet, as it seems to be in the codebase for a whole year.

10:32:46.327 [error] GenServer #PID<50061.5059.0> terminating
** (FunctionClauseError) no function clause matching in Membrane.Core.Element.AtomicDemand.DistributedAtomic.Worker.handle_cast/2
(membrane_core 1.1.0) lib/membrane/core/element/atomic_demand/distributed_atomic/worker.ex:46: Membrane.Core.Element.AtomicDemand.DistributedAtomic.Worker.handle_cast({:sub_get, #Reference<50061.4261161875.3378380802.63969>, 150}, nil)
(stdlib 5.1.1) gen_server.erl:1103: :gen_server.try_handle_cast/3
(stdlib 5.1.1) gen_server.erl:1165: :gen_server.handle_msg/6
(stdlib 5.1.1) proc_lib.erl:251: :proc_lib.wake_up/3
Last message: {:"$gen_cast", {:sub_get, #Reference<50061.4261161875.3378380802.63969>, 150}}
10:32:46.682 [error] GenServer #PID<0.2944.0> terminating
** (stop) exited in: GenServer.call(#PID<50061.5059.0>, {:get, #Reference<50061.4261161875.3378380802.63969>}, 5000)
** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
(elixir 1.16.1) lib/gen_server.ex:1114: GenServer.call/3
(membrane_core 1.1.0) lib/membrane/core/stalker.ex:496: anonymous fn/1 in Membrane.Core.Stalker.scrape_metrics/1
(elixir 1.16.1) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2
(elixir 1.16.1) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2
(membrane_core 1.1.0) lib/membrane/core/stalker.ex:372: Membrane.Core.Stalker.do_handle_info/2
(membrane_core 1.1.0) lib/membrane/core/stalker.ex:366: Membrane.Core.Stalker.handle_info/2
(stdlib 5.1.1) gen_server.erl:1077: :gen_server.try_handle_info/3
(stdlib 5.1.1) gen_server.erl:1165: :gen_server.handle_msg/6
Last message: :scrape_metrics

[dmorn]

We noticed also that now the issue of the latest hotfix (2a545e3) is still present apparently:

s    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.880 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:atomic_demand, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", ":voiceover"], :output}, {:__membrane_stalker_function_metric__, #Function<4.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:292: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.880 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:total_buffers, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", "{:transcript, :tee}"], :input}, {:__membrane_stalker_function_metric__, #Function<3.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:286: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:188: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.881 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:atomic_demand, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", "{:transcript, :tee}"], :input}, {:__membrane_stalker_function_metric__, #Function<4.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:292: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:188: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.881 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:auto_demand_size, ["<0.2788.0>/", "{:client, %{server: #PID<0.2788.0>, ref: #Reference<0.2642032877.967311361.96835>, translate_language_code: \"DE\", enable_voiceover: true, input_language_code: \"en\"}}/", "{:transcript, :tee}"], :input}, {:__membrane_stalker_function_metric__, #Function<8.87660500/0 in Membrane.Core.Element.PadController.merge_pad_mode_data/4>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:422: anonymous fn/2 in Membrane.Core.Element.PadController.merge_pad_mode_data/4
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:421: Membrane.Core.Element.PadController.merge_pad_mode_data/4
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:318: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:188: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4


10:47:38.882 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:total_buffers, ["<0.2788.0>/", ":core/", "{:transcript, :tee}"], {Membrane.Pad, :output, #Reference<0.2642032877.967311362.66038>}}, {:__membrane_stalker_function_metric__, #Function<3.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:286: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:47:38.882 [warning] Failed to insert a metric into the observability ETS.
Error: ** (ArgumentError) errors were found at the given arguments:

  * 1st argument: the table identifier does not refer to an existing ETS table

    (stdlib 5.1.1) :ets.insert(#Reference<0.2642032877.967180289.249079>, {{:atomic_demand, ["<0.2788.0>/", ":core/", "{:transcript, :tee}"], {Membrane.Pad, :output, #Reference<0.2642032877.967311362.66038>}}, {:__membrane_stalker_function_metric__, #Function<4.87660500/0 in Membrane.Core.Element.PadController.init_pad_data/8>}})
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:292: Membrane.Core.Element.PadController.init_pad_data/8
    (membrane_core 1.1.0) lib/membrane/core/element/pad_controller.ex:100: Membrane.Core.Element.PadController.do_handle_link/6
    (membrane_core 1.1.0) lib/membrane/core/element.ex:187: Membrane.Core.Element.do_handle_call/3
    (membrane_core 1.1.0) lib/membrane/core/element.ex:174: Membrane.Core.Element.handle_call/3
    (stdlib 5.1.1) gen_server.erl:1113: :gen_server.try_handle_call/4
    (stdlib 5.1.1) gen_server.erl:1142: :gen_server.handle_msg/6
    (stdlib 5.1.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3


10:50:04.301 [error] retry: got exception, will retry in 1000ms, 3 attempts left
10:50:04.301 [error] ** (Req.TransportError) socket closed

[FelonEkonom]

@philipgiuliani could you please update your branch with the master branch?
I will release this pull request in around 10 days in version 1.1.1, because currently I am on the vacations.
Thank you very much for your bug fix 👍 👍 👍

[philipgiuliani]

I've rebased the branch onto master.

[FelonEkonom]

@philipgiuliani your fix has been released in v1.1.1