Children groups #461

varsill · 2022-10-24T15:35:05Z

This PR introduces children groups - a way to aggregate children so that all the children being part of a group can be referenced at once.

…d of 'links' and 'children' fiedls. Update the tests accordingly Repair a bug with a trivial pipeline. Improve typespecs Rename the LinkParser into the StructureParser Add information about the children group in Membrane Logger's prints Change the Membrane.TestingPipeline options specification so that to have the `structure` field Improve functions naming in the StructureParser. Change the ParentSpec.spawn/2 into ParentSpec.spawn_child/2 to avois mistakes with Kernel.spawn Add conditional link building Change the return type of the ParenSpec.spawn_child/2 so that to unify the parent spec's structure field type Fix a bug with a single element children list in link builder. Add typespecs for ignore_unless/2 and end_ignore/1 Rename to/3 into to_new/3 improve typespecs in parent spec Fix a bug with nested conditional linking. Add tests checking if the nested conditional linking works properly Add tests checking the behaviour of to_new/3 Revoke to/3 function Refactor the `should_ignore` mechanism in parentspec Add link_new/2 and tests checking it performance. Put the additional modules used in child_removal_test in separate files Revert children groups implementation Remove conditional linking Rename ParentSpec into ChildrenSpec Remove link, link_new, to, to_new, spawn_child functions. Add child/2, child/3, get_child/1 and get_child2 Add handling of ignore_duplicates option in child/3 and child/4. Revoke tests checking that functionality. Add is_bin_name?/1, is_element_name?/1 and is_child_name?/1 guards. Add default child options Update ChildrenSpec moduledoc. Rename `to_bin_output` and `link_bin_input` into `bin_output` and `bin_input`, respectively. Rename `ignore_duplicates` option into `get_if_exists`. Specify proper typespec for child/3 Improve moduledoc Improve function docs in ChildrenSpec Improve moduledoc in childrenspec add test for element play fixes for CR Co-authored-by: Bartosz Błaszków <bartosz.blaszkow@swmansion.com> handle_play -> handle_playing play_request? -> playing_requested? Play -> Playing Apply suggestions from code review Co-authored-by: Bartosz Błaszków <bartosz.blaszkow@swmansion.com> refactor component path Improved ChildrenSupervisor, ResourceGuard and Observability (#447) * supervisors refactor * resource guard * add utility supervisor to context * refactor element tests not to use State.new * Update lib/membrane/resource_guard.ex Co-authored-by: Bartosz Błaszków <bartosz.blaszkow@swmansion.com> * run resource cleanup in a separate process * refactor observability setup * improve children supervisor API * add docs and test for utility supervisor and resource guard * add timeout to resource * use a boring function name instead of go_brrr * unlink cleanup task in resource guard before killing it * Add list of available callbacks in each component (#449) * Add an aggregated list of all callbacks that can be implemented by `Filter`, `Endpoint`, `Source` and `Sink` element behaviours. Co-authored-by: Mateusz Front <mateusz.front@swmansion.com> * add docs for Observability.setup Co-authored-by: Bartosz Błaszków <bartosz.blaszkow@swmansion.com> Co-authored-by: Łukasz Kita <lukasz.kita0@gmail.com> Make `dont_spawn_if_already_exists` one of keywords for options in child_spec. Update the tests with new syntax for spawning children

… concerning the children spawning. Repair a bug with ChildLifeController.handle_child_death typespec. Repair typespec of Pipeline.State struct

…d/3` function set the structure builder status to :done

…dd an ability to remove all the children from given children group

…ormation to error print in the structure parser

This reverts commit 9da1456.

…truct. Add an ability to remove all the children from given children group" This reverts commit 1a9a191.

* Validate link in ChildrenSpec * Add tests for links validation

* Match caps in bin's pads * Caps specs as Elixir patterns

* Rename def_*_pad option from :caps to :accepted_format * Rename :caps to :stream_format

…membrane_core into improved_spec

…ild options. Put the typespecs in order

…default children_spec options

…ildren_spec.ex. Improve spelling in children_spec.ex documentation. Refactor the handle_spec code

…ated with child()

…p_member__. Improve the children_spec.exs docs

* Add ability to spawn anonymous children

…ove theirs docs

…he v0.11 upgrading guide

…k/membrane_core into children_groups

…dule.

mat-hek · 2022-12-01T15:51:20Z

guides/upgrading/v0.11.md

 +   child(:filter, SomeFilter),
 +   get_child(:source) |> child(:another_filter, SomeFilter) |> get_child(:filter) |> child(:sink, SomeSink)
  ]
-+ spec = {structure, children_group_id: :first_group, crash_group_mode: :temporary, node: another_node}
+ spec = {structure, group: :first_group, crash_group_mode: :temporary, node: another_node}
 - {{:ok, spec: spec}, %{}}
 + {[spec: spec], %{}}


this won't be in 0.11 ;)

I have removed that a few commits before, I don't know how has it appeared there and why was it still visible to you :D

mat-hek · 2022-12-05T09:25:18Z

lib/membrane/core/parent/child_life_controller.ex

-           node: node() | nil,
-           log_metadata: Keyword.t()
-         }
+  @opaque parsed_children_spec_options_t :: %{


Why not just @type?

Well, this type is used externally only by StartupUtil.scheck_if_children_names_and_children_groups_ids_are_unique - and that function does not manipulate the internals of the type. In fact, StartupUtil.scheck_if_children_names_and_children_groups_ids_are_unique needs to use of that type since there is a need to emphasize, that the input argument is in the canonical form - perhaps we should add a children_spec_canonical_form_t type than, and make parsed_children_spec_options_t private? WDYT?

Good idea ;)

mat-hek · 2022-12-05T09:30:15Z

lib/membrane/core/parent/child_life_controller.ex

+      # child name created with child(...)
+      {:__membrane_just_child_name__, child_name} -> Membrane.Child.ref(child_name, group: group)
+      # child name created with get_child(...), bin_input() and bin_output()
+      ref -> ref


If all child names were wrapped, we could match them like this:

Suggested change

# child name created with child(...)

{:__membrane_just_child_name__, child_name} -> Membrane.Child.ref(child_name, group: group)

# child name created with get_child(...), bin_input() and bin_output()

ref -> ref

# child name created with child(...)

{:child_name, child_name} -> Membrane.Child.ref(child_name, group: group)

# child name created with get_child(...), bin_input() and bin_output()

{:child_ref, ref} -> ref

It looks cleaner IMO

mat-hek · 2022-12-05T09:45:23Z

lib/membrane/child.ex


  @type name_t :: Element.name_t() | Bin.name_t()
+  @type group_t() :: any()
+  @type ref_t :: {:__membrane_children_group_member__, group_t(), name_t()} | name_t()


I've realized that this is going to be printed out in each log... Maybe we could have

Suggested change

@type ref_t :: {:__membrane_children_group_member__, group_t(), name_t()} | name_t()

@type ref_t :: {Membrane.Child, group_t(), name_t()} | name_t()

like {Membrane.Pad, name, id}?

good idea, done

mat-hek · 2022-12-05T12:23:19Z

lib/membrane/core/parent/child_life_controller.ex

    [{spec, options}]
  end

+  defp equip_spec_with_children_refs(specs, group) do


actually these are structure builders

indeed, done

mat-hek · 2022-12-05T12:24:20Z

lib/membrane/core/parent/child_life_controller.ex

    [{spec, options}]
  end

+  defp equip_spec_with_children_refs(specs, group) do
+    specs =
+      Enum.map(specs, fn spec ->


maybe update both children and links in one pass?

well, sure ;) Done

…k/membrane_core into children_groups

* Get rid of using structure term in context of children specs. Update the tests accordingly. Removed unused support element pipeline

varsill added 9 commits October 21, 2022 16:44

Remove ChildrenSpec.link_linear/1. Update tests. Update documentation…

b525ae6

… concerning the children spawning. Repair a bug with ChildLifeController.handle_child_death typespec. Repair typespec of Pipeline.State struct

Update changelog

18bf76d

Put ChildrenSpec's typespecs in order

93666d1

Rename LinkBuilder into StructureBuilder

9f5a34e

Improve error prints when ChildrenSpec is used improperly. Make `chil…

d9c7508

…d/3` function set the structure builder status to :done

Remove a reference to not existing link_linear function in moduledoc

39434c0

Add children groups. Make it configurable with ChildrenSpec struct. A…

1a9a191

…dd an ability to remove all the children from given children group

Improve formatting of the children group id print in logs

9da1456

varsill marked this pull request as ready for review October 24, 2022 15:36

varsill requested a review from mat-hek as a code owner October 24, 2022 15:36

varsill and others added 19 commits October 26, 2022 11:41

Redesign LinkBuilder

7fa3e95

Raise meaningful exception when the link buillder fails. Add more inf…

cb10baf

…ormation to error print in the structure parser

Add an error raising when bin_input() is used improperly

7edda41

Revert "Improve formatting of the children group id print in logs"

0eb93c9

This reverts commit 9da1456.

Revert "Add children groups. Make it configurable with ChildrenSpec s…

2cac44d

…truct. Add an ability to remove all the children from given children group" This reverts commit 1a9a191.

Forbid link override (#464)

66b9444

* Validate link in ChildrenSpec * Add tests for links validation

Improve caps (#463)

79450ef

* Match caps in bin's pads * Caps specs as Elixir patterns

Rename caps (#468)

43dc85e

* Rename def_*_pad option from :caps to :accepted_format * Rename :caps to :stream_format

Remove ChildrenSpec struct. Update tests

317ba73

Merge branch 'improved_spec' of https://github.com/membraneframework/…

4c4d240

…membrane_core into improved_spec

Add support for nested childrenspec

1ea4ed6

Remove distinguishment between children_spec_extension options and ch…

25ac2f5

…ild options. Put the typespecs in order

Implement reviewers suggestions

4eaeb46

Implementing further reviewers suggestions

43b6587

Make sure that the nested spec options are properly set

4fad2c6

Move the make_canonical function to children_spec.exs

83c48ca

Update the documentation

67cfd94

Move the make_canonical back to child_life_controller.ex, along with …

a170aa0

…default children_spec options

Rename childrenspec_options into children_spec_options

1d673be

varsill added 11 commits November 29, 2022 15:27

Add membrane reserved prefix for members of children groups.

d395646

Add proper links resolving in handle_spec

0260df8

Remove elixir specifier from code snippets in documentation of the ch…

6659cbf

…ildren_spec.ex. Improve spelling in children_spec.ex documentation. Refactor the handle_spec code

Add distinguishement between link created in get_child() and link cre…

ad9ed17

…ated with child()

Rename membrane children group specifier to :__membrane_children_grou…

0d306c3

…p_member__. Improve the children_spec.exs docs

Add ability to spawn anonymous children (#482)

67a097a

* Add ability to spawn anonymous children

Improve typespecs for the Membrane.ChildrenSpec.child functions. Impr…

eb1b0be

…ove theirs docs

Update changelog

f323da6

Merge branch 'master' into children_groups

d1585b8

Remove misleading information about the crash_group API change from t…

9c894d1

…he v0.11 upgrading guide

Merge branch 'children_groups' of https://github.com/membraneframewor…

4e3633a

…k/membrane_core into children_groups

varsill requested a review from mat-hek December 1, 2022 11:36

varsill added 6 commits December 2, 2022 14:07

Add Child.ref() function. Move verification of the name into Child mo…

9fa3b08

…dule.

Refacto child_life_controller

9d7fab9

Improve the docs

0166d53

Remove unecessary newlines

ed2871d

Improve the remove_children typespec

3918938

Merge branch 'master' into children_groups

497cda8

mat-hek requested changes Dec 5, 2022

View reviewed changes

varsill added 4 commits December 5, 2022 15:21

Implement reviewers suggestions

258162b

Merge branch 'children_groups' of https://github.com/membraneframewor…

d49abd5

…k/membrane_core into children_groups

Merge branch 'master' into children_groups

0b80e14

Improve typespecs

393d130

varsill requested a review from mat-hek December 5, 2022 15:45

mat-hek approved these changes Dec 6, 2022

View reviewed changes

varsill added 3 commits December 9, 2022 09:10

Merge branch 'master' into children_groups

36e52b5

[MC-107] Rename structure into specification (#489)

25745d7

* Get rid of using structure term in context of children specs. Update the tests accordingly. Removed unused support element pipeline

Add missing backticks

4c04c45

varsill merged commit f9f9381 into master Dec 9, 2022

varsill deleted the children_groups branch December 9, 2022 08:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Children groups #461

Children groups #461

varsill commented Oct 24, 2022

mat-hek Dec 1, 2022

varsill Dec 5, 2022 •

edited

Loading

mat-hek Dec 5, 2022

varsill Dec 5, 2022 •

edited

Loading

mat-hek Dec 5, 2022

varsill Dec 5, 2022

mat-hek Dec 5, 2022

varsill Dec 5, 2022

mat-hek Dec 5, 2022

varsill Dec 5, 2022

mat-hek Dec 5, 2022

varsill Dec 5, 2022

mat-hek Dec 5, 2022

varsill Dec 5, 2022

	@type ref_t :: {:__membrane_children_group_member__, group_t(), name_t()} \| name_t()
	@type ref_t :: {Membrane.Child, group_t(), name_t()} \| name_t()

Children groups #461

Children groups #461

Conversation

varsill commented Oct 24, 2022

Choose a reason for hiding this comment

varsill Dec 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

varsill Dec 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

varsill Dec 5, 2022 •

edited

Loading

varsill Dec 5, 2022 •

edited

Loading