Ebuttd/Unnesting divs, spans

.travis.yml

@@ -1,44 +1,20 @@
 language: python
 
 python:
-  - '2.7'
+  - '3.7'
 
 branches:
   except:
   - gh-pages
 
-before_install:
-  - echo "Checking for java"
-  - java -version
-  - openssl aes-256-cbc -K $encrypted_f18bb5b3c3d4_key -iv $encrypted_f18bb5b3c3d4_iv -in publish-key.enc -out ~/.ssh/publish-key -d
-  - chmod u=rw,og= ~/.ssh/publish-key
-  - echo "Host github.com" >> ~/.ssh/config
-  - echo "    IdentityFile ~/.ssh/publish-key" >> ~/.ssh/config
-  - sudo apt-get -qq update
-  - sudo apt-get install graphviz
-  - git --version
-  - git remote set-url origin git@github.com:ebu/ebu-tt-live-toolkit.git
-  - git fetch origin -f gh-pages:gh-pages
-
 install:
-  - pip install pip==9.0.1
-  - make
-  - python setup.py develop
-  - pip install coveralls
-  - pip install ghp-import
+  - pip install -U pip
+  - make init
 
 script:
-  - pyxbgen --binding-root=./ebu_tt_live/bindings -m __init__ --schema-root=./ebu_tt_live/xsd/ -r -u ebutt_all.xsd
-  - python setup.py test
-  - export PR=https://api.github.com/repos/$TRAVIS_REPO_SLUG/pulls/$TRAVIS_PULL_REQUEST
-  - export BRANCH=$(if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then echo $TRAVIS_BRANCH; else echo `curl -s $PR | jq -r .head.ref`; fi)
-  - echo "TRAVIS_BRANCH=$TRAVIS_BRANCH, PR=$PR, BRANCH=$BRANCH"
-
-after_success:
-  - coveralls
+  - make test
+  - echo "Compiling documentation"
+  - make
+  - sudo apt-get -qq update
+  - sudo apt-get install graphviz
   - python setup.py build_sphinx
-  - if [ $BRANCH == "master" ]; then ghp-import -n -p -m "Update gh-pages." docs/build; fi
-
-notifications:
-  slack:
-    secure: RLs+ufYg4HcZfaTl73tTKKVja8DJ9LGk1WO+7B1qa4Fgpv0c/3uXcM+6tvDXllTlqcfLHgum0vbzvE4yapo0em0g/m75t1pEfkh5Pfrcas8IKAM9jg/xycWdYoAkKLRvNOcS4amO29FlzOHFaSSjX+V8JK8Mpm+QM2eGnqIbJokaXgBf/bQxHqGSRvYCeSisEG0f79Mn/0v5OOlbvmetFfvcQVNZDtgpA1CDbvK99F/vQT3qZ04Rh/vRnlcZyH9PEp/TEEJ0yX0NhHYEDnuBKPc86Ack/YCQpT12Ej2xTWjgoUrqv1pcr+h0ltYPXnnCyETrlbeLgNIMUsGW579MThqXekBZ7byRK2DMfjyL5+UgPSmTOIzEg7Qbc1u0r1bUuEi7nsJ5Vf65QeIAcewVASOUKmda9ag0MidTR/VoAsMtf5MzbDifSa1aBHRxZJToshNJVY9V8/lJLdwsbqsG2H+8v8ZuzyRLNiaFhNQTZQWpRXMipvbTDnnVxwpeFiLAqAwEKO9KIyGijqv2yeVvi4WtYvwzM7/1y/ycu7kTDFemKKuKmW2rzzBKJ30vo79ufMQCmn0riKkpaNxBf8R3EUEr86A9yuFpo6/Qc47i3PkeD7MszDqhEAfxl1+sQssWXOdVy6ALFb7RZ6EczpPvAjYLLO0aPa8OPaZjy3aJr+8=

docs/source/configurator.rst

@@ -125,6 +125,8 @@ Node type dependent options for [nodeN] : ::
 
    type="distributor" : No options
 
+   type="denester" : No options
+
 Output carriage type dependent options for "carriage": ::
 
    type="direct"

docs/source/denesting.rst

@@ -0,0 +1,33 @@
+Denesting of EBU-TT-Live documents
+======================================
+
+DenesterNode should be used when a EBU-TT-3 document has a div that contains
+other divs, or a span contains another span. These elements are not
+able to be nested inside each other in EBU-TT-D documents.
+
+When documents are Denested, any nested elements must be removed from
+their parent elements, while retaining attributes they would have inherited.
+To address this, the DenesterNode node processes the
+document(s) with the
+:py:func:`ebu_tt_live.node.denester.DenesterNode.recurse` and
+:py:func:`ebu_tt_live.node.denester.DenesterNode.recurse_span`
+functions. These will iterate through the file to locate the deepest
+nested element, and create a new copy of it with its content and
+expected inherited attributes. The end result is a file containing
+these newly created divs/spans in place of the nested ones.
+
+Once the new divs and spans are created, divs that are sequential in
+the list of divs and have the same attributes are combined into a single
+div. This is done by the
+:py:func:`ebu_tt_live.node.denester.DenesterNode.combine_divs`
+function to reduce the number of divs in the resulting file.
+
+In the combined divs, every p element should have the same region
+as its parent div, or be removed. The
+:py:func:`ebu_tt_live.node.denester.DenesterNode.check_p_regions`
+function iterates through the divs that have an assigned region and
+removes any p where its region does not match.
+It then removes the region attribute from any remaining p, as it will
+inhert the region of its parent div and the attribute is unnecessary.
+It will also remove any now-empty divs that exist as a result of having
+their p elements removed.

docs/source/ebu_tt_live.node.rst

@@ -80,3 +80,11 @@ node Package
     :members:
     :undoc-members:
     :show-inheritance:
+
+:mod:`denester` Module
+----------------------
+
+.. automodule:: ebu_tt_live.node.denester
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/inode.puml

@@ -70,6 +70,11 @@ class BufferDelayNode {
   +process_document()
 }
 
+class DenesterNode{
+  ..methods..
+  +process_document()
+}
+
 class RetimingDelayNode{
   ..methods..
   +process_document()

docs/source/overview.rst

@@ -18,4 +18,5 @@ The components mimic the nodes and carriage mechanisms defined in the specificat
     timing_resolution
     segmentation
     deduplication
+    denesting
     conversion

docs/source/scripts_and_their_functions.rst

@@ -107,6 +107,17 @@ attributes are duplicated.
 For the default configuration of the node, see:
 ``ebu-run --admin.conf=ebu_tt_live/examples/config/deduplicator_fs.json``
 
+Denester Node
+-------------
+This node flattens nested ``div`` and ``span`` elements such that no
+``div`` ends up containing a ``div`` and no ``span`` ends up containing
+a ``span``. It also removes any ``p`` elements that specify a ``region``
+attribute that differs from a specified region on an ancester element.
+
+If nested ``div`` or ``p`` elements might be present in a document, the
+Denester node should be used to flatten them before passing them to the
+EBU-TT-D Encoder, because EBU-TT-D does not permit such nested elements.
+
 Retiming Delay Node
 -------------------
 This script modifies the times within each Document and issues them without

ebu_tt_live/bindings/__init__.py

@@ -100,6 +100,24 @@ class style_type(StyledElementMixin, IDMixin, SizingValidationMixin, SemanticVal
     }
     _default_attrs = None
 
+    def check_equal(self, other):
+        return (self.backgroundColor == other.backgroundColor and
+            self.padding == other.padding and
+            self.unicodeBidi == other.unicodeBidi and
+            self.color == other.color and
+            self.direction == other.direction and
+            self.fontFamily == other.fontFamily and
+            self.fontStyle == other.fontStyle and
+            self.fontWeight == other.fontWeight and
+            self.linePadding == other.linePadding and
+            self.multiRowAlign == other.multiRowAlign and
+            self.textAlign == other.textAlign and
+            self.textDecoration == other.textDecoration and
+            self.fontSize == other.fontSize and
+            self.lineHeight == other.lineHeight and
+            self.wrapOption == other.wrapOption)
+
+
     def __repr__(self):
         return '<style ID: {id} at {addr}>'.format(
             id=self.id,
@@ -620,6 +638,7 @@ def _semantic_before_traversal(self, dataset, element_content=None, parent_bindi
         # attributes.
         dataset['timing_begin_stack'] = []
         dataset['timing_end_stack'] = []
+        dataset['div_stack'] = []
         dataset['timing_syncbase'] = timedelta()
         dataset['tt_element'] = self
         dataset['styles_stack'] = []
@@ -886,6 +905,9 @@ def __copy__(self):
         )
         return copied_div
 
+    def merge(self, elem):
+        return self
+
     def _semantic_before_traversal(self, dataset, element_content=None, parent_binding=None):
         self._semantic_register_id(dataset=dataset)
         self._semantic_timebase_validation(
@@ -922,6 +944,8 @@ def _semantic_after_subtree_copy(self, copied_instance, dataset, element_content
             copied_instance=copied_instance, dataset=dataset, element_content=element_content)
         self._semantic_copy_verify_referenced_styles(dataset=dataset)
         self._semantic_copy_verify_referenced_region(dataset=dataset)
+
+
 
 
 raw.div_type._SetSupersedingClass(div_type)

ebu_tt_live/bindings/_ebuttdt.py

@@ -326,8 +326,12 @@ def as_timedelta(cls, instance):
         :param instance:
         :return:
         """
-        hours, minutes, seconds, milliseconds = [cls._int_or_none(x) for x in cls._groups_regex.match(instance).groups()]
-        return timedelta(hours=hours, minutes=minutes, seconds=seconds, milliseconds=milliseconds)
+        hours_str, minutes_str, seconds_str, seconds_fraction_str = [x for x in cls._groups_regex.match(instance).groups()]
+        milliseconds = seconds_fraction_str and cls._int_or_none('{:0<3}'.format(seconds_fraction_str)[:3]) or 0
+        return timedelta(hours=cls._int_or_none(hours_str), 
+            minutes=cls._int_or_none(minutes_str), 
+            seconds=cls._int_or_none(seconds_str), 
+            milliseconds=milliseconds)
 
     @classmethod
     def from_timedelta(cls, instance):
@@ -374,8 +378,13 @@ def as_timedelta(cls, instance):
         :param instance:
         :return:
         """
-        hours, minutes, seconds, milliseconds = [cls._int_or_none(x) for x in cls._groups_regex.match(instance).groups()]
-        return timedelta(hours=hours, minutes=minutes, seconds=seconds, milliseconds=milliseconds)
+        hours_str, minutes_str, seconds_str, seconds_fraction_str = [x for x in cls._groups_regex.match(instance).groups()]
+        milliseconds = seconds_fraction_str and cls._int_or_none('{:0<3}'.format(seconds_fraction_str)[:3]) or 0
+        return timedelta(hours=cls._int_or_none(hours_str), 
+            minutes=cls._int_or_none(minutes_str), 
+            seconds=cls._int_or_none(seconds_str), 
+            milliseconds=milliseconds)
+
 
     @classmethod
     def from_timedelta(cls, instance):

ebu_tt_live/bindings/converters/ebutt3_ebuttd.py

@@ -294,6 +294,8 @@ def convert_body(self, body_in, dataset):
         return new_elem
 
     def convert_div(self, div_in, dataset):
+        if len(div_in.orderedContent()) == 0:
+            return None
         new_elem = d_div_type(
             *self.convert_children(div_in, dataset),
             id=div_in.id,

ebu_tt_live/bindings/test/test_times.py

@@ -0,0 +1,72 @@
+
+from unittest import TestCase
+from unittest import skip, SkipTest
+from datetime import timedelta
+from ebu_tt_live.bindings import ebuttdt
+from pyxb import SimpleTypeValueError, SimpleFacetValueError
+
+class TestTimecountTimingType(TestCase):
+
+    _test_cases = {
+        '1.3s': timedelta(seconds = 1, milliseconds = 300),
+        '1.03s': timedelta(seconds = 1, milliseconds = 30),
+        '1.30s': timedelta(seconds = 1, milliseconds = 300),
+        '1s': timedelta(seconds = 1),
+        '1.3m': timedelta(minutes = 1, seconds = 18),
+        '1.03m': timedelta(minutes = 1, seconds = 1, milliseconds = 800),
+        '1.30m': timedelta(minutes = 1, seconds = 18),
+        '1m': timedelta(minutes = 1),
+        '1.3h': timedelta(hours = 1, minutes = 18),
+        '1.03h': timedelta(hours = 1, minutes = 1, seconds = 48),
+        '1.30h': timedelta(hours = 1, minutes = 18),
+        '1h': timedelta(hours = 1),
+        '1.3ms': timedelta(milliseconds = 1, microseconds = 300),
+        '1.03ms': timedelta(milliseconds = 1, microseconds = 30),
+        '1.30ms': timedelta(milliseconds = 1, microseconds = 300),
+        '1ms': timedelta(milliseconds = 1),
+    }
+
+    _type_class = ebuttdt.TimecountTimingType
+
+    def test_as_timedelta(self):
+        for t, td in self._test_cases.items() :
+            test_instance = self._type_class(t)
+            self.assertEqual(test_instance.timedelta, td)
+            assert test_instance == t
+
+
+
+class TestFullClockTimingType(TestCase):
+
+    _test_cases = {
+        '111:22:33': timedelta(hours = 111, minutes = 22, seconds = 33),
+        '111:22:33.4': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
+        '111:22:33.04': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 40),
+        '111:22:33.40': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
+    }
+
+    _type_class = ebuttdt.FullClockTimingType
+
+    def test_as_timedelta(self):
+        for t, td in self._test_cases.items() :
+            test_instance = self._type_class(t)
+            self.assertEqual(test_instance.timedelta, td)
+            assert test_instance == t
+
+
+class TestLimitedClockTimingType(TestCase):
+
+    _test_cases = {
+        '11:22:33': timedelta(hours = 11, minutes = 22, seconds = 33),
+        '11:22:33.4': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
+        '11:22:33.04': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 40),
+        '11:22:33.40': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
+    }
+
+    _type_class = ebuttdt.LimitedClockTimingType
+
+    def test_as_timedelta(self):
+        for t, td in self._test_cases.items() :
+            test_instance = self._type_class(t)
+            self.assertEqual(test_instance.timedelta, td)
+            assert test_instance == t

ebu_tt_live/bindings/validation/presentation.py

@@ -32,6 +32,7 @@ class StyledElementMixin(object):
     """
     _compatible_style_type = None
     _referenced_styles = None
+    _referenced_divs = None
     _inherited_styles = None
     _region_styles = None
     _validated_styles = None

ebu_tt_live/bindings/validation/timing.py

@@ -109,7 +109,12 @@ def _post_calculate_begin(self, children):
         earliest_child_computed_begin = min(children_computed_begin_times)
         if earliest_child_computed_begin > self._computed_begin_time:
             # Adjustment scenario
-            self._computed_begin_time = earliest_child_computed_begin
+            # If no parent element specified a begin time, then we have found
+            # a case for the "earliest specified computed begin time" as per the
+            # specification and we can adjust the begin time to match the
+            # children's begin time.
+            if len(self._semantic_dataset['timing_begin_stack']) == 0:
+                self._computed_begin_time = earliest_child_computed_begin
 
     def _semantic_preprocess_timing(self, dataset, element_content):
         """
@@ -395,6 +400,35 @@ def _post_pop_end(self):
 
         return end_timedelta
 
+    def _post_calculate_begin(self, children):
+        """
+        The computed begin time shall be moved down to match that of the earliest child begin time in case the container
+        does not specify a begin time itself. NOTE: This does not modify the syncbase.
+
+        :param children:
+        :return:
+        """
+
+        if not children:
+            if 'availability_time' in self._semantic_dataset:
+                if self.begin is None:
+                    self._computed_begin_time = self._semantic_dataset['availability_time']
+                if self.end is None and self.dur is not None:
+                    self._computed_end_time = self._computed_begin_time + self._dur_timedelta
+            return
+
+        children_computed_begin_times = [item.computed_begin_time for item in children]
+
+        earliest_child_computed_begin = min(children_computed_begin_times)
+        if 'availability_time' in self._semantic_dataset:
+            earliest_child_computed_begin = max(earliest_child_computed_begin, self._semantic_dataset['availability_time'])
+
+        if earliest_child_computed_begin > self._computed_begin_time:
+            # Adjustment scenario
+            self._computed_begin_time = earliest_child_computed_begin
+            if self.dur is not None and self.begin is None:
+                self._computed_end_time = self._computed_begin_time + self._dur_timedelta
+
     def _semantic_timebase_validation(self, dataset, element_content):
 
         super(BodyTimingValidationMixin, self)._semantic_timebase_validation(dataset, element_content)

ebu_tt_live/config/node.py

@@ -192,6 +192,20 @@ def __init__(self, config, local_config):
         self._create_input(config)
         self._create_output(config)
 
+class Denester(ConsumerMixin, ProducerMixin, NodeBase):
+    required_config = Namespace()
+
+    def _create_component(self, config):
+        self.component = processing_node.DenesterNode(
+            node_id=self.config.id
+        )
+
+    def __init__(self, config, local_config):
+        super(RetimingDelay, self).__init__(config, local_config)
+        self._create_component(config)
+        self._create_input(config)
+        self._create_output(config)
+
 
 class SimpleProducer(ProducerMixin, NodeBase):
     required_config = Namespace()