TT1 to TT3/MSPA 704 sequence identifier (#42)

* Validate bindings for EBU-TT-1 documents by dynamically setting superseding class on document creation * MSPA-702: Check existence of Styling and Layout elements for EBU-TT-1 * MSPA-702: EBU-TT-1 document must contain a tt:region element * MSPA-702: body/dur attribute is not allowed in EBU-TT-1 * MSPA-702: Check that smpte timeBase is acceptable in EBU-TT-1 * tt3 to ttd conversion (#41) MSPA-728 ebu-tt-3 to ebu-tt-d conversion * Validate bindings for EBU-TT-1 documents by dynamically setting superseding class on document creation * MSPA-702: EBU-TT-1 document must contain a tt:region element * rebased and fixed ebuttd test files * extracting ebutt1object base into generic ebuttdocumentbase class * Add EBU-TT 1 to EBU-TT 3 conversion Does not handle time conversion. Sets `ebuttp:sequenceIdentifier` to the value of `tt/head/metadata/documentMetadata/documentIdentifier` if present, otherwise uses "TestConverter". Resets the `conformsToStandard` to say it is EBU-TT-3 conformant. Sets the `timeBase` to `media` whether you like it or not, but doesn't do any other conversions. * Tidy out debug prints and unneeded commented code * Address review comments Also allows for a setting that specifies whether or not to use `ebuttm:documentIdentifier` element value in the input as the `ebuttp:sequenceIdentifier` attribute value in the output. Adds a test for this. * Fix some documentation Address some warnings and add a page for EBU-TT 1 to EBU-TT 3 conversion. WIP. * Rename conversion docs Should have been committed with previous commit. * Unit test EBUTT1 to EBUTT3 conversion * Fix validation error messages for unexpected attributes so it doesn't say they are missing. * Fix cloning of unknown element, and conversion of metadata * Make EBUTT1Document instantiatable by including required attributes and elements in the constructor * Add unit test cases for programmatical construction with smpte (skipped) and media timebase * Add unit test cases for from-document construction with smpte (skipped) and media timebase * Address review comments Pass pep8 wherever we can. * Remove no-longer-used error string * Rename test file Helps distinguish test EBU-TT 1 files from test EBU-TT 3 files. * Address review comment
bbc · Nov 26, 2019 · 71ac319 · 71ac319
1 parent d4b1f6e
commit 71ac319
Show file tree

Hide file tree

Showing 22 changed files with 760 additions and 37 deletions.
diff --git a/docs/source/conversion_from_ebutt.rst b/docs/source/conversion_from_ebutt.rst
@@ -0,0 +1,30 @@
+Conversion of EBU-TT Part 1 documents to EBU-TT Live documents
+==============================================================
+
+The :py:func:`ebu_tt_live.documents.converters.ebutt1_to_ebutt3` function
+creates an EBUTT3Document from an EBUTT1Document using the helper class
+:py:class:`ebu_tt_live.bindings.converters.ebutt1_ebutt3.EBUTT1EBUTT3Converter`.
+
+This class manages various possible complications, including mapping SMPTE
+timecodes into media time, and setting a sequence identifier. 
+
+Here's some documentation from the coding process that captures some of our
+internal conversation about how to map font sizes, to give an idea of the
+complexity.
+
+The problem
+-----------
+
+Convert an EBU-TT part 1 document to an EBU-TT part 3 document
+
+EBU-TT part 1 can have smpte timebase; EBU-TT part 3 cannot.
+EBU-TT part 1 must not have a sequence identifier and must not
+have a sequence number. EBU-TT part 3 documents must have both.
+
+In order to set the sequence identifier the converter can be
+configured with the desired value, or it can be set to extract the
+document identifier from the `ebuttm:documentIdentifier` element
+and use it, if it exists.
+
+TODO: how to convert smpte timebase time expressions into clock or media
+timebase time expressions.
diff --git a/docs/source/conversion.rst → docs/source/conversion_to_ebuttd.rst b/docs/source/conversion.rst → docs/source/conversion_to_ebuttd.rst
diff --git a/docs/source/deduplication.rst b/docs/source/deduplication.rst
@@ -13,7 +13,7 @@ After copying ``styling`` and ``layout`` into a ``list()`` and setting them up f
 Because ``style`` and ``region`` elements can have ``style`` attributes, these
 are deduplicated first. At this stage, it's possible that where two identical elements
 that differed only in their style references, these may end up looking the same.
- Each element is then passed through the
+Each element is then passed through the
 :py:class:`ebu_tt_live.node.deduplicator.ComparableElement` class, which processes
 each attribute, omitting the ``xml:id`` and using the
 :py:func:`ebu_tt_live.node.deduplicator.ReplaceNone` function to replace empty

diff --git a/docs/source/ebu_tt_live.node.rst b/docs/source/ebu_tt_live.node.rst
@@ -74,7 +74,7 @@ node Package
     :show-inheritance:
 
 :mod:`deduplicator` Module
-----------------------
+--------------------------
 
 .. automodule:: ebu_tt_live.node.deduplicator
     :members:

diff --git a/docs/source/inode.puml b/docs/source/inode.puml
@@ -113,6 +113,7 @@ AbstractProducerNode <|-- AbstractCombinedNode
 AbstractProducerNode <|-- SimpleProducer
 AbstractProducerNode <|-- ReSequencer
 AbstractConsumerNode <|-- SimpleConsumer
+AbstractCombinedNode <|-- Denester
 AbstractCombinedNode <|-- EBUTTDEncoder
 AbstractCombinedNode <|-- BufferDelayNode
 AbstractCombinedNode <|-- RetimingDelayNode

diff --git a/docs/source/overview.rst b/docs/source/overview.rst
@@ -19,4 +19,5 @@ The components mimic the nodes and carriage mechanisms defined in the specificat
     segmentation
     deduplication
     denesting
-    conversion
+    conversion_from_ebutt
+    conversion_to_ebuttd
diff --git a/docs/source/scripts_and_their_functions.rst b/docs/source/scripts_and_their_functions.rst
@@ -23,7 +23,7 @@ into a sequence of EBU-TT Live documents using natural language processing. Use
 The default carriage mechanism is WebSocket, so you will need to listen to
 ``ws://127.0.0.1:9000``. Conveniently, we've created an HTML page that does just
 that. After you launch the Simple Producer, open `docs/build/ui/test/index.html <../ui/test/index.html>`_
-or the `current release pre-built page <http://ebu.github.io/ebu-tt-live-toolkit/ui/test/>`_ in your
+or the `current release pre-built test page <http://ebu.github.io/ebu-tt-live-toolkit/ui/test/>`_ in your
 browser. The 'Broadcast message' field should be populated with the correct
 address (``ws://localhost:9000``). Click 'Connect' and then 'Subscribe'. You can
 also change the identifier for the sequence. The documents should appear in the
@@ -58,7 +58,7 @@ activated and the code built, start one, with a command line such as  ``ebu-run
 --admin.conf ebu_tt_live/examples/config/user_input_producer_consumer.json`` -
 this one runs a simple consumer. Then, in your browser, open
 `docs/build/ui/user_input_producer/index.html <../ui/user_input_producer/index.html>`_ or the
-`current release pre-built page <http://ebu.github.io/ebu-tt-live-toolkit/ui/user_input_producer/>`_ and click
+`current release pre-built UIP page <http://ebu.github.io/ebu-tt-live-toolkit/ui/user_input_producer/>`_ and click
 'Connect'. Select the sending mode (manual, scheduled or asynchronous). You
 should see the documents arriving in the command line window where the simple
 consumer is listening. See detailed instructions here:

diff --git a/ebu_tt_live/bindings/__init__.py b/ebu_tt_live/bindings/__init__.py
@@ -19,7 +19,8 @@
 from .validation.validator import SemanticValidator
 from ebu_tt_live.errors import SemanticValidationError, OutsideSegmentError, RegionExtendingOutsideDocumentError, InvalidRegionOriginType, InvalidRegionExtentType
 from ebu_tt_live.strings import ERR_SEMANTIC_VALIDATION_MISSING_ATTRIBUTES, \
-    ERR_SEMANTIC_VALIDATION_INVALID_ATTRIBUTES, ERR_SEMANTIC_STYLE_CIRCLE, ERR_SEMANTIC_STYLE_MISSING, \
+    ERR_SEMANTIC_VALIDATION_UNEXPECTED_ATTRIBUTES, \
+    ERR_SEMANTIC_STYLE_CIRCLE, ERR_SEMANTIC_STYLE_MISSING, \
     ERR_SEMANTIC_ELEMENT_BY_ID_MISSING, ERR_SEMANTIC_VALIDATION_EXPECTED
 from pyxb.exceptions_ import IncompleteElementContentError, MissingAttributeError, SimpleTypeValueError, \
     UnrecognizedAttributeError
@@ -581,7 +582,7 @@ def __semantic_test_smpte_attrs_absent(self):
         extra_attrs = self._semantic_attributes_present(smpte_attrs)
         if extra_attrs:
             raise SemanticValidationError(
-                ERR_SEMANTIC_VALIDATION_INVALID_ATTRIBUTES.format(
+                ERR_SEMANTIC_VALIDATION_UNEXPECTED_ATTRIBUTES.format(
                     elem_name='tt:tt',
                     attr_names=extra_attrs
                 )
@@ -607,7 +608,7 @@ def __semantic_test_time_base_clock_attrs_absent(self):
         extra_attrs = self._semantic_attributes_present(clock_attrs)
         if extra_attrs:
             raise SemanticValidationError(
-                ERR_SEMANTIC_VALIDATION_MISSING_ATTRIBUTES.format(
+                ERR_SEMANTIC_VALIDATION_UNEXPECTED_ATTRIBUTES.format(
                     elem_name='tt:tt',
                     attr_names=extra_attrs
                 )
@@ -1320,7 +1321,6 @@ def create_default_value(cls):
     def _semantic_after_subtree_copy(self, copied_instance, dataset, element_content=None):
         pass
 
-
 raw.d_styling_type._SetSupersedingClass(d_styling_type)
 
 
@@ -1434,7 +1434,10 @@ def _validateBinding_vx(self):
         if self.sequenceNumber:
             raise UnrecognizedAttributeError(type(self), 'sequenceNumber')
 
-        super(tt_type, self)._validateBinding_vx()
+        # bypass the tt_type parent's _validateBinding_vx() 
+        # because it checks for constraints that are mutually 
+        # incompatible with the constraints checked here.
+        super(raw.tt_type, self)._validateBinding_vx()
 
 
 class tt1_head_type(SemanticValidationMixin, raw.head_type):
@@ -1449,7 +1452,7 @@ def _validateBinding_vx(self):
 
 
 class tt1_layout_type(layout):
-
+    
     def _validateBinding_vx(self):
         if len(self.region) == 0:
             raise IncompleteElementContentError(self, None, None, None)