Add Cityscapes format

CHANGELOG.md

@@ -11,11 +11,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Support for escaping in attribiute values in LabelMe format (<https://github.com/openvinotoolkit/datumaro/issues/49>)
 - Support for Segmentation Splitting (<https://github.com/openvinotoolkit/datumaro/pull/223>)
 - Support for CIFAR-10/100 dataset format (<https://github.com/openvinotoolkit/datumaro/pull/225>, <https://github.com/openvinotoolkit/datumaro/pull/243>)
-- Support COCO panoptic and stuff format (<https://github.com/openvinotoolkit/datumaro/pull/210>)
+- Support for COCO panoptic and stuff format (<https://github.com/openvinotoolkit/datumaro/pull/210>)
 - Documentation file and integration tests for Pascal VOC format (<https://github.com/openvinotoolkit/datumaro/pull/228>)
 - Support for MNIST and MNIST in CSV dataset formats (<https://github.com/openvinotoolkit/datumaro/pull/234>)
 - Documentation file for COCO format (<https://github.com/openvinotoolkit/datumaro/pull/241>)
 - Documentation file and integration tests for YOLO format (<https://github.com/openvinotoolkit/datumaro/pull/246>)
+- Support for Cityscapes dataset format (<https://github.com/openvinotoolkit/datumaro/pull/249>)
 
 ### Changed
 - LabelMe format saves dataset items with their relative paths by subsets without changing names (<https://github.com/openvinotoolkit/datumaro/pull/200>)

README.md

@@ -137,6 +137,7 @@ CVAT annotations                             ---> Publication, statistics etc.
   - [MNIST](http://yann.lecun.com/exdb/mnist/) (`classification`)
   - [MNIST in CSV](https://pjreddie.com/projects/mnist-in-csv/) (`classification`)
   - [CamVid](http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/)
+  - [Cityscapes](https://www.cityscapes-dataset.com/)
   - [CVAT](https://github.com/opencv/cvat/blob/develop/cvat/apps/documentation/xml_format.md)
   - [LabelMe](http://labelme.csail.mit.edu/Release3.0)
   - [ICDAR13/15](https://rrc.cvc.uab.es/?ch=2) (`word_recognition`, `text_localization`, `text_segmentation`)

datumaro/components/extractor.py

@@ -236,7 +236,7 @@ def __eq__(self, other):
 class CompiledMask:
     @staticmethod
     def from_instance_masks(instance_masks,
-            instance_ids=None, instance_labels=None):
+            instance_ids=None, instance_labels=None, dtype=None):
         from datumaro.util.mask_tools import make_index_mask
 
         if instance_ids is not None:
@@ -266,7 +266,7 @@ def from_instance_masks(instance_masks,
         m, idx, instance_id, class_id = next(it)
         if not class_id:
             idx = 0
-        index_mask = make_index_mask(m, idx)
+        index_mask = make_index_mask(m, idx, dtype=dtype)
         instance_map.append(instance_id)
         class_map.append(class_id)
 
@@ -282,8 +282,8 @@ def from_instance_masks(instance_masks,
         else:
             merged_instance_mask = np.array(instance_map,
                 dtype=np.min_scalar_type(instance_map))[index_mask]
-        merged_class_mask = np.array(class_map,
-            dtype=np.min_scalar_type(class_map))[index_mask]
+        dtype_mask = dtype if dtype else np.min_scalar_type(class_map)
+        merged_class_mask = np.array(class_map, dtype=dtype_mask)[index_mask]
 
         return __class__(class_mask=merged_class_mask,
             instance_mask=merged_instance_mask)
@@ -673,7 +673,21 @@ def __call__(self, path, **extra_params):
     @classmethod
     def _find_sources_recursive(cls, path, ext, extractor_name,
             filename='*', dirname='', file_filter=None, max_depth=3):
-        if path.endswith(ext) and osp.isfile(path):
+
+        def check_for_sources(path, ext, dirname):
+            if path.endswith(ext) and osp.isfile(path):
+                return True
+            if not ext and osp.isdir(path) and dirname:
+                dirname = osp.normpath(dirname)
+                path = osp.abspath(path)
+                if dirname in path:
+                    subpath = path.split(dirname)[0]
+                    if osp.isdir(subpath) and \
+                        osp.isdir(osp.join(subpath, dirname)):
+                        return True
+            return False
+
+        if check_for_sources(path, ext, dirname):
             sources = [{'url': path, 'format': extractor_name}]
         else:
             sources = []