Develop JsonSectionPageMapper in Rust API

[vinnamkim]

Summary

• Ticket no. 127135 and 127136.
• Develop JsonSectionPageMapper to construct page maps for top-level sections in a given JSON file.
• Enhance DatumaroImporter.detect()'s performance by replacing JSON file parsing logic with the JsonSectionPageMapper.

How to test

Our existing test will validate its functionality. For the performance comparison, please see the following.

• Before

from datumaro.rust_api import JsonSectionPageMapper
from time import time
import datumaro as dm

start = time()
format = dm.Dataset.detect("ws_test/coco/datumaro")
dt = 1000.0 * (time() - start)
print(f"Duration for detecting Datumaro data format: {dt:.1f}ms, format={format}")

Duration for detecting Datumaro data format: 25784.5ms, format=datumaro

• After

from datumaro.rust_api import JsonSectionPageMapper
from time import time
import datumaro as dm

start = time()
format = dm.Dataset.detect("ws_test/coco/datumaro")
dt = 1000.0 * (time() - start)
print(f"Duration for detecting Datumaro data format: {dt:.1f}ms, format={format}")

Duration for detecting Datumaro data format: 17234.7ms, format=datumaro

It saves ~7 secs.

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added the description of my changes into CHANGELOG.​
• I have updated the documentation accordingly

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

[vinnamkim]

Unless we merge this PR, we cannot reach our first objective, memory bounded detect(). Please see the following memory profiling result with this PR:

This is because there are many data formats which read the whole JSON file at once during their detect(). This is one of them:

datumaro/src/datumaro/plugins/data_formats/segment_anything/importer.py

Lines 17 to 29 in 4faaae5

	@classmethod
	def detect(
	cls,
	context: FormatDetectionContext,
	) -> Optional[FormatDetectionConfidence]:
	# test maximum 10 annotation files only
	ctr = 0
	for file in context.require_files_iter("*.json"):
	ctr += 1
	with context.probe_text_file(
	file, "Annotation format is not Segmentat-Anything format", is_binary_file=True
	) as f:
	anno = parse_json(f.read())

I'll fix the other data formats as well in the next PR.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (9b254ce) 80.10% compared to head (b2263ea) 80.44%.
Report is 1 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1224      +/-   ##
===========================================
+ Coverage    80.10%   80.44%   +0.34%     
===========================================
  Files          269      269              
  Lines        29915    29915              
  Branches      5850     5850              
===========================================
+ Hits         23962    24066     +104     
+ Misses        4617     4487     -130     
- Partials      1336     1362      +26     

Flag	Coverage Δ
macos-11_Python-3.8	79.10% <100.00%> (+0.34%)	⬆️
ubuntu-20.04_Python-3.8	80.44% <100.00%> (+0.34%)	⬆️
windows-2022_Python-3.8	80.41% <100.00%> (+0.34%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wonjuleee]

Here, I personally love to get access to context.root_path directly, but this is recommended to avoid during detection.

datumaro/src/datumaro/components/format_detection.py

Lines 178 to 179 in 4faaae5

	Detectors should avoid using this property in favor of specific
	requirement methods.

Do you have any objection to use this during detection

[vinnamkim]

That guide is because the methods provided by context are using self._root_path internally. For example,

datumaro/src/datumaro/components/format_detection.py

Lines 354 to 357 in 4faaae5

	try:
	if is_binary_file:
	with open(osp.join(self._root_path, path), "rb") as f:
	yield f

However, JsonSectionPageMapper Rust API has no information for root_path. There is no choice but to inject the full path osp.join(context.root_path, annot_file) to access the file.

[wonjuleee]

Agree, how about other formats with pure Python codes? I just want to hear your opinion.
Frankly speaking, I don't want to avoid the use of context.root_path.

[wonjuleee]

It looks great to me in overall. Especially, it is happy to reduce the total time consumption during detection. Thank you!