Implement dm.Dataset index access

[itrushkin]

Issue No: 126681

Summary

This PR introduces dataset random index access for the easy conversion from dm.Dataset to torch.Dataset. The implementation requires an additional list property for DatasetItemStorage class, which would cost additional O(n) memory. It doesn't store items but references to them (subset and ID).

Concern:
I wonder whether it is possible to store and manage this list for indexing, removal, addition, and iteration to reduce memory consumption. I tried to combine the following changes:

1. The _traversal_order dictionary seems unnecessary as all data is already stored and managed in the data property via .put() and .remove() methods. __len__, __iter__, and other methods can rely solely on data. Optionally, an _order property could enable O(1) __len__ implementation.
2. Instead of overwriting data values with None to mark removal, deleting the entire dictionary entry accurately reflects the dataset state and eliminates ambiguity.

As a result, tests set test_can_create_patch* started to fail. It appears that items were not actually removed from the dataset, but merely marked as such. This suggests either the cache initialization or element remove operation needs adjusting for this scenario. Resolving this with assistance could significantly improve dm.Dataset interaction.

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added the description of my changes into CHANGELOG.​
• I have updated the documentation accordingly

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (eb44355) 80.53% compared to head (dba46e6) 80.54%.
Report is 4 commits behind head on develop.

❗ Current head dba46e6 differs from pull request most recent head bdbe2c2. Consider uploading reports for the commit bdbe2c2 to get more accurate results

Files	Patch %	Lines
src/datumaro/components/dataset_item_storage.py	88.88%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1247      +/-   ##
===========================================
+ Coverage    80.53%   80.54%   +0.01%     
===========================================
  Files          270      270              
  Lines        30232    30260      +28     
  Branches      5898     5906       +8     
===========================================
+ Hits         24348    24374      +26     
- Misses        4506     4507       +1     
- Partials      1378     1379       +1     

Flag	Coverage Δ
ubuntu-20.04_Python-3.8	80.53% <94.11%> (+0.01%)	⬆️
windows-2022_Python-3.8	80.51% <94.11%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vinnamkim]

Hi @itrushkin,

Could we add a test for this feature about the scenarios where the size of dataset increases or decreases? For example,

a) Scenario with dm.Dataset.put() and dm.Dataset.remove().

1. Construct an arbitrary dm.Dataset
2. Get an item from the dataset with index
3. Increase the dataset size with dm.Dataset.put or decrease it with dm.Dataset.put couple of times.
4. Get an item from the dataset with index and check whether it behaves correctly

b) Scenario with transforms

1. Construct an arbitrary dm.Dataset
2. Get an item from the dataset with index
3. Increase or decrease the dataset size with Datumaro transforms (I guess you might use tile/untile transforms for both https://github.com/openvinotoolkit/datumaro/tree/develop/src/datumaro/plugins/tiling).
4. Get an item from the dataset with index and check whether it behaves correctly

[vinnamkim]

LGTM.