New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chongyangbai/kvpair #82

Merged

pj-ms merged 25 commits into microsoft:main from cy-bai:chongyangbai/kvpair

Aug 2, 2024

Contributor

cy-bai commented Jul 18, 2024 •

edited

Loading

Add key_value_pair dataset support.

Done in this PR:

New dataset type: key_value_pair
MultiImageLabelManifest, which binds a label and its image ids to the label manifest, so that multi-image annotation is supported
MultiImageDatasetManifest which binds a list of ImageDataManifest as images and a list of MultiImageLabelManifest as annotations
implement manifest adaptor from coco, manifest to coco, manifest merge for this dataset type
Implement KVPairSchema and KVPairFieldSchema classes
Implement KVPairDatasetManifest which inherits MultiImageDatasetManifest, and makes sure each label conforms the given schema
Update docs
check script for key_value_pair dataset

Todo:

more operations on MultiImageDatasetManifest: spawn, sample

chongyangbai added 9 commits

July 10, 2024 23:58


          add annotation wise dataset manifest, add key value pair dataset

b468446


          add dataset registry and hub, add manifest merge and coco generator o…

47c5eeb

…perations


          add schema check

5a78419


          add coco adaptor and dataset info tests

40eb56a


          test manifest dataset

776e82b


          pass all pytests

66f00b2


          add schema and fieldschema

39d574f


          get_targets testcase for visiondataset

77dcd56


          cleanup code

537ff42

cy-bai marked this pull request as ready for review

July 18, 2024 18:59

juliagongms reviewed

View reviewed changes

vision_datasets/common/data_manifest/coco_manifest_adaptor.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/common/data_manifest/data_manifest.py Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/kv_pair/manifest.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/kv_pair/manifest.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/kv_pair/manifest.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/kv_pair/manifest.py Outdated Show resolved Hide resolved

Collaborator

pj-ms commented Jul 22, 2024

Need documentation update at https://github.com/microsoft/vision-datasets/blob/main/COCO_DATA_FORMAT.md, if it is still coco

pj-ms reviewed

View reviewed changes

vision_datasets/common/constants.py Outdated Show resolved Hide resolved

vision_datasets/common/data_manifest/coco_manifest_adaptor.py Outdated Show resolved Hide resolved

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

vision_datasets/common/data_manifest/operations/generate_coco.py Outdated Show resolved Hide resolved

chongyangbai added 4 commits

July 23, 2024 22:00


          change class names, create MultiImageLabelManifest that encapsulates …

8b97cd4

…multi-image label data


          change class names, create MultiImageLabelManifest that encapsulates …

701940c

…multi-image label data


          update doc for creating key_value_pair dataset

35da2a2


          add a todo comment

8ff7d62

juliagongms reviewed

View reviewed changes

README.md Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

DATA_PREPARATION.md Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/kv_pair/manifest.py Outdated Show resolved Hide resolved

juliagongms reviewed

View reviewed changes

vision_datasets/common/dataset_info.py Outdated Show resolved Hide resolved

juliagongms approved these changes

View reviewed changes

Contributor

juliagongms left a comment

Added a few more minor comments, but looks good otherwise - thanks for the changes!

chongyangbai added 3 commits

July 24, 2024 17:04


          fix minor comments

30a18c3


          rename names like kvpair to key_value_pair


          update doc

a3d3a8c

juliagongms reviewed

View reviewed changes

tests/test_dataset_registry.py Outdated Show resolved Hide resolved

juliagongms approved these changes

View reviewed changes

Contributor

juliagongms left a comment

Thanks, Chongyang! Found one place where kv -> key_value might be good, but otherwise looks good to me!


          rename more kv to be key_value_pair

71dad94

juliagongms approved these changes

View reviewed changes

Collaborator

pj-ms commented Jul 30, 2024

version bump

Collaborator

pj-ms commented Jul 30, 2024

probably better to remove your design doc link from the PR, as this is public repo


          bump version

2c63ad4

pj-ms reviewed

View reviewed changes

COCO_DATA_FORMAT.md Outdated Show resolved Hide resolved

COCO_DATA_FORMAT.md Outdated Show resolved Hide resolved

DATA_PREPARATION.md Outdated Show resolved Hide resolved

DATA_PREPARATION.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Outdated Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Outdated Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Show resolved Hide resolved

vision_datasets/common/data_manifest/data_manifest.py Outdated Show resolved Hide resolved

chongyangbai added 6 commits

July 30, 2024 20:04


          fix formatting

090801a


          resolve comments

edf8c42


          rename MultiImageDatasetManifest to DatasetManifestWithMultiImageLabe…

0738cf0

…l, reformat README dataset contracts as a table


          update dataset contracts in README

57cd1db


          add dataset check support for key_value_pair dataset

8ec24d0


          fix py3.9 build error

58dca7a

pj-ms reviewed

View reviewed changes

COCO_DATA_FORMAT.md Outdated Show resolved Hide resolved

COCO_DATA_FORMAT.md Outdated Show resolved Hide resolved

COCO_DATA_FORMAT.md Outdated Show resolved Hide resolved

DATA_PREPARATION.md Outdated Show resolved Hide resolved

tests/resources/util.py Outdated Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Outdated Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Outdated Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Show resolved Hide resolved

vision_datasets/key_value_pair/manifest.py Outdated Show resolved Hide resolved

DATA_PREPARATION.md Outdated Show resolved Hide resolved


          resolve comments: (1)improve doc, (2)make schema a argument in ctor o…

c1d1684

…f KeyValuePairDatasetManifest, (3)bbox definition, (4) enum type

pj-ms approved these changes

View reviewed changes

pj-ms merged commit f760dec into microsoft:main

4 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet