Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[WIP][mxnet 2.0] Pure C++ threaded dataloader that shared the interface with gluon dataloader #17464

Closed
wants to merge 92 commits into from
Closed

[WIP][mxnet 2.0] Pure C++ threaded dataloader that shared the interface with gluon dataloader #17464

wants to merge 92 commits into from
+5,881 −266

Commits on Feb 6, 2020

  1. init MXDataset and python test

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    c3edf17 View commit details
    Browse the repository at this point in the history

  2. revert makefile

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    bf25545 View commit details
    Browse the repository at this point in the history

  3. fix python binder

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    8386f03 View commit details
    Browse the repository at this point in the history

  4. fix ImageSequenceDataset

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    8098e77 View commit details
    Browse the repository at this point in the history

  5. add NDArrayDataset and TupleDataset

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    2e62c22 View commit details
    Browse the repository at this point in the history

  6. use attr handle rather than NDArraybase

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    6f809c5 View commit details
    Browse the repository at this point in the history

  7. fix image copy

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    e92518f View commit details
    Browse the repository at this point in the history

  8. fix deleted ndarray reference in python

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    b925d17 View commit details
    Browse the repository at this point in the history

  9. fix mismatch ndarray shape and scalar case

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    b410909 View commit details
    Browse the repository at this point in the history

  10. create __handle__ only when required

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    20046ee View commit details
    Browse the repository at this point in the history

  11. remove mnist data

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    d963cb3 View commit details
    Browse the repository at this point in the history

  12. init sampler and dataloader

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    04ead8e View commit details
    Browse the repository at this point in the history

  13. fix samplers

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    34ddf7b View commit details
    Browse the repository at this point in the history

  14. fix sampler registration

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    36621b9 View commit details
    Browse the repository at this point in the history

  15. backup changes

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    4b01028 View commit details
    Browse the repository at this point in the history

  16. add randomsamplerIter

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    a55c032 View commit details
    Browse the repository at this point in the history

  17. fix sampler and batch sampler, need to fix first batch cache

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    d41af7c View commit details
    Browse the repository at this point in the history

  18. fix prefetching caused missing data

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    ca00dc4 View commit details
    Browse the repository at this point in the history

  19. add registry for batchify functions

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    83a3ad2 View commit details
    Browse the repository at this point in the history

  20. fix dataset init, make dataset const

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    ccd4063 View commit details
    Browse the repository at this point in the history

  21. add python binding for threadedDataLoader

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    c0bc373 View commit details
    Browse the repository at this point in the history

  22. cache c++ datasets, samplers and batchify_fns

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    4f29029 View commit details
    Browse the repository at this point in the history

  23. add unittest

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    9f36a20 View commit details
    Browse the repository at this point in the history

  24. fix getting all data from DataBatch

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    2661677 View commit details
    Browse the repository at this point in the history

  25. fix first batch and clean up

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    913eece View commit details
    Browse the repository at this point in the history

  26. fix datloader all together

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    89432f3 View commit details
    Browse the repository at this point in the history

  27. fix last_batch=keep padding

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    58d9fe3 View commit details
    Browse the repository at this point in the history

  28. fix scalar batched data

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    a4aaf9a View commit details
    Browse the repository at this point in the history

  29. MXDataIter ndarray type np/nd

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    0cd97d9 View commit details
    Browse the repository at this point in the history

  30. change Dataset GetItem to return vector

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    7ac0147 View commit details
    Browse the repository at this point in the history

  31. fix capi name

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    a2e9ac1 View commit details
    Browse the repository at this point in the history

  32. add lazy transform dataset

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    fae205d View commit details
    Browse the repository at this point in the history

  33. lazy transform syntax suger

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    ce5a3e2 View commit details
    Browse the repository at this point in the history

  34. wrap up syntax suger with identical dataloader support

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    4acffda View commit details
    Browse the repository at this point in the history

  35. add scope controller for convenience

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    389eb38 View commit details
    Browse the repository at this point in the history

  36. fix compilation

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    78aec77 View commit details
    Browse the repository at this point in the history

  37. add gluon.data.Datasets to supported, add unittests

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    9c9cf0e View commit details
    Browse the repository at this point in the history

  38. cifar training successful

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    936eb4b View commit details
    Browse the repository at this point in the history

  39. fix transform_first closure with mxhandle

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    e729a8f View commit details
    Browse the repository at this point in the history

  40. try duplicate cachedop

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    e686d9b View commit details
    Browse the repository at this point in the history

  41. backup

    @zhreshold
    zhreshold committed Feb 6, 2020
    Configuration menu
    Copy the full SHA
    7082b4e View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2020

  1. fix random crop op

    @zhreshold
    zhreshold committed Feb 8, 2020
    Configuration menu
    Copy the full SHA
    1326d48 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2020

  1. add batchify capi

    @zhreshold
    zhreshold committed Feb 11, 2020
    Configuration menu
    Copy the full SHA
    49c99e5 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2020

  1. fix batchify c_api

    @zhreshold
    zhreshold committed Feb 13, 2020
    Configuration menu
    Copy the full SHA
    9f487e2 View commit details
    Browse the repository at this point in the history

  2. fix pickle batchify functions

    @zhreshold
    zhreshold committed Feb 13, 2020
    Configuration menu
    Copy the full SHA
    85e1bdb View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2020

  1. add batchify functions

    @zhreshold
    zhreshold committed Feb 14, 2020
    Configuration menu
    Copy the full SHA
    3eb0813 View commit details
    Browse the repository at this point in the history

  2. add appendBatchify

    @zhreshold
    zhreshold committed Feb 14, 2020
    Configuration menu
    Copy the full SHA
    827e017 View commit details
    Browse the repository at this point in the history

  3. fix batchify input array

    @zhreshold
    zhreshold committed Feb 14, 2020
    Configuration menu
    Copy the full SHA
    28b2c42 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2020

  1. add RecordFileDataset in backend

    @zhreshold
    zhreshold committed Feb 15, 2020
    Configuration menu
    Copy the full SHA
    000b628 View commit details
    Browse the repository at this point in the history

  2. add ImageRecordDataset to backend

    @zhreshold
    zhreshold committed Feb 15, 2020
    Configuration menu
    Copy the full SHA
    0954edf View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2020

  1. fix ImageRecordFile decoding

    @zhreshold
    zhreshold committed Feb 18, 2020
    Configuration menu
    Copy the full SHA
    bad8417 View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2020

  1. fix prefetcher with dynamic shape arrays

    @zhreshold
    zhreshold committed Feb 19, 2020
    Configuration menu
    Copy the full SHA
    9811a91 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2020

  1. add more tests

    @zhreshold
    zhreshold committed Feb 20, 2020
    Configuration menu
    Copy the full SHA
    ec9bca4 View commit details
    Browse the repository at this point in the history

  2. fix potential dangerous index accessing

    @zhreshold
    zhreshold committed Feb 20, 2020
    Configuration menu
    Copy the full SHA
    5d146e6 View commit details
    Browse the repository at this point in the history

  3. add arguments rather than scope

    @zhreshold
    zhreshold committed Feb 20, 2020
    Configuration menu
    Copy the full SHA
    10ef8ad View commit details
    Browse the repository at this point in the history

  4. add tests for nopython mode

    @zhreshold
    zhreshold committed Feb 20, 2020
    Configuration menu
    Copy the full SHA
    6f53af6 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2020

  1. try add bbox transforms

    @zhreshold
    zhreshold committed Feb 21, 2020
    Configuration menu
    Copy the full SHA
    73c8697 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2020

  1. add PadBatchify, WIP for pad axis

    @zhreshold
    zhreshold committed Feb 22, 2020
    Configuration menu
    Copy the full SHA
    28c6316 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2020

  1. fix padding batchify, add unittests

    @zhreshold
    zhreshold committed Feb 25, 2020
    Configuration menu
    Copy the full SHA
    879252e View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2020

  1. allow non boolean nopython flag

    @zhreshold
    zhreshold committed Feb 27, 2020
    Configuration menu
    Copy the full SHA
    5139449 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2020

  1. add operator for random_resized_crop

    @zhreshold
    zhreshold committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    42f3f5f View commit details
    Browse the repository at this point in the history

  2. fix cuda names

    @zhreshold
    zhreshold committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    fa9a8b9 View commit details
    Browse the repository at this point in the history

  3. fix reference before assignment for area

    @zhreshold
    zhreshold committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    4c1300d View commit details
    Browse the repository at this point in the history

  4. fix try_nopython behavior

    @zhreshold
    zhreshold committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    b67ecf4 View commit details
    Browse the repository at this point in the history

  5. backup before reorg cached op

    @zhreshold
    zhreshold committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    ec0a1f1 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2020

  1. add naive forward mode for cached op

    @zhreshold
    zhreshold committed Feb 29, 2020
    Configuration menu
    Copy the full SHA
    09b03b9 View commit details
    Browse the repository at this point in the history

  2. fix dataloader internal dataset indexing

    @zhreshold
    zhreshold committed Feb 29, 2020
    Configuration menu
    Copy the full SHA
    f88f514 View commit details
    Browse the repository at this point in the history

  3. fix

    @zhreshold
    zhreshold committed Feb 29, 2020
    Configuration menu
    Copy the full SHA
    cec1ac3 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2020

  1. add skip_engine

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    1358787 View commit details
    Browse the repository at this point in the history

  2. add resize check

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    660352f View commit details
    Browse the repository at this point in the history

  3. fix center_crop

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    4d381c3 View commit details
    Browse the repository at this point in the history

  4. allow 0 cropping bound

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    00f825b View commit details
    Browse the repository at this point in the history

  5. allow 0 cropping bound

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    7443ca0 View commit details
    Browse the repository at this point in the history

  6. add profiler to dataloader

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    0034f41 View commit details
    Browse the repository at this point in the history

  7. omp batchify

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    2c5b408 View commit details
    Browse the repository at this point in the history

  8. fix missing imagerecorddataset flag

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    7cc635b View commit details
    Browse the repository at this point in the history

  9. fix missing imagerecorddataset flag

    @zhreshold
    zhreshold committed Mar 1, 2020
    Configuration menu
    Copy the full SHA
    349b0aa View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2020

  1. reorg c++ io

    @zhreshold
    zhreshold committed Mar 2, 2020
    Configuration menu
    Copy the full SHA
    16dbfef View commit details
    Browse the repository at this point in the history

  2. revert

    @zhreshold
    zhreshold committed Mar 2, 2020
    Configuration menu
    Copy the full SHA
    a3a1c95 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2020

  1. remove engine copy

    @zhreshold
    zhreshold committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    ffdfff0 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2020

  1. fix test errors

    @zhreshold
    zhreshold committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    d66ef96 View commit details
    Browse the repository at this point in the history

  2. adding augmentations into vision.transforms

    @zhreshold
    zhreshold committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    0f79fca View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2020

  1. reorg gluon vision transforms

    @zhreshold
    zhreshold committed Mar 7, 2020
    Configuration menu
    Copy the full SHA
    42eca6e View commit details
    Browse the repository at this point in the history

  2. start bbox transforms

    @zhreshold
    zhreshold committed Mar 7, 2020
    Configuration menu
    Copy the full SHA
    cfe291a View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2020

  1. add bbox utils

    @zhreshold
    zhreshold committed Mar 10, 2020
    Configuration menu
    Copy the full SHA
    0364a1d View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into cpp_data

    @zhreshold
    zhreshold committed Mar 10, 2020
    Configuration menu
    Copy the full SHA
    a2ff894 View commit details
    Browse the repository at this point in the history

  3. revert ndarray contrib.py

    @zhreshold
    zhreshold committed Mar 10, 2020
    Configuration menu
    Copy the full SHA
    ae1391e View commit details
    Browse the repository at this point in the history

  4. add more bbox util funcs

    @zhreshold
    zhreshold committed Mar 10, 2020
    Configuration menu
    Copy the full SHA
    56bb1c5 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2020

  1. add numpy vision test

    @zhreshold
    zhreshold committed Mar 11, 2020
    Configuration menu
    Copy the full SHA
    ade29f3 View commit details
    Browse the repository at this point in the history

  2. fix unittests

    @zhreshold
    zhreshold committed Mar 11, 2020
    Configuration menu
    Copy the full SHA
    dc049af View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2020

  1. reorg naive cached op

    @zhreshold
    zhreshold committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    bed2608 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/cpp_data' into cpp_dataset

    @zhreshold
    zhreshold committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    421f81d View commit details
    Browse the repository at this point in the history