why can not import own datasets?

[MagicSen]

Follow the https://classyvision.ai/tutorials/classy_dataset,
I create own dataset reader, like this

but when i use this in config file, the program can't not find this.

how to fix this?

[aadcock]

Thanks for trying out Classy Vision!

For a python decorator to work, the file it is in needs to be imported. In classy vision we do this in the init.py files. See for example here, this means we only need to add:

import classy_vision.dataset

to our training script to automatically register everything in that folder as we add datasets. You'll need to import your new dataset in the training script which you can do with a direct import or with an import all statement like in the above example.

I will also update the tutorial to make this clearer. Thanks for bringing it up!

[MagicSen]

Thanks a lot, I copy code into classy_vision/dataset/ , it worked.
but I think it is not a very elegant approach，because the user need to update the library.
By the way, I read the source, the design is very cool.

[mannatsingh]

@MagicSen glad you like the design!

Also, you don't need to update the library to get your dataset to work.

If you follow the instructions to set up your own project as per the links that follow, you can just move your dataset code to my-project/datasets/my_custom_dataset.py and my_project/classy_train.py will pick up your dataset -

• https://github.com/facebookresearch/ClassyVision#getting-started
• https://classyvision.ai/tutorials/getting_started#1.-Start-a-new-project

If you just want to use a dataset without creating a project for some reason, make sure you do import path.to.my_custom_dataset and you should be good to go. classy_train.py does that for you automatically and that's why we recommend creating a project to get started!

Let us know if you have any trouble with this.

Created #317 to document this.
cc @aadcock , @vreis .

[vreis]

@MagicSen: as Mannat mentioned above, classy_train.py automatically imports everything in the datasets folder of the project that was created for you, you don't have to import anything manually or modify the library itself. Please note I'm referring to the datasets folder under the project folder generated by classy-project, not the datasets folder under the classy vision repo. Feel free to re-open this if that's still not working for you.

[miguelvr]

I solved this by allowing the necessary imports to be configured from the configuration itself

e.g.

config.json

{
  "imports": [
    "mymodule.datasets",
    "mymodule.models"
  ]
  // add rest of classy vision config here
}

main.py

import json
import importlib

def parse_config(config_file):
    with open(config_file, 'r') as f:
        config = json.load(f)

    import_modules(config)

    return config


def import_modules(config):
    for import_str in config.get("imports", []):
        importlib.import_module(import_str)