Import DVC remote plugin config schemas from plugin packages

[sisp]

Disclaimer: I'm still learning how DVC's plugin system for DVC remotes works, so please correct me if my understanding is simply wrong.

I'm working on a DVC plugin to natively support GitLab's generic package registry as a DVC remote. While doing so, I noticed that DVC manages config schemas centrally in dvc/config_schema.py for all plugins, each of which is implemented in a separate project though. This means, it's impossible to create and use a custom DVC plugin because at least the config validation fails as DVC doesn't recognize the custom plugin. The error I'm seeing when running the partially working test suite is:

dvc.config.ConfigError: config file error: Unsupported URL type gitlab:// for dictionary value @ data['remote']['upstream']

I don't think a central config schema aligns well with a plugin system. Instead, I believe it would be much better to retrieve the config schemas for the plugins from the (installed and registered) plugin packages, i.e. each plugin should define its own config schema and DVC should combine them at runtime. This approach would make DVC truly extensible - which is the whole idea of a plugin system in the first place (isn't it?).

[efiop]

It makes total sense and it is and was in our plans, but we didn't get to it since there wasn't any demand from the outside.

[skshetry]

@sisp, are you still planning on creating a dvc-plugin/fsspec?

[sisp]

Not at the moment. It turned out that GitLab's packages API doesn't properly support listing files (which would be necessary for performing garbage collection) and there's a race condition when performing parallel upload of files to a new package which causes an HTTP 500 response, which is annoying behavior. Since DVC (via scmrepo) now supports Git LFS download, this seems like the way forward for me for now.

[skshetry]

Okay then. I am closing this issue, as we don't have plans to implement this in short to medium term.