directory structure for production deployment

[analystanand]

Currently we are planing to utilize dvc for our projects.

• Anyone utilizing the dvc for production as it can put everything(model.p) into dvc and will hide them for repo.

[efiop]

Hi @analystanand !

Could you please clarify your question? Are you asking if you can hide .dvc files into a separate directory? If so, the answer is you can, but we do not recommend that and instead suggest storing dvcfiles alongside your code. That being said, if you really need to hide them, consider using -f and -c options for dvc run and make sure that the commands you run are fully aware that they are being run from the directories that respective dvcfiles are located in. Also be aware of that dvc add will only put respective dvcfile alongside the data file that you are trying to add, so If you really need to hide it, you will have to modify paths in respective dvcfiles. Dvcfiles have a very simple yaml format, so editing them by-hand should not be a problem.

Thanks,
Ruslan

[analystanand]

I find quite easy to reproduce experiments from one system to another with the help of dvc.

But As we have to deploy projects.. that we need to manually take model.p from master branch and put them into git for loading while prediction.

In general while creating we add data/* to DVC but to deploy our remote repo. we need just model.pickle. which needs to manually placed out dvc..

My question is can we create model.pickle which will be in pipeline as well as available in git repo to deploy directly from continuous integration.

see the structure. https://drivendata.github.io/cookiecutter-data-science/

[efiop]

Ah, I see. Sure, just use -O model.pickle(-O is equivalent to --outs-no-cache, which will tell dvc to not cache this output, so that you can git add it yourself and store it in git) instead of -o model.pickle in dvc run when defining your pipeline(or add cache=true for model.pickle in the respective dvcfile that has model.pickle as an output).

[analystanand]

got it. thanks

[efiop]

You are welcome :) Closing for now, feel free to reopen.