Description
We've been talking about having light-weight mockers/stubbers for a long time #907 , but those never felt very pleasant to use and, especially, support. With hdfs/webhdfs and other clouds that don't have good python mockers we waste a lot of CI time on launching heavy-weight emulators or using real clouds.
We've been using moto https://github.com/spulec/moto for a long time now and it has been such a blast, that we wish there was something like that for other clouds. But there are no similar solutions for all our needs :( So we might consider using moto's structure as a guide to creating some lightweight mockers for our needs (no plan to support server-mode like moto does, just mocks for specific libraries). Our requirements are pretty basic(list, upload, download, remove - that's it), so that's all we need to be implemented for specific libraries, so there is very marginal overhead to doing it the right away, but at least this could be maintained, unlike DIY chaotic mocks.
Again, have to stress that there is no plan to create a feature-complete mockers like moto, just to use moto as a guide for a proper architecture and interface for our mockers.
Also, as @skshetry suggested, we might want to setup some env vars/test configs to run real clouds on a weekly basis, just to make sure that real stuff still works too.
- hdfs tests: add basic [web]hdfs mockers #5279
- webhdfs tests: add basic [web]hdfs mockers #5279
- azure
- gdrive
- gs
- oss