Update docs to include SparkStreamingDataSet and ManagedTableDataSet

[noklam]

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

Development notes

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[noklam]

As usual, never underestimate the time to fix an indentation error with Sphinx, but here we are! Big thanks to @AhdraMeraliQB !

[stichbury]

LGTM 🏆 and yes, Sphinx errors

[noklam]

It's very hard to debug since the build keep failing, I add a pip list so it's easier to know if it is installing the correction version of kedro-datasets

[noklam]

We need to wait until the next kedro-datasets release.

[astrojuanlu]

Approved when/if the RTD build passes 😄

[astrojuanlu]

Of course:

sphinx.errors.SphinxWarning: /home/docs/checkouts/readthedocs.org/user_builds/kedro/envs/2679/lib/python3.8/site-packages/kedro_datasets/databricks/managed_table_dataset.py:docstring of kedro_datasets.databricks.managed_table_dataset.ManagedTableDataSet:30:Unexpected indentation.

[stichbury]

@jmholzer @noklam I'm afraid I probably need your help to get this resolved, please. Sphinx is emitting the following:

sphinx.errors.SphinxWarning: /home/docs/checkouts/readthedocs.org/user_builds/kedro/envs/2679/lib/python3.8/site-packages/kedro_datasets/databricks/managed_table_dataset.py:docstring of kedro_datasets.databricks.managed_table_dataset.ManagedTableDataSet:30:Unexpected indentation.

And I can't see a docstring at that location in the latest version of the databricks dataset.

I want to get these docs built into our set because we have a blog post to publish that should point to them. Even if they don't get into release 0.18.11 it would be sensible to have them on latest, which needs a kedro-datasets release to fix any problematic docstrings.

Please could you take a look at the code and see what is causing Sphinx to fail and maybe separately we can review how to avoid this in future by running a suitable linter over each new dataset that we add?

[stichbury]

Thank you @noklam and @jmholzer for getting this building!

[noklam]

@stichbury Not sure where is the best place to fit this documentation. I can go through the doc build process with you.

1. The build process only happens when kedro have a new release
2. kedro will pull latest kedro-datasets, which means any change in kedro-datasets will require a new release in kedro-datasets before it shows up in our docs.

How do you fix Dataset docs?

Kedro's GitPod is the best place to test it. In our setup.py, it points to the latest kedro-datasets. When you need to test changes on a feature branch, you need to update the reference to point to your kedro-datasets feature branch.

1. You can either open a PR on kedro or just start GitPod and do make build-docs to check if documentation is build successfully. If you have a local setup you can also use your local environment, but I found GitPod convenient for these short-lived isolated environment.

2. Update the reference to point to your feature branch

3. Add new datasets in the relevant .rst file, usually docs/source/kedro_datasets.rst

4. Check if the build is success

5. Merge your doc PR in kedro-datasets

6. (Do a release)

7. Revert the reference to latest kedro-datasets -

# setup.py
-         # "kedro-datasets[all]~=1.4.1",
+        "kedro-datasets[all]@git+https://github.com/kedro-org/kedro-plugins@fix/rst-indentation-managedtabledataset#subdirectory=kedro-datasets",

[astrojuanlu]

Docs build passed! 🎉

• https://kedro--2679.org.readthedocs.build/en/2679/kedro_datasets.spark.SparkStreamingDataSet.html
• https://kedro--2679.org.readthedocs.build/en/2679/kedro_datasets.databricks.ManagedTableDataSet.html

[noklam]

Let's merge it then we can go with the release