Support versioning of PartitionedDataset classes

[deepyaman]

Description

Related to #521

Why is this important?
PartitionedDataSet is a very useful dataset, especially for advance use cases. It is particular important for advance use case, such as optimising performance with lazy-loading and lazy-saving. Recently we introduce #2161 as well. However, it doesn't support the versioned flag and user cannot version their data when they use this dataset. #521 is the last attempt.

Read more about ParittionedDataSet: #521

Context

• Initially driven by personal need
• Occasional user questions/demand (most recently, https://kedro-org.slack.com/archives/C03RKP2LW64/p1690448225699889)

Extra Context with previous discussion

If you are not familiar with IncrementalDataSet or PartitionedDataSet, we have a https://github.com/kedro-org/kedro/milestone/40 and this is a great start! It’s one of the wrapper dataset that we created few years ago and I think this is a powerful way to extend Kedro’s datasets. We also need to improve our docs as I think these wrapper datasets should be used more often.
If you are not familiar with lazy loading, that’s great! Read our own doc https://docs.kedro.org/en/stable/data/kedro_io.html#partitioned-dataset and see if it makes sense! If not then we will have more idea about how to improve this doc.
DataEngineeringOne tutorial of IncrementalDataSet - https://www.youtube.com/watch?v=v7JSSiYgqpg
IncrementalDataSet’s docs - https://docs.kedro.org/en/stable/kedro.io.IncrementalDataSet.html

Possible Implementation

#521

Possible Alternatives

[noklam]

I am linking all of these in the milestone.

• Migrate PartitionedDataSet and IncrementalDataSet to kedro-datasets kedro-plugins#249
• Replace eager loading with lazy loading in IncrementalDataSet kedro-plugins#250
• IncrementalDataSet lazy loading #2456

[noklam]

Excited to add versioning to PartitionedDataset! Thanks @deepyaman

2023-08-02 Technical Design
cross-post @deepyaman comments

• We discussed the options laid out by Dmitrii in Support versioning of the underlying dataset with PartitionedDataset #521 (comment); it seems that some people (e.g. Yetu, Sajid) would be happy to just go with kedro.io.CSVS3DataSet does not use load_args  #2 and gather feedback sooner, while others (e.g. Merel, Ahdra, Nok) would be happy to start with Release test (#1) #1 (bit more involved, but hopefully not significantly). Nobody who was in favor of kedro.io.CSVS3DataSet does not use load_args  #2 has any objection to Release test (#1) #1.
• A number of people (e.g. Marcin, Ahdra) would be in favor of getting some more user feedback first. I will send out a feeler to #user-research with these options + "I don't want any of this; I need versioning of the top-level dataset" to see if anybody has strong opinions they want to voice. I'll try to send it out today or tomorrow.
• Unless the #user-research poll comes out with significant feedback to change our view, will eventually go forward with Release test (#1) #1 to get this feature in the hands of users.

[merelcht]

Completed in kedro-org/kedro-plugins#447