[Spike] Decide on versioning: keep or remove

[merelcht]

Description

While we've done a lot of exploration into integrating Kedro with other tools to facilitate versioning, we haven't actually made the decision to keep or remove the native Kedro versioning feature.

The purpose of this spike is to collect information and insights into what it would mean if we removed the native versioning feature, so that we can make a decision.

• Pros & cons of removal
• Impact of removal: what does this mean for users? For other features?
• Estimate of engineering effort of removal

Context

#4199 (comment)
#4199 (comment)
#4129 (comment)

[astrojuanlu]

This was discussed on Tech Design 2025-02-19

@merelcht and @lrcouto exposed the pros and cons of keeping and removing versioning, as well as a "third way" which was moving versioning information to the catalog. From the discussion, it was clear that, even with the available quantitative evidence that versioning has low usage #4129 (comment), there's some qualitative evidence that this is very much desired.

A comment from today:

Since I'm using other versioned datasets I didnt want to introduce a different tool to manage versioning.
I dont know much about delta lake and iceberg, but seems like an overkill for most usecases, no? dvc seems promising.

@jccalvojackson in kedro-org/kedro-plugins#1010 (comment)

And on top of that, despite how promising Delta and Iceberg are, they are not yet fully mature. @deepyaman wanted us to have close integration with those, which is a good idea anyway.

@ElenaKhaustova made some clarifications on the "move to the catalog" idea (details to be filled) but in the end the decision was to keep versioning as is.

I'm closing this spike and making some updates in the other (scattered) issues we have around the topic. Thanks everyone!