[Spark] Mark InCommitTimestamp as a preview feature

[dhruvarya-db]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Follow up for #2946. Updates the configs and table feature name associated with InCommitTimestamp to indicate that it is in the preview phase --- the RFC has not been finalized but the feature is code complete.
 

How was this patch tested?

No additional testing.

Does this PR introduce any user-facing changes?

No

[scovich]

🎉

[dhruvarya-db]

Spark Master Build failures are unrelated to the changes in this PR:

[anilreddaboina54]

Thanks team for working on ICT feature.

Could you please answer my below question, that we are currently facing an issue with one particular scenario:

We have a requirement to move the delta files from one location to another directory, and we are using the "time-travel" feature with timestamp to read the particular version data.

After upgrading to the delta version with the ICT feature, if we move the delta files from one directory to another directory, will it support time travel after copying delta tables that were created with the earlier delta version without the ICT feature?

thanks in advance.

[dhruvarya-db]

Hi @anilreddaboina54

After upgrading to the delta version with the ICT feature, if we move the delta files from one directory to another directory, will it support time travel after copying delta tables that were created with the earlier delta version without the ICT feature?

Time travel will still work for those non-ICT delta commits but it won't be as reliable because those commits don't have an in-commit-timestamp and the file modification timestamp changes on copy. To put it simply, the behaviour is unchanged for those older commits.

Note that this feature is still in a preview phase, which means that if you enable ICT-preview on a table, it is not guaranteed that future clients will be able to write to this ICT-preview table. This is because the feature name will change when it is out of the preview phase. In the case of ICT, you might be able to make the table compatible again by dropping ICT-preview using the DROP TABLE FEATURE command and then re-enabling it using a non-preview ICT client.

Also note that when a feature is in preview, it is not intended for production use-cases. The feature can undergo big changes until the date it is out of preview. You can still use it to validate your use-case on test tables.