Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"SET TBLPROPERTIES" does not persist changes in the _delta_log #339

Closed
Jiaweihu08 opened this issue Jul 2, 2024 · 0 comments · Fixed by #342
Closed

"SET TBLPROPERTIES" does not persist changes in the _delta_log #339

Jiaweihu08 opened this issue Jul 2, 2024 · 0 comments · Fixed by #342
Assignees
Labels
type: bug Something isn't working

Comments

@Jiaweihu08
Copy link
Member

What went wrong?

Running ALTER TABLE table_name SET TBLPROPERTIES () does not persist the changes in the _delta_log/.

How to reproduce?

import org.apache.spark.sql.delta.actions.Action
import org.apache.spark.sql.delta.DeltaLog
import org.apache.spark.sql.delta.actions.CommitInfo
import org.apache.spark.sql.delta.util.FileNames

val df = loadTestData(spark)
val tmpDir = "/tmp/test"
df.write.format("qbeast").option("columnsToIndex", "user_id,price").save(tmpDir)
spark.sql(s"CREATE TABLE t1 USING QBEAST LOCATION $tmpDir")
spark.sql("ALTER TABLE t1 SET TBLPROPERTIES ('k' = 'v')")

val deltaLog = DeltaLog.forTable(spark, tablePath)
val conf = deltaLog.newDeltaHadoopConf()
val snapshot = deltaLog.update()
val v = snapshot.version

val isCommitted = (deltaLog.store
	.read(FileNames.deltaFile(deltaLog.logPath, v), conf)
	.map(Action.fromJson)
	.exists {
		case c: CommitInfo =>
			println(c)
			c.operation == "SET TBLPROPERTIES" &&
				c.operationParameters == Map("properties" -> """{"k":"v"}""")
		case _ => false
	}
)

isCommitted shouldBe true
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug Something isn't working
Projects
None yet
2 participants