[AUTHZ] Support CreateTableAs command for Paimon

[bowenliang123]

Why are the changes needed?

Support CreateTableAs command for Paimon

• Create Table As Command: https://paimon.apache.org/docs/master/engines/spark3/#create-table
• to validate [Umberella] Support Paimon in Authz #5470 (comment)

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before make a pull request

Was this patch authored or co-authored using generative AI tooling?

No.

[codecov-commenter]

Codecov Report

Merging #5590 (4287e90) into master (c24e984) will increase coverage by 0.31%.
Report is 14 commits behind head on master.
The diff coverage is n/a.

❗ Current head 4287e90 differs from pull request most recent head c997b63. Consider uploading reports for the commit c997b63 to get more accurate results

@@             Coverage Diff              @@
##             master    #5590      +/-   ##
============================================
+ Coverage     61.35%   61.67%   +0.31%     
  Complexity       23       23              
============================================
  Files           600      603       +3     
  Lines         34316    35670    +1354     
  Branches       4501     4869     +368     
============================================
+ Hits          21054    21998     +944     
- Misses        11128    11290     +162     
- Partials       2134     2382     +248     

see 63 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[davidyuan1223]

I want ask a question, why the admin to execute doAs in the next, don't use sql to execute, i found your code is doAs(admin, createTableAs), why not is doAs(admin,sql(createTableAs)).
And i found the DeltaCatalogRanger also use doAs(user,mySql),but the HudiCatalog use doAs(user,sql(mySql)).

doAs code is

  protected def doAs[T](user: String, f: => T): T = {
    UserGroupInformation.createRemoteUser(user).doAs[T](
      new PrivilegedExceptionAction[T] {
        override def run(): T = f
      })
  }

If we not use spark.sql to execute our sql, it's only a string format???

Which is the right way???

@pan3793 @wForget @yaooqinn could you give some advice

[yaooqinn]

@davidyuan1223 I believe you are right, and @bowenliang123 made a mistake. Nice catch.

[davidyuan1223]

@davidyuan1223 I believe you are right, and @bowenliang123 made a mistake. Nice catch.

And I found the DeltaCatalogRanger also is this code style, I think you need to check the authz module's test case to avoid this problem. This problem just like this comment, I want know why admin create table but it can not select from the table?

[bowenliang123]

Sorry about that. Will fix and have further investigation.

[bowenliang123]

@davidyuan1223 More investigation in the logical plan is required to find the reason. Maybe missing fields in parsed privilege objects. Let me update the test case for Paimon first. And thanks for mentioning possible bug for Delta test suite as well.

[davidyuan1223]

@davidyuan1223 More investigation in the logical plan is required to find the reason. Maybe missing fields in parsed privilege objects. Let me update the test case for Paimon first. And thanks for mentioning possible bug for Delta test suite as well.

thanks your reply, if you find the root cause about this bug, wish you can teach me how to fix this, so that I can complete the issue #5470

[bowenliang123]

@davidyuan1223 More investigation in the logical plan is required to find the reason. Maybe missing fields in parsed privilege objects. Let me update the test case for Paimon first. And thanks for mentioning possible bug for Delta test suite as well.

thanks your reply, if you find the root cause about this bug, wish you can teach me how to fix this, so that I can complete the issue #5470

Sure. Will ping you when I came to any conclusion in the reason. Thanks for creating a bug issue for it.

[bowenliang123]

The initial reason is that missing database field in RangerAccessRequest, and the admin user is granted on db-table-column *-*-* resource rules.

[davidyuan1223]

RangerAccessRequest

LGTM, I will debug in my env

[davidyuan1223]

The initial reason is that missing database field in RangerAccessRequest, and the admin user is granted on db-table-column *-*-* resource rules.

this is the logic plan about createTableAs, looks like it's plan have some problem

CreateTableAsSelect org.apache.paimon.spark.SparkCatalog@4a5f435b, paimon_ns.table2, [provider=PAIMON], true
+- RelationV2[id#6, name#7, city#8] table1

[bowenliang123]

Yes. Will have a dive-in debugging in extractors later.

[bowenliang123]

Seems that there's a Paimon's org.apache.paimon.AppendOnlyFileStore as the Spark Table inside the input query plan, which is unsupported by the DataSourceV2RelationTableExtractor.

[davidyuan1223]

Seems that there's a Paimon's org.apache.paimon.AppendOnlyFileStore as the Spark Table inside the input query plan, which is unsupported by the DataSourceV2RelationTableExtractor.

looks like the paimon can not supported in kyuubi's authz,? we can not make sure the paimon's plan is same as spark's plan

[bowenliang123]

It's normal to see lakehouse has individual implementation of spark plan in some commands. They're still part of spark plans. To support them, we could adapt our extractor for the details of the plan. But the precondition is that table identifier is carried by the plan. Should have look at filestore class of paimon for that.

[davidyuan1223]

It's normal to see lakehouse has individual implementation of spark plan in some commands. They're still part of spark plans. To support them, we could adapt our extractor for the details of the plan. But the precondition is that table identifier is carried by the plan. Should have look at filestore class of paimon for that.

The paimon's FileStoreTable top interface's method name is only get the tableName, what can we do to improve the paimon could return database? The source code in 0.5 only will return like xxx.db/tableName to tableName