Introduce the reservedData to support more custom pluggable accessCheckers

[zuston]

Background

In current implementation, DelegationRssShuffleManager could decide whether to use uniffle or sort shuffle service by calling remote coordinator. The pluggable accessCheckers can be extended in coordinator to support more custom requirements.

Motivation

When we hope the uniffle can be applied to specified spark jobs to control which shuffle service to use, the mechanism of pluggable accessCheckers is useful.

We can implement the CustomDelegationRssShuffleManager(like DelegationRssShuffleManager)to inject some custom accessInfo and remote call the coordinator which use the custom access policy to decide which spark job use the uniffle shuffle service. However the current codebase dont support inject more accessinfo in client, only have tags and accessid. So this proposal hopes to introduce the data structure to store more custom extensible requirement.

Goals

1. Support injecting more custom data in AccessClusterRequest, which can be set in client

Tasks

• Introduce the extraProperties to support user-defined pluggable accessCheckers #42
• Explicitly set the constructor with AccessManager when extending AccessChecker #43
• No need to check the empty of access #41

[duanmeng]

It's a good idea to add a pb message AccessInfo, but we also need make sure that the new field in AccessInfo is common like accessId and tags.
IMH It is better to add a plugin feature in the coordinator to add customized checkers to support specific business requirements, not another delegation manager.

[zuston]

As i know the pluggable access checkers have been supported in current implementation of coordinator, which are managed by AccessManager.

And to support customized acess checker, i hope we can introduce a general pb message in AccessInfo. Feel free to discuss more. @duanmeng

[duanmeng]

hope we can introduce a general pb message in AccessInfo

We coud add a pb message AccessInfo and put it inside the pb message AccessClusterRequest. And then make the client can populate these extraProperties through spark conf. So that we can inject special access checkers (not part of the code base of uniffle) for business requirements.

message AccessInfo {
  string accessId = 1;
  repeated string tags = 2;
  map<string, string> extraProperties;
}

@zuston

[zuston]

All done. Close