[Bug] Memory leak caused by UserGroupInformation.createProxyUser

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

The hadoop filesystem instance will be created too many time in cache, which will cause the shuffle server memory leak.

As we know, the filesystem cache's key is built by the scheme、authority and ugi. The scheme and authority are not changed everytime. But for ugi, if we invoke the createProxyUser, it will always create a new one, that means the every invoking Filesystem.get() , it will be cached due to different key.

We should cache the proxy user ugi to avoid caching every time.

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zuston]

cc @advancedxy @jerqi @smallzhongfeng This is a critical bug, I will fix this ASSP.

[advancedxy]

Do you any any heap dump to indicate this memory leak.

I just checked the related code, the proxy user created from the same ugi should equal to the same proxy user?

[zuston]

Do you any any heap dump to indicate this memory leak.

After dumping all heap, this bug was found.

I just checked the related code, the proxy user created from the same ugi should equal to the same proxy user?

I have not get your point.

[advancedxy]

Do you any any heap dump to indicate this memory leak.

After dumping all heap, this bug was found.

Is it possible for you to upload the screenshot here?

I just checked the related code, the proxy user created from the same ugi should equal to the same proxy user?

I have not get your point.

I mean that two proxy ugi for the same proxy user should be equal to each other, that is what I have checked for.
But never mind, the hash code for proxy ugi is different for each instance.

[zuston]