Please help!!! TENANT_AUTHORIZER_ERROR

[netscrol2016]

Hello, please help me, I use the ozone with Ranger and can perform all the actions to create and delete volumes, but tenants can only be viewed, but not created or changed users. But a few days ago everything worked without problems. Help me understand what's going on.

[netscrol2016]

/opt/ozone/bin/ozone tenant create room-tenant --om-service-id=cluster1


2023-11-07 16:28:23,247 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-11-07 16:28:24,223 INFO rpc.RpcClient: Creating Tenant: 'room-tenant', with new volume: 'room-tenant'
TENANT_AUTHORIZER_ERROR java.io.IOException: org.apache.ranger.RangerServiceException: Ranger API org.apache.ranger.RangerClient$API@4a92034f failed: statusCode=401, status=Unauthorized, response:

[dombizita]

Hi @netscrol2016,
Could you check the OM logs? Do you find anything like this in the logs?

ERROR org.apache.hadoop.ozone.om.multitenant.RangerClientMultiTenantAccessController: Auth failure. Please double check Ranger-related configs

Also what version of Ozone and Ranger do you use?

[netscrol2016]

Hello @dombizita , in the OM logs you can only see that Ranger policies are being synchronized. Ranger version 2.3.0 Ozone 1.3.0

[netscrol2016]

ranger access log

10.10.1.225 - - [07/Nov/2023:18:23:01 +0000] "POST /service/public/v2/api/roles?serviceName=test-ozone HTTP/1.1" 401 - 3 "-" "Java/1.8.0_372"

[netscrol2016]

but on another node I found this, but is this old date.

2023-10-31 11:03:48,031 ERROR org.apache.hadoop.ozone.om.multitenant.RangerClientMultiTenantAccessController: Auth failure. Please double check Ranger-related configs

[netscrol2016]

Thanks for the info, but I'm doing this on the leader

[dombizita]

This is really older, I'm not sure, since Oct 31st was it working at some point?

I remember a similar issue, where this was the error while creating tenant and in the OM logs this was there. It was found that RangerClient doesn't auto renew its Kerberos ticket when auth'ed to Ranger. cc @smengcl
The solution was from Ranger side and I think RANGER-3848 fixed it, it seems that 2.3.0 is affected by this problem. I don't know any workarounds from Ozone side, maybe others can help with it.

[netscrol2016]

Hi @dombizita, I installed ranger and these on the 31st. The error could have been due to the setup process. Everything worked for about a week, but yesterday it broke(((

[dombizita]

Since it's not working (yesterday) on the leader OM there is no ERROR related to this? This message only occurred once on the OM leader on 31st?

[netscrol2016]

@dombizita yes

[DaveTeng0]

hey @netscrol2016 , what happen if you try :
kinit -kt {path_to_your_om_keytab} {user_name_om} to refresh the kerberos ticket?
does that work?

[netscrol2016]

Hello @DaveTeng0 i check it

[root@hadoop-master001 ]# kinit -k -t /etc/hadoop/keytabs/om.keytab OM
kinit: Keytab contains no suitable keys for OM@KL.MAN.COM while getting initial credentials

I found om keitab this way etc/hadoop/keytabs/om.keytab and used it

[netscrol2016]

i check it

kinit -kt /etc/hadoop/keytabs/om.keytab OM/hadoop-master001.lk.man.com@KL.MAN.COM
klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: OM/hadoop-master001.lk.man.com@KL.MAN.COM

Valid starting       Expires              Service principal
11/08/2023 09:29:06  11/09/2023 09:18:03  krbtgt/KL.MAN.COM@KL.MAN.COM
	renew until 11/15/2023 09:29:06

[netscrol2016]

kerberos ticket good work

[netscrol2016]

My full DEDUG

/opt/ozone/bin/ozone tenant create tenant4 --om-service-id=cluster1
2023-11-08 08:18:58,802 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[GetGroups])
2023-11-08 08:18:58,807 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)])
2023-11-08 08:18:58,807 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)])
2023-11-08 08:18:58,807 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Renewal failures since last successful login])
2023-11-08 08:18:58,808 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[Renewal failures since startup])
2023-11-08 08:18:58,808 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
2023-11-08 08:18:58,928 DEBUG util.Shell: setsid exited with exit code 0
2023-11-08 08:18:59,019 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
2023-11-08 08:18:59,035 DEBUG security.Groups:  Creating new Groups object
2023-11-08 08:18:59,037 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
2023-11-08 08:18:59,037 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
2023-11-08 08:18:59,037 DEBUG util.NativeCodeLoader: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2023-11-08 08:18:59,037 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-11-08 08:18:59,038 DEBUG util.PerformanceAdvisory: Falling back to shell based
2023-11-08 08:18:59,039 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2023-11-08 08:18:59,064 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
2023-11-08 08:18:59,078 DEBUG security.UserGroupInformation: Hadoop login
2023-11-08 08:18:59,081 DEBUG security.UserGroupInformation: hadoop login commit
2023-11-08 08:18:59,083 DEBUG security.UserGroupInformation: Using kerberos user: s.p.cher@KL.MAN.COM
2023-11-08 08:18:59,083 DEBUG security.UserGroupInformation: Using user: "s.p.cher@KL.MAN.COM" with name: s.p.cher@KL.MAN.COM
2023-11-08 08:18:59,083 DEBUG security.UserGroupInformation: User entry: "s.p.cher@KL.MAN.COM"
2023-11-08 08:18:59,092 DEBUG security.UserGroupInformation: UGI loginUser: s.p.cher@KL.MAN.COM (auth:KERBEROS)
2023-11-08 08:18:59,094 DEBUG security.UserGroupInformation: Current time is 1699431539094, next refresh is 1699500092400
2023-11-08 08:18:59,186 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc.ProtobufRpcEngine2$RpcProtobufRequest, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker@550ee7e5
2023-11-08 08:18:59,197 DEBUG ipc.Client: getting client out of cache: Client-2cc893738d9e4ad39b3a816ae3b4d1b6
2023-11-08 08:18:59,436 DEBUG ipc.Client: The ping interval is 60000 ms.
2023-11-08 08:18:59,436 DEBUG ipc.Client: Connecting to hadoop-master001.kl.man.com/10.10.1.104:9862
2023-11-08 08:18:59,436 DEBUG ipc.Client: Setup connection to hadoop-master001.kl.man.com/10.10.1.104:9862
2023-11-08 08:18:59,451 DEBUG security.UserGroupInformation: PrivilegedAction [as: s.p.cher@KL.MAN.COM (auth:KERBEROS)][action: org.apache.hadoop.ipc.Client$Connection$2@518caac3]
java.lang.Exception
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:839)
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1677)
	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
	at org.apache.hadoop.ipc.Client.call(Client.java:1455)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:122)
	at com.sun.proxy.$Proxy15.submitRequest(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy15.submitRequest(Unknown Source)
	at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
	at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
	at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
	at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:114)
	at org.apache.hadoop.ozone.shell.OzoneAddress.createClientForS3Commands(OzoneAddress.java:190)
	at org.apache.hadoop.ozone.shell.tenant.TenantHandler.createClient(TenantHandler.java:51)
	at org.apache.hadoop.ozone.shell.Handler.call(Handler.java:94)
	at org.apache.hadoop.ozone.shell.Handler.call(Handler.java:44)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.lambda$execute$0(TenantShell.java:44)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:162)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.execute(TenantShell.java:43)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.main(TenantShell.java:53)
2023-11-08 08:18:59,551 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE

2023-11-08 08:18:59,563 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.ozone.security.OzoneDelegationTokenSelector)
2023-11-08 08:18:59,568 DEBUG security.OzoneDelegationTokenSelector: Got tokens: null for service 10.10.1.104:9862
2023-11-08 08:18:59,568 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one
2023-11-08 08:18:59,568 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=, serverPrincipal=ozone.om.kerberos.principal)
2023-11-08 08:18:59,569 DEBUG security.SaslRpcClient: getting serverKey: ozone.om.kerberos.principal conf value: OM/_HOST@KL.MAN.COM principal: OM/hadoop-master001.kl.man.com@KL.MAN.COM
2023-11-08 08:18:59,569 DEBUG security.SaslRpcClient: RPC Server's Kerberos principal name for protocol=org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB is OM/hadoop-master001.kl.man.com@KL.MAN.COM
2023-11-08 08:18:59,569 DEBUG security.SaslRpcClient: Creating SASL GSSAPI(KERBEROS)  client to authenticate to service at hadoop-master001.kl.man.com
2023-11-08 08:18:59,572 DEBUG security.SaslRpcClient: Use KERBEROS authentication for protocol OzoneManagerProtocolPB
2023-11-08 08:18:59,623 DEBUG security.SaslRpcClient: Sending sasl message state: INITIATE
token: "`\202\a\256\006\t*\206H\206\367\022\001\002\002\001\000n\202\a\2350\202\a\231\240\003\002\001\005\241\003\002\001\016\242\a\003\005\000 \000\000\000\243\202\006\267a\202\006\2630\202\006\257\240\003\002\001\005\241\017\033\rKL.MAN.COM\242402\240\003\002\001\000\241+0)\033\002OM\033#hadoop-master001.kl.man.com\243\202\006_0\202\006[\240\003\002\001\021\241\003\002\001\030\242\202\006M\004\202\006I\fc\306\355\342~\332\207\201\325l!\212\227\304\277\333J#\025\221\263+\362?\325\030\2321:#\256%{\266f\037:\331{\306\f\230;U\r$c\312+h|\377Gz\031\377@\367\376\266\210\271A\356:#\000\031\364\333\366[ST#\341R\265u\200\033\265\365jB\023\230\352\277\002\\0\'\222_\317o\031\231_\272\362\231\241B\301\252\366\210bS\020da\367\026\257\377\036\304\370\373\257\314\330\275)a\031K\324\204\302\275\372\025\323\344\003\373\321\'$\225\327A>\275 E\005\177\216\274\255\177\003\"h\030m,\020\220\035\3127)\376\024\360o_\024\030@N\331\365\2554\017\346\270,!\331=>^u\272=\321i$@\246\366f\262\200zn\327\367\315z2\254\3527t\372\362\360\360\317\246\355\225\330\214\004\3264\214\351\213\333[7\212\201<\337\245\364\020q\341\323kO\247v5\322\212xk\247\246\371l\275\244!\353\301r\032K\373T\354D\023\nW\247\312\366\032[7\250l\326\3316\037\320N\211e\002\344\233\215\356\327A\314\230 \252\365b\\\334\211\3569\302qS:V\000\245K\001?\215\226=\224c\231\t\355A\360_\302T\327Ts\350\242\365*I\343Zz\322\213\251\255\324\000\356E\221\220\205\226\312\001\375\213\360\260\343\v\321\204F\253\353\356\341\027\253{\250\254I\326J\321\365\024\304\357\313D\252\273\306\367\305\245\233\267\024 <\314\002\325\256hW\370s\271\372\301\253\336\256\004c\244 \332%\323\320\362=\245\276wY0c\'\003m:\267\362\036\234Q\242d&\316,8\343\206&q\324\306\300u}r^\242\346S\033_K\237t\025\307\347\307~%\207\372a\006\270%\316!.n#7\327\304\331\276\314\262\356!<\2257;\311}\235\346\246\370\377\202\315\317\356v\016\022\361\200\310\232\0240\023p\234\223\357\361\251\325_\227h\367\\\360\227\a\363o\2539Q/C\247`#\301\304\'\201\\\272\362P#\265\2561F\301\364Fx}\302\3550\322\a5w\016\304:LP{\217\032\275[mV:e\227\356\235\214>\327\337K\347T\005\311&z\241\017Q022C\324r\343\302\255V\314r\227.\204j_\345\022\241\2759\n\337\352\224\255jW\276\026\317\005j~p\315\"a\330`\331\300\223Wn\301+\223\244F\364Ew5!\034\362}\262\223\fUbzh\017d)\220b\262\216F\356\231\211\351\241\3459\241\232\360\267u\221\f\311\204\313\323\357\231\263\353\214o\254R\220|2\347x\006x\256#\345\373\016}\327w\016\270:\233\\\321\371h\022-(\210\351nh\254]a\032~\322p\324\001\257\216j\"\020@\326\215\3669\f\326s\273\n\267gA\267\316\022]\212/\221h\303\213;\311\261\344\207\361\257\262\326\032\243pU\325U\002\346\206\310\333\305W\301\246:\201\362D\256\035\303\340\331\345=\231\316(\331\257gY[[\037\277U2TN$\306&\305\340Tl\b\022\264]\264\321h\230\372\354\037<bC\367\364\243u\356\372]\270\255cO&\317l\355bn\257\326R\022\332\201\314\221~\201\325\360\240W\\\200\004 \363\274\374\203C\3379h\220\307O\240a\222\2638\334\354\223\276\306\307C.\bb\220m\340\361X\243\310\214\222\335y\263\023\265\321\206k\324b\337L,1\311\245\235\300\206Z\031 {\361a\300\367Z\360\350@\336\311\366J\336?d1/\377\216\332\023\311Y7\367;\367\220xF4\005R_\264_z\366P\256\220\224\250\a\235\345\266@-\351\222Wx\365\023\255\361\263\260{h\033_\020\006\243\026\376\233\325\027\330\231\25084W\274O-\320u\r$\353\322\252d\305\270\264G\252\036.\265N%\026\324\326\025n\235\372\300{\344-\021\367\324\310\t\tg\211E\246e\320^\242\356\216\"\332\211\a\256\202\375\312\253\374\213\363r6;?!\317B/m\362/\252a!j\330!\355z\v\305\376\t\024\025E|\263\037\211\277d\'\002\001/\262/\t\340\316\325\231\b\356\002\254\265\036@\232\255\r\363\260]\300\354\363\304@\212\264\305D\205\213\315y\356\350\331\343Z\205\t\331\321\016z\320\177\311_\372\323\322\000\205x\331\333?\201l\213\256\236@\213\245d\006\332\331\244\360\274\026\343=\n\025z\020l\250\022\215\244u\336D\272=\214\364P{\034N}\222\b\363\336\272W\002n\320\203\'\231R\274$\336\367\306\365\005G>E\310\375m\214\211\262\347\2013/\331\a\301\251\323i\323\261\263\242\232\267}\fn\355\346\333\277\333\247a\214\351\230AX!\005\002\250\350\343MS\236\351\201\350B\204\246j\316A-\177E\255\367\223\215.\035$\021\034q\'\325\252\274e\r2\271\366\335\242\213~\270<oX0^\274\2174\325\267\203\261 o\201\274*X\377\350\337W\266\326-:\352b\"6\355\272\242\363\224\303\262u\240+/,\372d\302{\271\371\331\022\t:\222\347R)\353\032\327\315\277\373\245a\b\356\026\032\f\335\253\327B\332\270s\025\031\255\356\330\345=1\335\016\223\215\240\216S\267\205\002\244\300\357\000\316\\{\352\311\026y%\216\'\324\331Ch\304`}\356\364\272O\274bQL\023\207\367\340\263\336\241\255a?2<\332/,\262&bINEo\3206\322\353\244,\253a\310Ao\203\217\2535h\201\373\250\257\266_z2,\270:U\026\225\332!N\227\255\024\036\252O\264u\002\234\";\024\225\350\3144\214F\3424\023\362 8\253y\336\231|s\a\375E\3410\255&\a\233\360\336\363\3476\242\201\261\r\245\377\272\207\377\301\367\017Q\203V\321n<\033@aUA\243n{k\213K\330\374T\244\204\247\260Q\030\271\336\371\016\310\367\b[\244\255/\326\217\345\b\216\245\036\273\027D\373\aG\255\304\372\254\330\025\200\360I\021\'\201\n5\352\250d\345t\323S\311&\275,R.\314\343\231T\244\201\3100\201\305\240\003\002\001\021\242\201\275\004\201\272rU\330F\326\t\003\371\245<-9\325s=\335\245\354\244\370\354\376\275\210\022y\306\276\357\a\245(\'D%\250\022\016\333\177\246m\337D\354\261\356\330pBub\t\340\316\374\361\241+??\022&\364\200\375O\226kx\320\016{\357^\210(\233\035\326\v\374\016\344\263\227\267\036\216F\024\333\272\367{\245\217^\257\345\371?\t\353\025\347\027\270\035$\360_\266a\303\236\325\346\222\177\021\316P\2543\233\a8f\232IR/\326C\343\031\004\207\350\212K\277fB\207\277\034\230\200\365\235\251\f\006\276\nZ\004\335\303\004\204Z\333<<\210&\373\202;\3135\301t\310\020\v\206\262\331\277\256\213\221"
auths {
  method: "KERBEROS"
  mechanism: "GSSAPI"
  protocol: "OM"
  serverId: "hadoop-master001.kl.man.com"
}

2023-11-08 08:18:59,638 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: ""

2023-11-08 08:18:59,641 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: "\005\004\000\377\000\f\000\000\000\000\000\000/C.\372\001\001\000\000\335\326\3566\033{f\271Z}\304\247"

2023-11-08 08:18:59,642 DEBUG ipc.Client: Negotiated QOP is :auth
2023-11-08 08:18:59,648 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master001.kl.man.com/10.10.1.104:9862 from s.p.cher@KL.MAN.COM: starting, having connections 1
2023-11-08 08:18:59,652 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master001.kl.man.com/10.10.1.104:9862 from s.p.cher@KL.MAN.COM sending #0 org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2023-11-08 08:18:59,667 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master001.kl.man.com/10.10.1.104:9862 from s.p.cher@KL.MAN.COM got value #0
2023-11-08 08:18:59,669 DEBUG ha.OMFailoverProxyProviderBase: RetryProxy: OM om1: RemoteException: OM:om1 is not the leader. Could not determine the leader node.
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:248)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createLeaderErrorException(OzoneManagerProtocolServerSideTranslatorPB.java:235)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:228)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:175)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)

2023-11-08 08:18:59,671 DEBUG ha.OMFailoverProxyProviderBase: Incrementing OM proxy index to 1, nodeId: om2
2023-11-08 08:18:59,672 DEBUG retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:om1 is not the leader. Could not determine the leader node.
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:248)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createLeaderErrorException(OzoneManagerProtocolServerSideTranslatorPB.java:235)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:228)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:175)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)
, while invoking $Proxy15.submitRequest over nodeId=om1,nodeAddress=hadoop-master001.kl.man.com:9862. Trying to failover immediately.
com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:om1 is not the leader. Could not determine the leader node.
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:248)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createLeaderErrorException(OzoneManagerProtocolServerSideTranslatorPB.java:235)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:228)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:175)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:249)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:122)
	at com.sun.proxy.$Proxy15.submitRequest(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy15.submitRequest(Unknown Source)
	at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
	at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
	at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
	at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:114)
	at org.apache.hadoop.ozone.shell.OzoneAddress.createClientForS3Commands(OzoneAddress.java:190)
	at org.apache.hadoop.ozone.shell.tenant.TenantHandler.createClient(TenantHandler.java:51)
	at org.apache.hadoop.ozone.shell.Handler.call(Handler.java:94)
	at org.apache.hadoop.ozone.shell.Handler.call(Handler.java:44)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.lambda$execute$0(TenantShell.java:44)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:162)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.execute(TenantShell.java:43)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.main(TenantShell.java:53)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMNotLeaderException): OM:om1 is not the leader. Could not determine the leader node.
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:248)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createLeaderErrorException(OzoneManagerProtocolServerSideTranslatorPB.java:235)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:228)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:175)
	at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
	at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
	at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)

	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
	at org.apache.hadoop.ipc.Client.call(Client.java:1558)
	at org.apache.hadoop.ipc.Client.call(Client.java:1455)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
	... 35 more
2023-11-08 08:18:59,673 DEBUG ha.OMFailoverProxyProviderBase: Failing over OM from om1:0 to om2:1
2023-11-08 08:18:59,674 DEBUG ipc.Client: getting client out of cache: Client-2cc893738d9e4ad39b3a816ae3b4d1b6
2023-11-08 08:18:59,674 DEBUG ipc.Client: The ping interval is 60000 ms.
2023-11-08 08:18:59,674 DEBUG ipc.Client: Connecting to hadoop-master002.kl.man.com/10.10.1.225:9862
2023-11-08 08:18:59,674 DEBUG ipc.Client: Setup connection to hadoop-master002.kl.man.com/10.10.1.225:9862
2023-11-08 08:18:59,675 DEBUG security.UserGroupInformation: PrivilegedAction [as: s.p.cher@KL.MAN.COM (auth:KERBEROS)][action: org.apache.hadoop.ipc.Client$Connection$2@5f9be66c]
java.lang.Exception
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:839)
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1677)
	at org.apache.hadoop.ipc.Client.call(Client.java:1502)
	at org.apache.hadoop.ipc.Client.call(Client.java:1455)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:122)
	at com.sun.proxy.$Proxy15.submitRequest(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
	at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
	at com.sun.proxy.$Proxy15.submitRequest(Unknown Source)
	at org.apache.hadoop.ozone.om.protocolPB.Hadoop3OmTransport.submitRequest(Hadoop3OmTransport.java:80)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.submitRequest(OzoneManagerProtocolClientSideTranslatorPB.java:289)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.getServiceInfo(OzoneManagerProtocolClientSideTranslatorPB.java:1454)
	at org.apache.hadoop.ozone.client.rpc.RpcClient.<init>(RpcClient.java:239)
	at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:247)
	at org.apache.hadoop.ozone.client.OzoneClientFactory.getRpcClient(OzoneClientFactory.java:114)
	at org.apache.hadoop.ozone.shell.OzoneAddress.createClientForS3Commands(OzoneAddress.java:190)
	at org.apache.hadoop.ozone.shell.tenant.TenantHandler.createClient(TenantHandler.java:51)
	at org.apache.hadoop.ozone.shell.Handler.call(Handler.java:94)
	at org.apache.hadoop.ozone.shell.Handler.call(Handler.java:44)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
	at picocli.CommandLine.access$1300(CommandLine.java:145)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
	at picocli.CommandLine.execute(CommandLine.java:2078)
	at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.lambda$execute$0(TenantShell.java:44)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:162)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.execute(TenantShell.java:43)
	at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
	at org.apache.hadoop.ozone.shell.tenant.TenantShell.main(TenantShell.java:53)
2023-11-08 08:18:59,675 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE

2023-11-08 08:18:59,678 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.ozone.security.OzoneDelegationTokenSelector)
2023-11-08 08:18:59,678 DEBUG security.OzoneDelegationTokenSelector: Got tokens: null for service 10.10.1.225:9862
2023-11-08 08:18:59,678 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one
2023-11-08 08:18:59,678 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=, serverPrincipal=ozone.om.kerberos.principal)
2023-11-08 08:18:59,678 DEBUG security.SaslRpcClient: getting serverKey: ozone.om.kerberos.principal conf value: OM/_HOST@KL.MAN.COM principal: OM/hadoop-master002.kl.man.com@KL.MAN.COM
2023-11-08 08:18:59,678 DEBUG security.SaslRpcClient: RPC Server's Kerberos principal name for protocol=org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolPB is OM/hadoop-master002.kl.man.com@KL.MAN.COM
2023-11-08 08:18:59,678 DEBUG security.SaslRpcClient: Creating SASL GSSAPI(KERBEROS)  client to authenticate to service at hadoop-master002.kl.man.com
2023-11-08 08:18:59,679 DEBUG security.SaslRpcClient: Use KERBEROS authentication for protocol OzoneManagerProtocolPB
2023-11-08 08:18:59,688 DEBUG security.SaslRpcClient: Sending sasl message state: INITIATE
token: "`\202\a\256\006\t*\206H\206\367\022\001\002\002\001\000n\202\a\2350\202\a\231\240\003\002\001\005\241\003\002\001\016\242\a\003\005\000 \000\000\000\243\202\006\267a\202\006\2630\202\006\257\240\003\002\001\005\241\017\033\rKL.MAN.COM\242402\240\003\002\001\000\241+0)\033\002OM\033#hadoop-master002.kl.man.com\243\202\006_0\202\006[\240\003\002\001\021\241\003\002\001\030\242\202\006M\004\202\006I\267s\320\177w\255*F/\337\314>\252\016\215\276\307\200\376\362\327L2N\375\313\000\"\'H\214\301\257U\001\361\031\324k\316$\027\267\244\205\313\231\005\032O\300R\364\343\022Q\264=\202v\230\fR\272\023y\205e\356\364Q\357\\6 r\375\'h\250__\251\344\023\310p\324\332}\344n\254)\3765\275,\215\226\2657\234\3043\247\314zL\200\037\327\223\002\367\250\365\'\330\002\376e\f<\025E\364>LC\267\3370\377\324Hf\332MZ\337\372\215#\236bV\361$\327\343w\245\321\0013=\343\002\345+@\230g2\257?\255\200\2766;\213\021\370\237E\332B\237\223}\206K6\260fX\213\221w\361\243\354O\234H\021U\021\300\334\344\2601\254\223\2144\030\177\227Ws\330\b\225]D\a\367\263\235N\220\t\234\2671\265\303\217\261P\001\n\264\250\345>\355(* \350}\212Sc\266\200\337\'\275x\247\221Y\347\254\002\350L^\225V\202\3434Zx]f\210pv6\321V\035y\344\373\332a\314\027\334\334\016\215v\272\244\337\225\353\002\0330\311\207\347\360\377-1\267;\373\2259T\230\003\336\374WU\235\216\310my\372\231K\321C\034\037);G\36112[\222S\246\356\222\322\020\033w\375u\365\241\237\035\246\a\223m\374\244a\\w\367\373\364\227q\327\250s\'t\021G\231\355\322 \000\350\323+\271\312\276Rr\326\256\351\016d\004wh\"Bp=\005\204X\005/\276<\307\267\232\347\3639aw\006]s\276\031`G\376\246\213\356\"\302VO`\255\345\262Z5\177\232\203\270Pq\302>\233\264Zvz\32679\247\350\300\346n\263\272*\222\367=\344C\257\t\024\224U\331\254\337\247\224\340#G\344\266[\231\310\243lL\200\263\360\200\033E*m\275\333=\221@~\265\375\327\b\206\aX2F\347K\250o\216\214\023\fl\033\314$\306\264J\232e\337\312\177\227\3671\242\324_P\\\360tC\222\326M8\177uOf\234\234\023\351\030\263\006h?\020\304\363r\332^\205c\247\345\021|/\230E]I}4\252\321\202\341n\016\202\"\245\217\313\001,\326g\275\375E\225w\373[\024\"\374\026%\a\267}19\033\211!l\351CY[\265\v\312\f\232\334\376\346\261\033\275\376q\201\226PtK.\000\233\254\241\240~\324o\223\031\331\006\247\207\270\224\020\030QJ\215rw[u\317I\341x|+\321\232+\2119\232\272\266b\222\026\237#\361=\352:\201}Z\024\036N\215\307Sa>\305?\033=\357\223\276\306\310!\223\001J\314\274\241\237\231`\370w-\205\301i\366A\3177\304%3\360J\340\255\306o\003\337\271yF~\243-\020\016\305{\266\330c\024\251\375\232\243\211S\247\204\312}\027\036\310\231Q\372\355]\342\263\275|%\213a\220\b\225\264jb\037\346\274\0248v\023j5\253\vM\362-.\027[\017\251\255o\005|\372\300A\267\334\n\246e\360$g\363\031\303\n\262\036\377?\fyWfc\3708\346\237%\f\244\270\243Y\326\2701\253\231)\n=_8\377\376\240\327\371\026?\223\360\335\204h\207\314`\321\324\243\325\255\255[8:\213\315\230\323\\\211\273\237w\347\215\332\333q\275\257\253E\r\025o:\272\b\320\212\354\273\337\250\216\036\353Q\341\326@<d\024\277\331]\301y\310\276={\351%-\356h\200j\307\r\3769\031a\303 \0214#t\222\327\030\207\372\200\320U\275?6^\241\377f.\322{}y,\330\361[\020\252r\204@\316,\3077\\\335_)\206\370Z\305&+u\237\372\257\315\305=\360\347\264\316Bs\177\357\367\351T\221tw\313M\345|1\337\263!d\210\v\220\351ZG\255\336\3538k\220\355\215\f6\254\177g\270\337M^\031A[*\261vP\212l&.\334\221\024\353_\334p\333\307\374\211\'\276 x\033[\373$~W\317\3406!\334\273\005\232`\237\361\020\232\262\375i\212\353Dj\313V\fQ\316\220Q\321\264\341)\321+\262\205S$\031d\264\223\037\017\237l\356)\323vT\2065\352\244\250\245\343\362\340\000\355u\001\357\345:\031\001_\364K\022\320\245\360\201\267\005}Y\030-Q\360<\223\n\n\376U\310\217\273\343\320\372T\303T\257Y&\250\315\254\034\342\272\274\254\f\316D\376\355\310\r\002\300\240\277\003-C\210\a1\235=\a\311\253\222\302\262\253\202\256%X\372G\247\207\310\310\025\346I\344\251\374\262W\255\242=\006\273s4\303\306\2735b\302c\vx?\202\031gk\245\340\361\214\322\357\026\2632\312l\344\206\267%$\256\032\327\353G\306\254=T\324\302\267\324h\355<D\277\213\216\327\034\200\n\252H\237;B\364\374w\222\322\233\3248CHq\027\354\314AIn\324\323}#\214\340\355J\220\001\207!\365\243\220Y`\r\3218\365Eu\020\021S\256\t@RF\"\267\241\225\333E\006\342AZ\026z\266\225_\266\3120\227\240\3644\306\263,\317\324\257\241\272\353\265\262\331\016^\033\341\275\311\3531\320\003\022\274\262,\266U\271\202\0238\306\031\267%\245?\306\372\224ac#\026\331f\022\002\006\255\214\004\024\317S\222f\333\304\263\223\'\204s\016\321&PLK\342~Xw\231jc<1P\030\003EKo\361*,\350\257QV\272y\3352\034=\321\343c\253)Z\330\273W#\267R\312\313\251\3625}@\334\036\025\022\031y\'M\344\230\3434\037sc\244\247\371\326\250]2\236\031!\270\204\020\002euW;\223\202\031lj\320\272\210_\342\263*\344\354^\322\321\353<\214E\v_2t[Z\n2\t\313w\031\267\343\311in@W\357\020\237\314\0266\204eC\t\034\310\033\342) \261\356\027\266\341kE\2622\263\372\312dj\244j\343%.\332m\324\'\b\242\025dH\256\301\203W\022\266\031\\\253\267^h9\000\246i.\037\337\244\201\3100\201\305\240\003\002\001\021\242\201\275\004\201\272\340\223st(\306\033\350\232\022\363-\307{\345\223\300%\333\030\325\303)\3001\022\301\021\347|\036\307\350\033\265j\341\304\347\300o\226\3729Zo\325\216\376\363=\336\222}9\222\246\004\345\310E~HS\300,\330\223<\330\216^\202\027\313\n\034xf\222\231,\271\216\234\232\357^(\024\330\225\363\376\303\307\357\314\253lu\022\031\247a\356\032G\264\266\343\377>\236\311\2769\220-\231_\350!p>\302\312\213\022\231\321O,\321\335\256L@\001u\004k\313\005\344\031Wb{\215\024\230=\273\346\261\244%n6\rd?\244\230\276_\200Lj\265\263i\353\2773\245O\345\366\226\365Rv\324\271"
auths {
  method: "KERBEROS"
  mechanism: "GSSAPI"
  protocol: "OM"
  serverId: "hadoop-master002.kl.man.com"
}

2023-11-08 08:18:59,697 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: ""

2023-11-08 08:18:59,698 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: "\005\004\000\377\000\f\000\000\000\000\000\000\025\323\304+\001\001\000\000\320\272\271\3326\t\277\033\376h\207}"

2023-11-08 08:18:59,698 DEBUG ipc.Client: Negotiated QOP is :auth
2023-11-08 08:18:59,699 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master002.kl.man.com/10.10.1.225:9862 from s.p.cher@KL.MAN.COM: starting, having connections 2
2023-11-08 08:18:59,699 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master002.kl.man.com/10.10.1.225:9862 from s.p.cher@KL.MAN.COM sending #0 org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2023-11-08 08:18:59,703 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master002.kl.man.com/10.10.1.225:9862 from s.p.cher@KL.MAN.COM got value #0
2023-11-08 08:18:59,704 DEBUG ipc.ProtobufRpcEngine: Call: submitRequest took 30ms
2023-11-08 08:18:59,944 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableCounterLong org.apache.hadoop.hdds.scm.ContainerClientMetrics.totalWriteChunkBytes with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[])
2023-11-08 08:18:59,945 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableCounterLong org.apache.hadoop.hdds.scm.ContainerClientMetrics.totalWriteChunkCalls with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName=Ops, valueName=Time, about=, interval=10, type=DEFAULT, value=[])
2023-11-08 08:18:59,946 DEBUG impl.MetricsSystemImpl: ContainerClientMetrics1, Ozone Client Metrics
2023-11-08 08:18:59,968 INFO rpc.RpcClient: Creating Tenant: 'tenant4', with new volume: 'tenant4'
2023-11-08 08:18:59,969 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master002.kl.man.com/10.10.1.225:9862 from s.p.cher@KL.MAN.COM sending #1 org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol.submitRequest
2023-11-08 08:18:59,981 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master002.kl.man.com/10.10.1.225:9862 from s.p.cher@KL.MAN.COM got value #1
2023-11-08 08:18:59,981 DEBUG ipc.ProtobufRpcEngine: Call: submitRequest took 12ms
2023-11-08 08:18:59,983 DEBUG ipc.Client: stopping client from cache: Client-2cc893738d9e4ad39b3a816ae3b4d1b6
2023-11-08 08:18:59,984 DEBUG ipc.Client: stopping client from cache: Client-2cc893738d9e4ad39b3a816ae3b4d1b6
2023-11-08 08:18:59,984 DEBUG ipc.Client: removing client from cache: Client-2cc893738d9e4ad39b3a816ae3b4d1b6
2023-11-08 08:18:59,985 DEBUG ipc.Client: stopping actual client because no more references remain: Client-2cc893738d9e4ad39b3a816ae3b4d1b6
2023-11-08 08:18:59,985 DEBUG ipc.Client: Stopping client
2023-11-08 08:18:59,985 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master001.kl.man.com/10.10.1.104:9862 from s.p.cher@KL.MAN.COM: closed
2023-11-08 08:18:59,985 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master002.kl.man.com/10.10.1.225:9862 from s.p.cher@KL.MAN.COM: closed
2023-11-08 08:18:59,985 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master001.kl.man.com/10.10.1.104:9862 from s.p.cher@KL.MAN.COM: stopped, remaining connections 0
2023-11-08 08:18:59,985 DEBUG ipc.Client: IPC Client (464676531) connection to hadoop-master002.kl.man.com/10.10.1.225:9862 from s.p.cher@KL.MAN.COM: stopped, remaining connections 0
TENANT_AUTHORIZER_ERROR java.io.IOException: org.apache.ranger.RangerServiceException: Ranger API org.apache.ranger.RangerClient$API@4a92034f failed: statusCode=401, status=Unauthorized, response:

[netscrol2016]

@dombizita updated ranger to version 2.4.0 the problem remained

[netscrol2016]

The problem was solved by setting the export OZONE_CLASSPATH=/opt/ozone/lib/libext/* variable and reinstalling the plugin again.

[DaveTeng0]

hm, interesting, @netscrol2016 how do you find that env solves your problem? (I guess your ozone jars are in /opt/ozone/lib/libext/* right?) From what I understand, that 'OZONE_CLASSPATH' provides additional, custom CLASSPATH.
May we know what're these env value from your cluster?

• HADOOP_HOME
• HADOOP_CLASSPATH
• OZONE_HOME
• HDDS_LIB_JARS_DIR
• OZONE_LIBEXEC_DIR
  Thanks!

[netscrol2016]

hi @DaveTeng0 I read about it here https://ozone.apache.org/docs/1.3.0/security/securitywithranger.html and decided to add it directly to the environment for all users in etc/profile.d and it worked after reboot.

# export HADOOP_HOME=
# export HADOOP_CLASSPATH="/some/cool/path/on/your/machine"
export OZONE_HOME=/opt/ozone

HDDS_LIB_JARS_DIR and OZONE_LIBEXEC_DIR not have.

[netscrol2016]

This env from file /etc/hadoop/conf/hadoop-env.sh and /etc/hadoop/conf/ozone-env.sh