[Bug][Manager] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin #11412

PeterZh6 · 2024-10-27T08:16:17Z

What happened

Summary

When starting Inlong with Docker, the manager continuously throws an exception, leading to system malfunction. This issue was previously mentioned in Issue #11071. However, the associated PR only addressed a null check without resolving the core issue. Reopening this ticket for further investigation.

Description

After deploying Inlong with Docker, the manager produces the following error:

[ ] 2024-10-27 07:19:22.293 -ERROR [http-nio-8083-exec-4] m.s.c.InlongClusterServiceImpl:938 - group not found by groupId=inlong_agent_system 
[ ] 2024-10-27 07:19:22.294 -ERROR [http-nio-8083-exec-4] w.c.ControllerExceptionHandler:129 - failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin

This error prevents the manager from functioning, rendering the entire system unusable.

Click to view complete log

[ ] 2024-10-27 07:19:22.293 -ERROR [http-nio-8083-exec-4] m.s.c.InlongClusterServiceImpl:938 - group not found by groupId=inlong_agent_system 
[ ] 2024-10-27 07:19:22.294 -ERROR [http-nio-8083-exec-4] w.c.ControllerExceptionHandler:129 - failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin 
org.apache.inlong.manager.common.exceptions.BusinessException: group not found by groupId=inlong_agent_system
        at org.apache.inlong.manager.service.cluster.InlongClusterServiceImpl.getDataProxyNodes(InlongClusterServiceImpl.java:939) ~[manager-service-2.0.0.jar:2.0.0]
        at org.apache.inlong.manager.service.cluster.InlongClusterServiceImpl$$FastClassBySpringCGLIB$$5ac56ec1.invoke() ~[manager-service-2.0.0.jar:2.0.0]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.34.jar:5.3.34]
        at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:704) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.apache.inlong.manager.service.cluster.InlongClusterServiceImpl$$EnhancerBySpringCGLIB$$8a4f27dd.getDataProxyNodes() ~[manager-service-2.0.0.jar:2.0.0]
        at sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_342]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_342]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.27.jar:5.3.27]
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-5.3.27.jar:5.3.27]
        at com.sun.proxy.$Proxy210.getDataProxyNodes(Unknown Source) ~[?:?]
        at org.apache.inlong.manager.web.controller.openapi.DataProxyController.getIpList(DataProxyController.java:59) ~[manager-web.jar:2.0.0]
        at sun.reflect.GeneratedMethodAccessor296.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_342]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_342]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) ~[tomcat-embed-core-9.0.75.jar:4.0.FR]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.27.jar:5.3.27]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-embed-core-9.0.75.jar:4.0.FR]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.filter.TenantInsertionFilter.doFilter(TenantInsertionFilter.java:71) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.inlong.manager.web.auth.tenant.TenantAuthenticatingFilter.doFilter(TenantAuthenticatingFilter.java:81) ~[manager-web.jar:2.0.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.inlong.manager.web.auth.openapi.OpenAPIFilter.doFilter(OpenAPIFilter.java:84) ~[manager-web.jar:2.0.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) ~[shiro-core-1.13.0.jar:1.13.0]
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) ~[shiro-core-1.13.0.jar:1.13.0]
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) ~[shiro-core-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.13.0.jar:1.13.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.trace.TraceFilter.doFilter(TraceFilter.java:80) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.filter.EmptyTenantFilter.doFilter(EmptyTenantFilter.java:50) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.inlong.manager.web.filter.HttpServletRequestFilter.doFilter(HttpServletRequestFilter.java:45) ~[manager-web.jar:2.0.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.34.jar:5.3.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.34.jar:5.3.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.34.jar:5.3.34]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.34.jar:5.3.34]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.75.jar:9.0.75]
        at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]

Here is a screenshot explaining what's going on.

What you expected to happen

In version 1.13.0 of Inlong Docker, this issue does not occur. The latest version seems to introduce a change where the Inlong Agent module queries the inlong_agent_system group, which has not been created. This results in a request the manager cannot handle.

Possible reasons of this issue

In PR #10757, a Sender object was introduced with group_id=INLONG_AGENT_SYSTEM, created with DefaultMessageSender. Within this setup:

The group_id=INLONG_AGENT_SYSTEM is passed to the Sender constructor with ProxyClientConfig. The Sender constructor will instantiate ClientMgr.
After the instantiation, ClientMgr invokes the getGroupIdConfigureInfo() method, which likely triggers the root exception due to an attempt to request a configuration for the non-existent INLONG_AGENT_SYSTEM group by calling requestProxyList() eventually.

How to reproduce

Pull the latest Inlong code.
Deploy Inlong using Docker:
- Navigate to docker/docker-compose and follow README.md instructions.
- Copy apache_inlong_manager.sql and apache_inlong_audit_mysql.sql into a sql directory under docker/docker-compose.
Run docker-compose up -d.
Enter the Inlong manager container with docker exec -it manager and inspect logs/manager-all.log, or use the Logs option in Docker Desktop if available.
Locate the error messages as detailed above.

Environment

WSL2 Ubuntu 22.04.4 LTS
Docker Engine v25.0.3

InLong version

master

InLong Component

InLong Manager

Are you willing to submit PR?

Yes, I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

fuweng11 · 2024-10-27T15:09:41Z

This exception will not leading to system malfunction. What problem do you want to solve?

PeterZh6 · 2024-10-27T16:50:15Z

This exception will not leading to system malfunction. What problem do you want to solve?

Thanks for your attention :)

Firstly, this exception keeps popping up once the manager container is started. It is not only annoying as it fills the screen with exception message, but also does contains unwanted behaviour as mentioned above.

Secondly, the configuration of data syncronization job fails immediately after it is approved, while this problem didn't exist on the 1.13.0 version of docker image. I am not sure whether I'm doing it right. It would be great if you could try to configure a data syncronization job with docker deployment of inlong personally and check whether this issue will cause the system to malfunction.

I am try to reproduce this example but encountered this issue.

PeterZh6 · 2024-10-28T12:06:10Z

This exception will not leading to system malfunction. What problem do you want to solve?

I have gone over the exception problem again. This Exception doesn't cause the system to malfunction. I am sorry for the mistake.

PeterZh6 added the type/bug Something is wrong label Oct 27, 2024

PeterZh6 changed the title ~~[Bug][Manger] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin~~ [Bug][Manager] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin Oct 27, 2024

justinwwhuang mentioned this issue Oct 28, 2024

[INLONG-11412][Agent] Do not report Agent status and file metrics #11414

Merged

3 tasks

dockerzhang added this to the 2.1.0 milestone Oct 28, 2024

dockerzhang closed this as completed in #11414 Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug][Manager] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin #11412

[Bug][Manager] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin #11412

PeterZh6 commented Oct 27, 2024

fuweng11 commented Oct 27, 2024

PeterZh6 commented Oct 27, 2024 •

edited

Loading

PeterZh6 commented Oct 28, 2024

[Bug][Manager] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin #11412

[Bug][Manager] Failed to handle request on path: /inlong/manager/openapi/dataproxy/getIpList/inlong_agent_system by user: admin #11412

Comments

PeterZh6 commented Oct 27, 2024

What happened

Summary

Description

What you expected to happen

Possible reasons of this issue

How to reproduce

Environment

InLong version

InLong Component

Are you willing to submit PR?

Code of Conduct

fuweng11 commented Oct 27, 2024

PeterZh6 commented Oct 27, 2024 • edited Loading

PeterZh6 commented Oct 28, 2024

PeterZh6 commented Oct 27, 2024 •

edited

Loading