RequestDispatcher#forward() does not work properly on java21 EE8

[miztaka]

Summary

I have an appengine app with legacy bundle which has been migrated from java8 to java17.
Now I'm testing to migrate to java21 and faced this problem, where forwarded request does not seem to be handled correctly and returns 404 error.

Sample Code

To simplify this problem, I have made a simple code bellow where TestRewriteFilter forwards request to the new URI and TestActionFilter does return the response.

TestRewriteFilter

public class TestRewriteFilter implements Filter {

  private static final Log log = LogFactory.getLog(TestRewriteFilter.class);

  protected ServletContext servletContext;

  @Override
  public void init(FilterConfig filterConfig) throws ServletException {
    servletContext = filterConfig.getServletContext();
  }

  @Override
  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
      throws IOException, ServletException {
    HttpServletRequest req = (HttpServletRequest) request;
    String uri = req.getRequestURI();
    log.info("Request URI: " + uri);
    if (uri.startsWith("/tenant/")) {
      log.info("Continue chain");
      chain.doFilter(request, response);
    } else {
      uri = "/tenant" + uri;
      log.info("Forward to: " + uri);
      req.getRequestDispatcher(uri).forward(request, response);
      //servletContext.getRequestDispatcher(uri).forward(request, response);
    }
  }

  @Override
  public void destroy() {
  }

}

TestActionFilter

public class TestActionFilter implements Filter {

  private final static Log log = LogFactory.getLog(TestActionFilter.class);

  @Override
  public void init(FilterConfig filterConfig) throws ServletException {
  }

  @Override
  public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
      throws IOException, ServletException {
    HttpServletRequest req = (HttpServletRequest) request;
    HttpServletResponse resp = (HttpServletResponse) response;
    String uri = req.getRequestURI();
    log.info("URI to execute: " + uri);
    if (!uri.startsWith("/tenant/")) {
      resp.sendError(403, "unauthorized access");
      return;
    }
    resp.getWriter().write("Hello " + uri);
  }

  @Override
  public void destroy() {
  }

}

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.jcp.org/xml/ns/javaee" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" version="3.1">
  <filter>
    <filter-name>TestRewriteFilter</filter-name>
    <filter-class>jp.honestyworks.powerapps.forum.servlet.TestRewriteFilter</filter-class>
  </filter>
  <filter>
    <filter-name>TestActionFilter</filter-name>
    <filter-class>jp.honestyworks.powerapps.forum.servlet.TestActionFilter</filter-class>
  </filter>
  <filter-mapping>
    <filter-name>TestRewriteFilter</filter-name>
    <url-pattern>/*</url-pattern>
    <dispatcher>REQUEST</dispatcher>
    <dispatcher>FORWARD</dispatcher>
    <dispatcher>INCLUDE</dispatcher>
    <dispatcher>ERROR</dispatcher>
  </filter-mapping>
  <filter-mapping>
    <filter-name>TestActionFilter</filter-name>
    <url-pattern>/*</url-pattern>
    <dispatcher>REQUEST</dispatcher>
    <dispatcher>FORWARD</dispatcher>
    <dispatcher>INCLUDE</dispatcher>
    <dispatcher>ERROR</dispatcher>
  </filter-mapping>
</web-app>

appengine-web.xml

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <runtime>java21</runtime>
  <app-engine-apis>true</app-engine-apis>

  <service>default</service>
  <instance-class>F2</instance-class>
  <automatic-scaling>
    <min-idle-instances>0</min-idle-instances>
    <max-idle-instances>1</max-idle-instances>
    <min-pending-latency>6s</min-pending-latency>
    <max-pending-latency>6s</max-pending-latency>
  </automatic-scaling>

  <precompilation-enabled>true</precompilation-enabled>

  <system-properties>
    <property name="appengine.use.EE8" value="true"/>
    <property name="appengine.use.httpconnector" value="true"/>
    <property name="java.util.logging.config.file" value="WEB-INF/logging.properties" />
  </system-properties>

  <sessions-enabled>true</sessions-enabled>
  <warmup-requests-enabled>true</warmup-requests-enabled>
  <static-files>
    <include path="/css/**" />
    <include path="/js/**" />
    <include path="/img/**" />
    <include path="/static/**" />
    <include path="/v2/**" />
    <include path="/favicon.ico" />
    <include path="/robots.txt" />
  </static-files>

</appengine-web-app>

Expected results

You can get expected result by executing it on local devserver.
You can see the text "Hello /tenant/foo" on your browser.
Application log looks like:

devserver-1  | Oct 23, 2025 3:15:31 AM jp.honestyworks.powerapps.forum.servlet.TestRewriteFilter doFilter
devserver-1  | INFO: Request URI: /foo
devserver-1  | Oct 23, 2025 3:15:31 AM jp.honestyworks.powerapps.forum.servlet.TestRewriteFilter doFilter
devserver-1  | INFO: Forward to: /tenant/foo
devserver-1  | Oct 23, 2025 3:15:31 AM jp.honestyworks.powerapps.forum.servlet.TestRewriteFilter doFilter
devserver-1  | INFO: Request URI: /tenant/foo
devserver-1  | Oct 23, 2025 3:15:31 AM jp.honestyworks.powerapps.forum.servlet.TestRewriteFilter doFilter
devserver-1  | INFO: Continue chain
devserver-1  | Oct 23, 2025 3:15:31 AM jp.honestyworks.powerapps.forum.servlet.TestActionFilter doFilter
devserver-1  | INFO: URI to execute: /tenant/foo

Actual results

However on production server, you get 404 error.
Application log looks like:

2025-10-23 16:18:37.420
Started JettyServerConnectorWithReusePort@7dda48d9{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
2025-10-23 16:18:37.431
Started cgarj.JettyServletEngineAdapter$@e383572{STARTING}[12.0.26,sto=0] @3064ms
2025-10-23 16:18:38.335
Session workerName=node0
2025-10-23 16:18:38.547
Started oeje8n.ContextHandler$CoreContextHandler@28ef2851{ROOT,/,b=file:///workspace/,a=AVAILABLE,h=oeje8n.ContextHandler$CoreContextHandler$CoreToNestedHandler@541814a3{STARTED}}
2025-10-23 16:18:38.989
Request URI: /foo
2025-10-23 16:18:38.989
Forward to: /tenant/foo

Production version

2025-10-23 16:18:34.785
INFO: appengine runtime jars built on 2025-10-13T05:21:02Z from commit cf2bdf5, version 3.0.0-SNAPSHOT
2025-10-23 16:18:34.849
INFO: AppEngine is using jetty 12. EE8 profile.
2025-10-23 16:18:37.338
jetty-12.0.26; built: 2025-09-04T00:15:57.829Z; git: 6d5b3cfe27f67b72e6f393025eb93655a5406991; jvm 21.0.8+9-Ubuntu-0ubuntu122.04.1

[lachlan-roberts]

Thanks for the detailed report.

This looks to be a regression in the Jetty quickstart between Jetty 9.4 and Jetty 12.0. I can see in the generated quickstart-web.xml file there are no <dispatcher> tags at all under the filter mapping. I will work on a fix for this in Jetty.

[ludoch]

Yes, same with Jetty 12.1.3 as well. Jetty 9.x is fine

[ludoch]

Fix is now pushed to Jetty new version.
We have the GAE runtime build using the new Jetty, hopefully the production push will happen next week and will last 5 days. But real issue was in client side, so Cloud CLI tooling used via Maven or Gradle builds.
Unless unrelated blocker, you can try again with the newer Cloud CLI build from next Tuesday afternoon (PST)

[ludoch]

built 548 of Cloud CLI next Tuesday will have the fix.
Track https://docs.cloud.google.com/sdk/docs/release-notes