URL Rewriting doesn't work together with Yarp Proxy Forwarding

[sgarnovsky]

Describe the bug

I reproduced it with the Blazor static SSR app.
I've been migrating an old ASP.Net WebForms app to Blazor static SSR.
Using Yarp to proxy not migrated page requests to the ASP.Net WebForms app.

Rewritten URLs to the new Blazor static SSR app are trying to be proxied to the old ASP.Net Webforms app.

To Reproduce

Here is an app initialization code I use:

services.AddReverseProxy()
  .LoadFromConfig(Configuration.GetSection("Migration:ReverseProxy"));
....
var rewriteOptions = new RewriteOptions()
    // static cache folder rewrite rule
    .Add(UrlRewritingUtility.RewriteCachedStaticFilePathRequests);
app.UseRewriter(rewriteOptions);
....
app.MapReverseProxy();

RewriteCachedStaticFilePathRequests - a simple rule to remove the virtual cache folder for the static resource urls:
something like:
https://localhost:7159/cache/211108_0636182/img/logo.png
to
https://localhost:7159/img/logo.png

public static class UrlRewritingUtility
{
    private static Regex StaticFilesCachedPathRegex = new Regex(@"^(/cache/\d{6}_\d{7})(/.+)$",
      RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.Singleline);

    public static void RewriteCachedStaticFilePathRequests(RewriteContext context)
    {
        var request = context.HttpContext.Request;

        var path = request.Path.Value;
        if (path is null || path.Length <= 21 || !path.Contains("/cache/", StringComparison.CurrentCultureIgnoreCase))
            return;

        var m = StaticFilesCachedPathRegex.Match(path);

        if (m.Success)
        {
            context.Result = RuleResult.SkipRemainingRules;
            request.Path = m.Groups[2].Value;
        }
    }
}

Forwarding Proxy configuration is:

"Migration": {
    "ReverseProxy": {
        "Routes": {
            "fallbackRoute": {
                "ClusterId": "fallbackCluster",
                "Order": "1",
                "Match": {
                    "Path": "/{**catch-all}"
                    //"Path": "/AppVDir/{**catch-all}" // virtual dir is here as w/o it UseRewriter doesn't work as expected.
                }
            }
        },
        "Clusters": {
            "fallbackCluster": {
                "Destinations": {
                    "fallbackApp": {
                        "Address": "https://localhost/" // should be w/o vdir if app.MapReverseProxy(); is used
                    }
                }
            }
        }
    }
},

Url Rewriter doesn't work correct if Yarp proxing is enabled.
It seems that Url is rewritten correct but the Forwarding Proxy still decided to send this rewritten request to the old app.

I figured out that if Match Path is set to: "/{**catch-all}" URL Rewriting doesn't work as expected. Setting it to the "/AppVDir/{**catch-all}" fixes the issue. Really, it works only in the development environment only in this case.

Further technical details

.Net 8, Yarp.ReverseProxy 2.1.0

[MihaZupan]

Can you enable Debug logs? That should give you a hint as to which/when routing decisions were taken.

[sgarnovsky]

Yes, sure, I tried to enable logging debug entries for the referenced packages.
There was no any useful info.

After I found this article: Diagnosing YARP-based proxies

I applied to log RewriteMiddleware entries as well as "Forwarder Telemetry" data:

dbug: Microsoft.AspNetCore.Rewrite.RewriteMiddleware[3]
      Request is done applying rules. Url was rewritten to https://192.168.102.9:7159/img/logo.png
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Host: 192.168.102.9:7159
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
Cookie: <...some value is here...>
Referer: https://192.168.102.9:7159/
Upgrade-Insecure-Requests: 1
sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-site: same-origin
sec-fetch-mode: navigate
sec-fetch-user: ?1
sec-fetch-dest: document
priority: u=0, i
{"RouteId":"fallbackRoute","Match":{"Methods":null,"Hosts":null,"Path":"/{**catch-all}","QueryParameters":null,"Headers":null},"Order":1,"ClusterId":"fallbackCluster","AuthorizationPolicy":null,"RateLimiterPolicy":null,"TimeoutPolicy":null,"Timeout":null,"CorsPolicy":null,"MaxRequestBodySize":null,"Metadata":null,"Transforms":null}
info: Web.ForwarderTelemetry[0]
      Forwarder Telemetry [17:02:11.151] => OnForwarderInvoke:: Cluster id: fallbackCluster, Route Id: fallbackRoute, Destination: fallbackApp
info: Web.ForwarderTelemetry[0]
      Destinations: https://localhost/
info: Web.ForwarderTelemetry[0]
      Forwarder Telemetry [17:02:11.325] => OnForwarderStart :: Destination prefix: https://localhost/
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
      Proxying to https://localhost/img/logo.png HTTP/2 RequestVersionOrLower
info: Web.ForwarderTelemetry[0]
      Forwarder Telemetry [17:02:11.478] => OnForwarderStage :: Stage: SendAsyncStart
info: Web.ForwarderTelemetry[0]
      Forwarder Telemetry [17:02:11.557] => OnForwarderStage :: Stage: SendAsyncStop
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[56]
      Received HTTP/2.0 response 404.
info: Web.ForwarderTelemetry[0]
      Forwarder Telemetry [17:02:11.703] => OnForwarderStage :: Stage: ResponseContentTransferStart
info: Web.ForwarderTelemetry[0]
      Forwarder Telemetry [17:02:11.775] => OnContentTransferred :: Is request: False, Content length: 4868, IOps: 2, Read time: 0.000, Write time: 0.001
info: Web.ForwarderTelemetry[0]
      Forwarder Telemetry [17:02:11.841] => OnForwarderStop :: Status: 404

These log entries mean that the request to "https://192.168.102.9:7159/cache/211108_0636182/img/logo.png" was successfully rewritten to "https://192.168.102.9:7159/img/logo.png" but Yarp proxy still tries to forward it via a used fallback rule.

Disabling Yarp Proxy fixes url rewriting rule processing.

I also noticed that StaticFilesMiddleware ignores to process rewritten "logo.png" request in a case if Yarp proxy is initialized.

var rewriteOptions = new RewriteOptions()
    // static cache folder rewrite rule
    .Add(UrlRewritingUtility.RewriteCachedStaticFilePathRequests);
app.UseRewriter(rewriteOptions);

StaticFileOptions staticFileOptions = new StaticFileOptions
{
    // this event is not called for rewritten "logo.png" request in a case if Yarp Proxy is declared
    OnPrepareResponse = ctx => {
        var fi = ctx.File;

    },
};
app.UseStaticFiles(staticFileOptions);

If Yarp Proxy is enabled and I request this url - "https://192.168.102.9:7159/img/logo.png" - the response is well - i.e. image is transfered to my browser as needed.

I can only say that Yarp Proxy Forwarder prevents rewritten requests to be handled by the proxy host app.

[MihaZupan]

The more interesting logs here would be from Microsoft.AspNetCore.Routing. I'd recommend turning on all logs to debug this instead of filtering by package.

E.g. if I try something similar to your setup:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 GET https://localhost:7071/cache/211108_0636182/bootstrap/bootstrap.min.css - - -
dbug: Microsoft.AspNetCore.Rewrite.RewriteMiddleware[3]
      Request is done applying rules. Url was rewritten to https://localhost:7071/bootstrap/bootstrap.min.css
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
      4 candidate(s) found for the request path '/bootstrap/bootstrap.min.css'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
      Endpoint '(null)' with route pattern 'bootstrap/bootstrap.min.css' is valid for the request path '/bootstrap/bootstrap.min.css'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
      Endpoint '(null)' with route pattern 'bootstrap/bootstrap.min.css' is valid for the request path '/bootstrap/bootstrap.min.css'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
      Endpoint 'fallbackRoute' with route pattern '/{**catch-all}' is valid for the request path '/bootstrap/bootstrap.min.css'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
      Endpoint 'Fallback {**path:file}' with route pattern '{**path:file}' is valid for the request path '/bootstrap/bootstrap.min.css'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
      Request matched endpoint '(null)'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[3]
      Endpoint '(null)' already set, skipping route matching.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'Microsoft.AspNetCore.Routing.RouteEndpoint'
info: Microsoft.AspNetCore.StaticAssets.StaticAssetsInvoker[2]
      Sending file. Request path: 'bootstrap/bootstrap.min.css'. Physical path: 'somePath'
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'Microsoft.AspNetCore.Routing.RouteEndpoint'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 GET https://localhost:7071/bootstrap/bootstrap.min.css - 200 162773 text/css 56.6798ms

It would be really useful if you could narrow down your problem to a runnable example we can test.

Re: StaticFilesMiddleware, see #2393 (comment)

[sgarnovsky]

I tried this one configuration.
Sorry, but initially, I missed to update development app settings which overridden Microsoft logs to Warning level.

"LogLevel": {
"Default": "Debug",
"Microsoft": "Debug",
"Microsoft.AspNetCore": "Debug",
"Microsoft.Hosting.Lifetime": "Debug",
"Yarp": "Debug",
"Microsoft.AspNetCore.Rewrite": "Debug",
"Microsoft.AspNetCore.Routing": "Debug",
"Microsoft.AspNetCore.StaticFiles": "Debug"
}

The log entries below means that something wrong with the Routing after as URL was overridden.
I also tried to change an Order value in the forwarder settings. But it didn't change anything.
"Routes": {
"fallbackRoute": {
"ClusterId": "fallbackCluster",
"Order": "1000",
"Match": {
"Path": "/{**catch-all}"
//"Path": "/MsCms4/{**catch-all}" // virtual dir is here as w/o it UseRewriter doesn't work as expected.
}
}
},

https://192.168.102.9:7159/img/logo.png

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/2 GET https://192.168.102.9:7159/img/logo.png - - -
dbug: Microsoft.AspNetCore.Rewrite.RewriteMiddleware[1]
Request is continuing in applying rules. Current url is https://192.168.102.9:7159/img/logo.png
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
Sending file. Request path: '/img/logo.png'. Physical path: 'D:\Sources\App\Web\Core\wwwroot\img\logo.png'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/2 GET https://192.168.102.9:7159/img/logo.png - 200 3374 image/png 14.2128ms

https://192.168.102.9:7159/cache/211108_0636182/img/logo.png

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/2 GET https://192.168.102.9:7159/cache/211108_0636182/img/logo.png - - -
dbug: Microsoft.AspNetCore.Rewrite.RewriteMiddleware[3]
Request is done applying rules. Url was rewritten to https://192.168.102.9:7159/img/logo.png
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
1 candidate(s) found for the request path '/img/logo.png'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
Endpoint 'fallbackRoute' with route pattern '/{**catch-all}' is valid for the request path '/img/logo.png'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
Request matched endpoint 'fallbackRoute'
dbug: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[15]
Static files was skipped as the request already matched an endpoint.
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[3]
Endpoint 'fallbackRoute' already set, skipping route matching.
dbug: Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler[8]
AuthenticationScheme: Auth.Cookies was successfully authenticated.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'fallbackRoute'
<..snip..>
Forwarder Telemetry [17:32:50.551] => OnForwarderStop :: Status: 404
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'fallbackRoute'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/2 GET https://192.168.102.9:7159/img/logo.png - 404 4868 text/html;+charset=utf-8 676.9152ms

It would be really useful if you could narrow down your problem to a runnable example we can test.

Yes, sure will try to create a sample soon.

[MihaZupan]

Static files was skipped as the request already matched an endpoint.

That sounds like #1712 (comment)

[sgarnovsky]

Here is services initialization on my side:

services.AddSystemWebAdapters()
...

services.AddReverseProxy()
  .LoadFromConfig(Configuration.GetSection("Migration:ReverseProxy"));
...

services.AddRazorComponents()
    .AddInteractiveServerComponents();

Here is middlewares initialization:

app.UseHttpsRedirection();

var rewriteOptions = new RewriteOptions()
    // static cache folder rewrite rule
    .Add(UrlRewritingUtility.RewriteCachedStaticFilePathRequests);
app.UseRewriter(rewriteOptions);

StaticFileOptions staticFileOptions = new StaticFileOptions
{
    OnPrepareResponse = ctx => {
        var fi = ctx.File;
    },
};
app.UseStaticFiles(staticFileOptions);

// https://github.com/microsoft/reverse-proxy/issues/2393
// if a routing is not set up then the StaticFiles mapping doesn't work together with the used Forwarder
app.UseRouting();

app.UseSystemWebAdapters();

app.UseAuthentication();
app.UseAuthorization();

// About a position to register this middleware https://github.com/dotnet/aspnetcore/issues/50612
app.UseAntiforgery();

InitializeEnsureStorages(app);

app.UseAjaxResponseMiddleware();

// register this middleware after EnsureStorageMiddleware<IMessagesAccessor<Messages>>
app.UseMessagesCleanerMiddleware();

app.UseMiddleware<AppApplyFormattingMiddleware>();

app.MapRazorComponents<App>()
  //.AddInteractiveServerRenderMode()
  .RequireSystemWebAdapterSession();

app.MapWhen(
    context => !context.Request.Path.ToString().Contains("/App/", StringComparison.OrdinalIgnoreCase)
             && context.Request.Path.ToString().EndsWith("/ckeditorhandlers/GetCKEditorData.ashx", StringComparison.OrdinalIgnoreCase),
appBranch => {
    appBranch.UseCKEditorDataHandler();
});

// SG: but really it doesn't look as used
app.MapWhen(
    context => !context.Request.Path.ToString().Contains("/App/", StringComparison.OrdinalIgnoreCase)
             && context.Request.Path.ToString().EndsWith("/ckeditorhandlers/GetCKEditorStyles.ashx", StringComparison.OrdinalIgnoreCase),
    appBranch => {
        appBranch.UseCKEditorStylesHandler();
    });

app.MapWhen(
    context => !context.Request.Path.ToString().Contains("/App/", StringComparison.OrdinalIgnoreCase)
             && context.Request.Path.ToString().EndsWith("/ckeditorhandlers/CKEditorUploadImage.ashx", StringComparison.OrdinalIgnoreCase),
    appBranch => {
        appBranch.UseCKEditorUploadImageHandler();
    });

app.MapWhen(
    context => !context.Request.Path.ToString().Contains("/App/", StringComparison.OrdinalIgnoreCase)
             && context.Request.Path.ToString().EndsWith("/component.aspx", StringComparison.OrdinalIgnoreCase),
    appBranch => {
        appBranch.UseComponentHandler();
    });

// https://microsoft.github.io/reverse-proxy/articles/diagnosing-yarp-issues.html
app.MapReverseProxy(proxyPipeline =>
{
    // Use a custom proxy middleware, defined below
    proxyPipeline.Use(MyCustomProxyStep);
    //// Don't forget to include these two middleware when you make a custom proxy pipeline (if you need them).
    //proxyPipeline.UseSessionAffinity();
    //proxyPipeline.UseLoadBalancing();
});

Actually, w/o registering "app.UseRouting();" after static middleware, requested static resource are not handled at all.
But as you can see above, requests w/o a virtual cache folder are handled well in my case.

I see one important difference between your sample log entries and I posted above:

dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
      **4 candidate(s)** found for the request path '/bootstrap/bootstrap.min.css'

dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
     **1 candidate(s)** found for the request path '/img/logo.png'

It means, that my Blazor static SSR app provides just one routing candidate to match instead of 4 in your sample.

MihaZupan, please, can you provide any information on what I can further check?

[sgarnovsky]

Implemented a sample to reproduce the describes issue.

Please check it here: BlazorAppUrlRewritingTest

Blazor app has 2 image html elements defined on a home page.

<img src="/logo.png" style="width: 150px;" border="1" />
<br />
<img src="/cache/211108_0636182/logo.png" style="width: 150px;" border="1" />

The first is loaded well, but the second is forwarded to the legacy app url base (https://localhost/)

The console log is the same I posted above:

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 GET https://localhost:7162/cache/211108_0636182/logo.png - - -
dbug: Microsoft.AspNetCore.Rewrite.RewriteMiddleware[3]
      Request is done applying rules. Url was rewritten to https://localhost:7162/logo.png
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1001]
      1 candidate(s) found for the request path '/logo.png'
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1005]
      Endpoint 'fallbackRoute' with route pattern '/{**catch-all}' is valid for the request path '/logo.png'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[1]
      Request matched endpoint 'fallbackRoute'
dbug: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[15]
      Static files was skipped as the request already matched an endpoint.
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[3]
      Endpoint 'fallbackRoute' already set, skipping route matching.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'fallbackRoute'
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[9]
      Proxying to https://localhost/logo.png HTTP/2 RequestVersionOrLower
info: Yarp.ReverseProxy.Forwarder.HttpForwarder[56]
      Received HTTP/2.0 response 404.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'fallbackRoute'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 GET https://localhost:7162/logo.png - 404 4838 text/html;+charset=utf-8 46.4899ms

In case if a forward configuration is changed to filter a virtual directory, the overridden request is processed well:

"Match": {
    //"Path": "/{**catch-all}"
    "Path": "/AppVDir/{**catch-all}" // virtual dir is here as w/o it UseRewriter doesn't work as expected.
}

info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 GET https://localhost:7162/cache/211108_0636182/logo.png - - -
dbug: Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware[0]
      Wildcard detected, all requests with hosts will be allowed.
dbug: Microsoft.AspNetCore.Rewrite.RewriteMiddleware[3]
      Request is done applying rules. Url was rewritten to https://localhost:7162/logo.png
dbug: Microsoft.AspNetCore.Routing.Matching.DfaMatcher[1000]
      No candidates found for the request path '/logo.png'
dbug: Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware[2]
      Request did not match any endpoints
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/logo.png'. Physical path: 'D:\Sources\Symmetric\Misc\AspNetCoreIssues\BlazorAppUrlRewritingTest\wwwroot\logo.png'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 GET https://localhost:7162/logo.png - 200 79421 image/png 40.3930ms

[benjaminpetit]

Sorry for the delay.

I don't think this is a YARP specific issue:

The rewrite middleware seems to force a call to the routing middleware immediately if the path changed. So in your case:

• call to /logo.png

1. Rewrite middleware -> no rewrite
2. static file middleware -> match

• call to /cache/211108_0636182/logo.png

1. Rewrite middleware -> rewrite
2. (injected by the rewrite middleware) routing middleware -> match (yarp in this case)
3. static file middleware -> skipped since an endpoint was already found.

I think we should forward this issue to aspnetcore repo?

[karelz]

Triage: Let's move it to ASP.NET Core repo.