Optimize file delivery for web apps

[javiercn]

History

Serving files in ASP.NET Core following production best practices requires a significant amount of work and technical expertise. This usually means that users end up delivering files to the browser in a suboptimal way in their app, even though, for the majority of cases, files are known during the build/publish process and don't change dynamically.This includes optimizations like compression, caching, fingerprinting, and more.

As a result of this, the browser is forced to do additional requests on every page load, more bytes get transferred through the network, and in some cases, even stale versions get served to customers.

Goals

• Serving a given asset once until it changes or the browser clears its cache.
• Preclude the browser from using old or stale assets after an app has updated.
• Preclude the browser from using old asset versions during development.
• Speed up page load times by delivering assets faster using modern browser features.
• Minimize the size of assets served to the browser.

Non goals

• JS minification, CSS minification, and other similar transformations.

Details

We will serve compressed versions for all the assets in your app that benefit from compression, and not just Blazor files. This includes JS, CSS, etc. and any file except for those that are already compressed, like png, jpg, woff, etc.

We will fingerprint all the assets at build time with a hash of the content. This will make the file name unique based on its contents, which will prevent old versions from being reused in lieu of newer versions even if the app has cached the old version.

We will set up caching headers for all the assets that are known at build time. This will make the browser cache the assets for a long time, and only request them again if they change or the browser clears its cache. Fingerprinted assets will be cached using the immutable directive, which will make the browser never request them again until they change, and we will also include a max-age directive to the same effect for browsers that don't support immutable.

We will set up the content-hash as the ETag for all the assets. This will make sure that even if we don't fingerprint a given asset, the browser will be forced to check and download a newer version.

We will provide a map between the assets that aren't fingerprinted and the assets that are that can be used at runtime to find the fingerprinted version of a given asset. This will allow the app to use the fingerprinted version of an asset for files that are automatically generated, like our CSS, and the non-fingerprinted version in the code, like in Razor pages and similar.

This can be use to produce an import map for the app, which will allow the imports in JS files to use the non fingerprinted names, and the browser will automatically use the fingerprinted version. It can also be used to for example, generate link tags in the head of the page to preload the assets without waiting for the browser to discover them.

• [Static web assets] Define static web asset endpoints in the SDK and generate an endpoints manifest #54873
• [Static Web Assets] Define mechanism to fingerprint an asset relative path and generate new endpoints for it. #54874
• [Static Web Assets] Runtime APIs to consume the generated endpoint manifest from the SDK #54875

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 9 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s).
If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues.
To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[javiercn]

#31922

[javiercn]

#38552

[tbolon]

Could the optimization also targets embedded resources? Because if static files are fingerprinted and are not supposed to change after compilation, why not simply allow files stored as embedded resources to behave the same?

This would allow single-file publishing where all static resources are also integrated in the published app (think single file web apps) and benefit from all these optimizations.

[pavelsavara]

This would allow single-file publishing where all static resources are also integrated in the published app (think single file web apps) and benefit from all these optimizations.

I think it's non-goal here. We have another issue for that dotnet/runtime#86162

[tbolon]

I don't think it's the same issue: the goal of #86162 seems to give the ability to bundle a single file for all javascript files, kind of bundler similar to what webpack allows.

My suggestion here is to support embedded resources as static assets (images, css, files, js). Go has a similar feature: when you publish a trimmed/single-file exe, you have an easy way to serve embedded static files as resources in your web app (think of UseStaticFiles() with a namespace for embedded resources). It allows real 1 file deployment where even the assets are included in the binary.

Even if this implementation does not exists yet, during the development of this content delivery optimization you could allow interacting during the build process to customize "all the assets that are known at build time" and not simply rely on a hardcoded mechanism which scan the wwwroot directory (for example).

[danroth27]

@tbolon Thanks for the suggestion! Since your scenario is somewhat different (single file deployment with static web assets), I think we should track it as a separate issue so we can evaluate it independently of this one. Would you be willing to open a separate issue with your feature suggestion and scenarios?

[danroth27]

We will fingerprint all the assets at build time with a hash of the content. This will make the file name unique based on its contents, which will prevent old versions from being reused in lieu of newer versions even if the app has cached the old version.

I think there are scenarios where the specific file name is important, like creating a custom 404 page using a 404.html file in GitHub Pages. So presumably we'll need some way of excluding certain files from name modification.

[Bartmax]

I want to add that using static files from a class library have a very different workflow on debug and release. When on debug, there's a File Provider that is used to load files from the class library but on production those files are copied to the build output.

Why this matters? Having to reference class library disk files (for example, for computing a crc, check existance, etc), I currently have to know/hardcode where that library is on disk but on production since files are copied to output folder, the location will be not only discoverable/predictable but different.

One usecase I happen to have that show this as problematic is a component for import maps.
on production, I have manually included files and every file using the js/controllers/*_controllers.js pattern while on debug I include files manually (because, as I said, I have no way to know where to look them up on disk).

<ScriptImportMap Folder="js" SearchPattern="controllers/*_controller.js">

    <ImportMap Key="@("@hotwired/stimulus")" Value="/libs/hotwired/stimulus/dist/stimulus.js" />
    <ImportMap Key="@("@hotwired/stimulus-loading")" Value="/js/hotwired/stimulus-loading.js" />
    <ImportMap Key="@("@hotwired/stimulus-autoloader")" Value="/js/hotwired/stimulus-autoloader.js" />
    @{
    #if DEBUG
/* on debug I add all files needed manually */
    }
    <ImportMap Key="controllers/overlay_controller" Value="/js/controllers/overlay_controller.js" />
    <ImportMap Key="controllers/toggle_controller" Value="/js/controllers/toggle_controller.js" />
    <ImportMap Key="controllers/sticky_top_controller" Value="/js/controllers/sticky-top_controller.js" />
    <ImportMap Key="controllers/carousel_controller" Value="/js/controllers/carousel_controller.js" />
    <ImportMap Key="controllers/swipe_controller" Value="/js/controllers/swipe_controller.js" />
    <ImportMap Key="controllers/trailer_controller" Value="/js/controllers/trailer_controller.js" />
    <ImportMap Key="controllers/modal_controller" Value="/js/controllers/modal_controller.js" />
    @{

    #endif
    }
</ScriptImportMap>

The location of the files are important because I do process those files, for example var mapValue = "/"+fileVersionProvider.AddFileVersionToPath($"/", relativeFilePath.ToString()); to get cache busting version.

the output of that component in production is (note scripts are auto included):

<script type="importmap">{
  "imports": {
    "@hotwired/stimulus": "/libs/hotwired/stimulus/dist/stimulus.js?v=2LM-bmB2aiUduPOGHZ43kcJ3ylIhtpKvRGAbs_v84WI",
    "@hotwired/stimulus-loading": "/js/hotwired/stimulus-loading.js?v=U9WK7Q1hR4lAr88chcdVl9QXYSHr9MmQ5OBmnomIab0",
    "@hotwired/stimulus-autoloader": "/js/hotwired/stimulus-autoloader.js",
    "controllers/carousel_controller": "/js/controllers/carousel_controller.js?v=FwZSAdobk653OBFclVNGAEW2wk7i9KwKa4p_PWRSoG8",
    "controllers/debug_controller": "/js/controllers/debug_controller.js?v=bvteyQb8E0RKl82k59trwhYTjafCWZnpI3oyaLAIHsY",
    "controllers/modal_controller": "/js/controllers/modal_controller.js?v=kP3F-mFiI1KarsLfBUZOdHNnkhN-59W74wfB_RhviEg",
    "controllers/offcanvas_controller": "/js/controllers/offcanvas_controller.js?v=QCdpzD_SyvlTg9fKszYFrg9CVaZ3TJaZPMz8x68fpeM",
    "controllers/overlay_controller": "/js/controllers/overlay_controller.js?v=iPM10adCa4pbxTWEKnOmaim2s4dq8LXKI2Y3zLGlSmo",
    "controllers/sticky_top_controller": "/js/controllers/sticky-top_controller.js?v=_CWLFZ1TzCqacT4NEzOjfswU4-w4liYkHc86AbabsJA",
    "controllers/swipe_controller": "/js/controllers/swipe_controller.js?v=F8WduDA7Q-pLXGGiD3pGXgajApcj0rFcOnyjExtoWFM",
    "controllers/toggle_controller": "/js/controllers/toggle_controller.js?v=Ln64fgsgeCk_2SDEmw8StX5Htup9hywxI0cBt40erRg",
    "controllers/trailer_controller": "/js/controllers/trailer_controller.js?v=npqPBvEFKdPziZdRjXio6PIqUpmHQerw4EXPNUupTuk"
  }
}</script>

it would be very important for me to support this scenario without all this hackery.
Thanks.

[javiercn]

I want to add that using static files from a class library have a very different workflow on debug and release. When on debug, there's a File Provider that is used to load files from the class library but on production those files are copied to the build output.

Why this matters? Having to reference class library disk files (for example, for computing a crc, check existance, etc), I currently have to know/hardcode where that library is on disk but on production since files are copied to output folder, the location will be not only discoverable/predictable but different.

Not sure what the first part is about. Static web assets perform the mapping between the physical path and the "virtual path" so that it's the same both during build and publish. If you use the webroot file provider from IWebHostEnvironment you'll get to the physical file in both cases.

If you have a build step where you fingerprint all the files with a version for cache-busting, then my recommendation is that you generate a file with that mapping at build time as JSON and read it at runtime to use it within your .NET code (this is our plan).

[javiercn]

Optimized asset delivery

We want to ensure that all static content from the app can be served as fast and as efficiently as possible. During the build process we compute a lot of metadata about the assets which gets thrown away after the build. Ideally, we can use this metadata to define how to server the assets at runtime in the most efficient way.

In particular we want to:

• Enable serving pre-compressed assets for all files that support it, without the user having to add additional code to their app.
• Ensure we can properly cache assets on the client side for a long time, using fingerprints where possible and relying on ETags when not possible.

One of the limitations we have is the lack of infrastructure to understand and process the dependencies in all the different languages (JS, CSS, etc) that are involved in the build process. For that reason, our approach needs to be backwards compatible and "opt-in", without us requiring to rewrite all the files to use fingerprinted assets (which we will do where possible).

Since we don't have control over this, we never change the name of the files to include the fingerprint value by default. We only do this when the user explicitly configures it and in that case all references must be updated to use the new name. We provide utilities to achieve this from C# (via a class that reads the manifest and provides the mapping between the asset name and its fingerprinted name) and from JS (via importmaps, which we can generate based on the available manifest).

High level design

• As part of the build process we will generate a manifest with a list of endpoints and their corresponding assets. This manifest will be used to serve the assets at runtime, and will include:
• Route: The route for the asset.
• File: The path from the wwwroot folder to the asset.
• ResponseHeaders: The headers that should be added to the response when serving the asset.
• Selectors: The conditions that indicate when this asset should be served. For example, the asset should be served only when the request accepts gzip encoding.

ASP.NET Core runtime design

The runtime design is "straightforward". A new method "MapStaticResources" will map all the endpoints that we generated during the build/publish process. The method will read the manifest we generated and dumped into the bin folder during the build/publish process (<<AppName>>.staticresourcesendpoints.json) and will use that information to map the files as endpoints.

A matcher policy in routing will handle content negotiation based on the Accept-Encoding to serve the correct asset when multiple encoded versions are available.

app.MapStaticResources();

Usage from Blazor

[CascadingValue] public ResourceMap Resources { get; set; }

<script src="@Resources["_framework/blazor.web.js"]">
</script>

The ResourceMap cascading value will be provided by the runtime and will choose the best mapping to use for referencing an asset (which is the fingerprinted url).

The importMap component will generate a JS import map for all the JS assets that are defined in the manifest. This way, ES6 modules don't have to be updated to include the fingerprinted paths on the import statements.

Usage from ASP.NET Core

<script src="~/my.asset.js">
</script>

In this case the URL tag helper is enhanced to use the manifest to map the asset to the correct fingerprinted path.

<script type="importmap">
</script>

We'll have an importmap tag helper that will generate the same import map that the component one generates.

public class StaticResourceResolver : IEnumerable<StaticResource>
{
    public StaticResourceResolver(string manifestPath);

    public StaticResource[] GetResources(string asset);
}

public class StaticResource
{
    public string Route { get; }  
    public IReadOnlyCollection<string, string> ResponseHeaders { get; }
    public IReadOnlyCollection<string, string> Selectors { get; }
}

SDK design

The static web assets needs to be extended to support the new features.
StaticWebAsset will get two new properties:

• Fingerprint: A fingerprint of the asset that is a Base36 encoded string of the SHA256 hash of the file. This is the same format we use for the dotnet.* files in wasm and is suitable to appear in URLs, etc.
• Integrity: A base64 encoded string of the SHA256 hash of the file. This is used by Blazor WebAssembly to load the dlls and ensure they match the contents defined in the blazor.boot.json manifest. We want to have it here so that we avoid having to compute it in different locations during the build (other elements of the build also compute a hash and it ends us being computed multiple times).

There is a new StaticWebAssetEndpoint item that defines one or more endpoints for a given file. An endpoint defines:

• Route: The route the endpoint needs to be mapped to.
• AssetFile: The identity of the associated static web asset.
• Selectors: A list of selectors that determine when this resource must match. Each selector contains a Name, Value, and Quality.
  • The Name indicates what criteria to use for matching, for example, "Content-Encoding" indicates that it needs to match based on the Accept-Encoding header. The mapping between selector names and runtime matching criteria is defined in the runtime. (For example, the runtime is the one that understands the Content-Encoding selector and configures the metadata necessary on the endpoint for the matcher policy to work).
  • The value indicates the value that the request must accept for the given selector. For example, "gzip" or "br" for the Content-Encoding selector. The valid values are completely dependent on the selector.
  • The quality indicates the tie-breaker when multiple endpoints are "tied" for a given request. The quality is a value between 0 and 1, and is only used after the request preferences have been applied.
• ResponseHeaders: The list of headers to add to the response. Each header has a Name and a Value that were computed during the build/publish steps and are added to the response verbatim.

The selectors and response headers are encoded as JSON when represented inside MSBuild metadata items.

Pipeline changes

The pipeline needs to be extended at 4 places:

• Build -> StaticWebAssetsPrepareForRun.
• Assets from class libraries -> GetCurrentProject(Build|Publish)StaticWebAssetItems.
• Pack -> GenerateStaticWebAssetsPackManifest.
• Publish -> Create a list of publish assets.

In each of these cases, we will have a list of defined endpoints. These endpoints will be filtered based on the static web assets that we need to use for the given operation, that way we don't have to add additional logic to filter and process endpoints in different scenarios.

For example, when we are packing, we need to include only the assets and endpoints that are relevant when the project is consumed as a reference. Static Web Assets already contains that logic, so we just compute the set of assets for that scenario (as we do today) and then we filter the endpoints to those that have an AssetFile that matches the assets we are packing.

Same applies to Build, Publish, etc.

Static web assets must exist at the time they are defined

The static web assets must exist at the time they are defined because otherwise we can't compute the fingerprint and integrity of the asset. This means that we need to have a strict build order defined for different steps of the build, like compression or service worker generation, as opposed to the partial order that exists today.

Assets that are generated based on existing assets need to use endpoints instead

All assets that are generated based on existing assets (like the styles bundle in scoped CSS) must switch to use the set of endpoints as input instead of using the static web assets directly. These tasks need to potentially generate two sets of assets. One with the "standalone" endpoints and another with the "hosted" endpoints. The "hosted" endpoints are the ones that are used when the app is hosted in a server that supports the new features, and the "standalone" endpoints are the ones that are used when the app is hosted in a server that doesn't support the new features like Github Pages.

Standalone assets are those whose route matches the relative path of its associated asset, because they can be served directly by the server without additional configuration. Hosted assets require in turn, additional configuration for the app to work properly.

At publish time, we'll decide which set of assets to use. In most scenarios we'll use the "hosted" assets since we will be hosted in an ASP.NET Core server in all cases except for webassembly standalone.

• The default model for web is to use the "hosted" assets.
• The default model for webassembly is to use the "standalone" assets, but can be overriden to use the "hosted" assets (if you are for example hosting on IIS and generate a webconfig that matches the manifest semantics)

This includes:

• Scoped CSS
• JS initializers
• Service worker manifest

[Fydar]

I love this design so far! ❤️

I'm very excited about the potential for this to allow for integration with third-party CDNs like Azure Cloud Delivery Network or Amazon CloudFront.

An application could be configured to use a CDN with a simple:

builder.Services.DistrubuteResourcesViaCDN();

[Bartmax]

If you have a build step where you fingerprint all the files with a version for cache-busting, then my recommendation is that you generate a file with that mapping at build time as JSON and read it at runtime to use it within your .NET code (this is our plan).

how do I create a JSON map of files (auto-discover) of a razor class library ? I don't know how to get the actual file path.

Even with this approach, referencing those files on DEBUG and RELEASE is different.
RCL uses a file provider to "find" those files (so the physical path on DEBUG will be {some_path}/RCL/wwwroot/... while on RELEASE it would be /wwwroot.

I'm probabily missing something and/or I don't know how to use the physical file provider propery, I cannot find a way to read the file from code (not middleware) using a single approach.
Any insight will be really appreciated.

StaticFiles and similar middleware does work perfectly, but as I understand, you pass the path/filename and it tries every file provider but I want to "read all files from the RCL file provider" and I found no way to do that. Does it make sense?

Thanks

[javiercn]

#56076

[javiercn]

Completed as part of #56045