Chromium NetworkService API Changes

[amaitland]

Chromium is moving from an in-process network stack implementation to a separate NetworkService process.

Due to Chromium changes CEF will be implementing some dramatic API changes. Relevant upstream issue is https://bitbucket.org/chromiumembedded/cef/issues/2622/implement-support-for-networkservice

Chromium is moving to a Network Service, this means that network requests will be handled using a separate process, just like rendering, gpu acceleration, etc.

Known behaviour changes:

• Modifying the newUrl parameter in OnResourceRedirect will no longer result in the method being called an additional time (likely a bug in the old implementation).
• Modifying the request URL in OnResourceResponse would previously cause a redirect. This behaviour is now deprecated because the NetworkService does not support this functionality when using default network loaders. Temporary support has been added in combination with IResourceHandler usage only. - Other changes to the request object in OnResourceResponse will now cause the request to be restarted. This means that OnBeforeResourceLoad, etc, will be called an additional time with the new request information.
• IResponse.MimeType will now be empty for non-200 responses.
• Requests using custom schemes can now be handled via IResourceRequestHandler with the same callback behaviour as builtin schemes.
• Redirects of custom scheme requests will now be followed as expected.
• Default handling of builtin schemes can now be disabled by setting disableDefaultHandling to true in GetResourceRequestHandler.
• Unhandled requests (custom scheme or builtin scheme with default handling disabled) will fail with an Response.Error value of ERR_UNKNOWN_URL_SCHEME.
• The ISchemeHandlerFactory.Create callback will now include cookie headers.
• Unsupported chrome hosts no longer redirect to chrome://version.
• All CachePath values must be relative to a new CefSettings.RootCachePath value.
• ICookieManager callbacks are now executed on the CEF UI thread (previously the CEF IO thread).
• Resource-related callbacks have been moved from IRequestHandler to a new IResourceRequestHandler interface which is returned via the IRequestHandler.GetResourceRequestHandler method. If the IRequestHandler declines to handle a resource it can optionally be handled by the IRequestContextHandler, if any, associated with the loading context.
• The OnProtocolExecution callback has been moved from IRequestHandler to IResourceRequestHandler and will be called if a custom scheme request is unhandled.
• Cookie send/save permission callbacks have been moved from IRequestHandler and IResourceHandler to IResourceRequestHandler.
• New methods added to IResourceHandler that better match NetworkService execution sequence expectations. The old methods are now deprecated.
• New methods added to IRequest and IResponse.
• All pending and in-progress requests will now be aborted when the CEF context
  or associated browser is destroyed. The OnResourceLoadComplete callback will
  now also be called in this case for in-progress requests that have a handler.
• The IResourceHandler.Cancel method will now always be called when resource
  handling is complete, irrespective of whether handling completed successfully.
• Request callbacks that arrive after the OnBeforeClose callback for the
  associated browser (which may happen for in-progress requests that are aborted
  on browser destruction) will now always have a non-nullptr IBrowser
  parameter.
• Allow empty parameters to IRequest and IResponse methods where it makes
  sense (e.g. resetting default response state, or clearing a referrer value).
• Modifying the URL in OnBeforeResourceLoad causes an internal redirect response.
• In cases where the request is cross-origin (containing a non-null "Origin"
  header) the redirect response must include the "Access-Control-Allow-Origin"
  header, otherwise the request will be blocked.
• For 303 redirects all request methods except HEAD are converted to GET as per
  the latest http draft. For historical reasons the draft also allows POST
  requests to be converted to GETs when following 301/302 redirects. Most major
  browsers do this and so shall we. When a request is converted to GET any POST
  data should also be removed.
• Use 307 redirects instead if you want the request to be repeated using the same
  method and POST data.

API changes:

• Add IResourceRequestHandler
• Add ICookieAccessFilter
• IRequestHandler.OnBeforeResourceLoad moved to IResourceRequestHandler.OnBeforeResourceLoad
• IRequestHandler.OnResourceRedirect moved to IResourceRequestHandler.OnResourceRedirect
• IRequestHandler.OnResourceResponse moved to IResourceRequestHandler.OnResourceResponse
• IRequestHandler.GetResourceResponseFilter moved to IResourceRequestHandler.GetResourceResponseFilter
• IRequestHandler.OnResourceLoadComplete moved to IResourceRequestHandler.OnResourceLoadComplete
• IRequestHandler.OnProtocolExecution moved to IResourceRequestHandler.OnProtocolExecution
• IRequestHandler.GetAuthCredentials signature change, IFrame frame replaced with string originUrl
• IDragHandler.OnDraggableRegionsChanged now includes the IFrame param

Cookie Handling

• Cookies are access is now handled using ICookieAccessFilter which is returned using IResourceRequestHandler.GetCookieAccessFilter
• Removed IRequestHandler.CanGetCookie
• Removed IRequestHandler.CanSetCookie
• Removed IResourceHandler.CanGetCookie
• Removed IResourceHandler.CanSetCookie
• Removed ICookieManager.CreateManager
• Removed ICookieManager.GetBlockingManager
• Removed ICookieManager.SetStoragePath
• Removed IRequestContextHandler.GetCookieManager
• IRequestContext.GetDefaultCookieManager renamed to IRequestContext.GetCookieManager
• ICookieManager.SetSupportedSchemes has additional bool includeDefaults param.

Request/Scheme Handling

The IRequestHandler interface has been rewritten as the CEF API changed

• Ability to Seek within a stream has been added
• Instead of implementing IRequestHandler directly it's reccomended for most cases to inherit from RequestHandler and override ProcessRequestAsync. See https://github.com/cefsharp/CefSharp/blob/cefsharp/75/CefSharp.Example/CefSharpSchemeHandler.cs for an example.

ResourceHandler Headers

IRequest now has SetHeaderByName and GetHeaderByName which greatly simplifies header get/set.

CefSharp Specific API Changes

• IResourceHandler.ProcessRequestAsync now returns CefReturnValue instead of bool.
• IResourceHandler.ProcessRequestAsync is now called in IResourceHandler.Open which is a new method, as the structure has changed.
• IResourceHandler.GetResponse has been removed
• 'DefaultRequestHandler' renamed to RequestHandler (DefaultRequestHandler exists as a class for now that's marked as Obsolete)
• RequestHandler now implements IRequestHandler explicitly an exposes protected virtual methods to override, previously they were public.
• ISchemeRegistrar.AddCustomScheme now takes a set of SchemeOptions flags to represent the different options instead of a bool for each param type. Brings the interface in line with the CEF API.
• CefSharp.Legacy.LegacyResourceHandlerFactory has been removed.

[mitchcapper]

From a documentation clarifying point of view even if rootcachepath is set to a real path, CachePath can still be set empty for using in-memory cache.

[amaitland]

From a documentation clarifying point of view even if rootcachepath is set to a real path, CachePath can still be set empty for using in-memory cache.

Thanks 👍 It's probably worth seeing if the CEF documentation can be revised, there are a few inconsistencies/clarifications required.

If non-empty this must be either equal to or a child directory of CefSettings.root_cache_path

Which implies you have to set CefSettings.root_cache_path, which isn't the case, you can leave it blank and it will default to CefSettings.cache_path as per If this value is empty and CefSettings.cache_path is non-empty then this value will default to the CefSettings.cache_path value

I've made some improvements in c0ebbb4 Hopefully it makes sense, feedback welcome.

[amaitland]

Experiencing a strange issue where CefFrame::GetIdentifier() returns a different value in the browser process than it does in the render process. As a result IJavascriptCallback's are broken currently. I've asked https://magpcss.org/ceforum/viewtopic.php?f=6&t=16830 for clarification and that's not the expected behaviour, will test with cefclient to confirm that it's reproducible.

Experiencing a strange issue where CefFrame::GetIdentifier() returns a different value in the browser process than it does in the render process. As a result IJavascriptCallback's are broken currently. I've asked https://magpcss.org/ceforum/viewtopic.php?f=6&t=16830 for clarification and that's not the expected behaviour, will test with cefclient to confirm that it's reproducible.

Not the expected behaviour, opened issue https://bitbucket.org/chromiumembedded/cef/issues/2687/cefframe-getidentifier-differs-between

To allow IJavascriptCallback in the main frame (and unfortunately only the main frame), I've hacked around the problem see 9d47c2e

This should be reverted when the upstream issue is resolved.

• Revert hack
• Confirm issue resolved.

[amaitland]

List of know CEF issues

Network Service
https://bitbucket.org/chromiumembedded/cef/issues/2703/crash-when-closing-if
https://bitbucket.org/chromiumembedded/cef/issues/2715/onprotocolexecution-page-goes-blank-after

https://bitbucket.org/chromiumembedded/cef/issues/2685/networkservice-custom-scheme-unable-to
https://bitbucket.org/chromiumembedded/cef/issues/2668/multi-threaded-message-loop-crashes-again
https://bitbucket.org/chromiumembedded/cef/issues/2316/crash-during-shutdown-due-to
https://bitbucket.org/chromiumembedded/cef/issues/2718/networkservice-proxy-and-request
https://bitbucket.org/chromiumembedded/cef/issues/2720/networkservice-does-not-honor-the
https://bitbucket.org/chromiumembedded/cef/issues/2699/networkservice-issue-with-cross-origin
https://bitbucket.org/chromiumembedded/cef/issues/2708/windows-networkservice-crash-in
https://bitbucket.org/chromiumembedded/cef/issues/2689/networkservice-unpredictable-behavior-with
https://bitbucket.org/chromiumembedded/cef/issues/2709/windows-networkservice-crash-on-shutdown
https://bitbucket.org/chromiumembedded/cef/issues/2727/networkservice-pdfs-not-loading-with-and
https://bitbucket.org/chromiumembedded/cef/issues/2605/cef-crashes-on-cefcontentrendererclient

Unrelated to network service
https://bitbucket.org/chromiumembedded/cef/issues/2678/cefclient-crashes-with-out-of-memory-when
https://bitbucket.org/chromiumembedded/cef/issues/2276/drag-and-drop-is-broken-in-combination
https://bitbucket.org/chromiumembedded/cef/issues/2662/crash-on-mouse-event-from-devtools-remote

[amaitland]

The BrowserSubProcess has been updated as, the new Network Service process needs to handle custom scheme registration.

[amaitland]

• New RequestHandler is available to override only the methods you require
• New ResourceRequestHandler is available to override only the methods you require
• ICookieAccessFilter is used to control cookie access. There is no default implementation as it currently only has two methods and you likely need to implement both of them. Might add a default implementation later if there is enough need for one.

[amaitland]

Changes are complete, any bugs please open a new issue making sure to use the Bug Report Template

[amaitland]

For reference it's still possible to disable the Network Service, see https://github.com/cefsharp/CefSharp/blob/cefsharp/75/CefSharp.Example/CefExample.cs#L108 for an example for both WPF and WinForms (For OffScreen use the WPF example).

This will only work with version 75, this won't do anything for past or future versions.

[amaitland]

The ResponseFilter examples have been updated to reflect the NetworkService changes, see 195569d for the relevant commit.

ResourceRequestHandler maps to a single Resource Request and as such there is no need for a dictionary to track IResponseFilter implementations like in the previous example. IResponseFilter.Dispose is also called before OnResourceLoadComplete now, the example has been adapted to deal with this.

[joaompneves]

For reference it's still possible to disable the Network Service, see https://github.com/cefsharp/CefSharp/blob/cefsharp/75/CefSharp.Example/CefExample.cs#L108 for an example for both WPF and WinForms (For OffScreen use the WPF example).

This will only work with version 75, this won't do anything for past or future versions.

Tried disabling the Network Service, as you mentioned, (due to a severe libcef crash: https://bitbucket.org/chromiumembedded/cef/issues/2795/crash-in-openinputstreamwrapper) but I cannot get custom schemes to work. Navigation aborts right away with an unknown custom schema error code. Probably expected, or not? (Most likely a cef issue, but still asking here)

[joaompneves]

I had to put back the SchemeHandlerFactory when creating the CustomScheme (removed when upgraded to 75) and had to implement my own ResourceHandler because the CefSharp builtin ResourceHandler has deprecated the ReadResponse and ProcessRequest methods, and those seem to be used by the old network layer.