`sentry_set_context` opens file handles resulting in hitches

[jeremy-wildlight]

Description

System profiling demonstrates that sentry_set_context can easily take 300 to 800 ms in a frame, depending on OS environmental factors because internally, the function appears to create file handles (which also touches USN journal state and so on).

In the context of a single-process crash collection workflow, is this file lock needed? It would be preferable to have a mode of operation where this file was just created once or ideally not at all.

When does the problem happen

• During build
• During run-time
• When capturing a hard crash

Environment

• OS: Windows 10/11 64-bit
• Compiler: Clang 18.1.8

Steps To Reproduce

Run any game and attempt to call sentry_set_context every frame.

[linear]

NATIVE-124

[JoshuaMoelans]

Hi, thanks for writing in. All our sentry_set_context does is store the given key-value pair into scope->contexts using a sentry_value_set_by_key. The only thing we additionally do is create a short-lived lock on the scope, but we do not touch any file handles.

Can you maybe share a snippet of where you're calling sentry_set_context? Maybe the value that is passed in is accidentally using some file handles.

[supervacuus]

Hi @jeremy-wildlight!

System profiling demonstrates that sentry_set_context can easily take 300 to 800 ms in a frame, depending on OS environmental factors because internally, the function appears to create file handles (which also touches USN journal state and so on).

Every scope-mutating function (sentry_set_x() or more generally sentry_[verb]_x()) will need to flush the scope to disk, depending on the combination of crash backend (only the crashpad backend currently does this) and platform requirements (macOS depends on this entirely, Windows only for WER handled exceptions and on Linux it isn't required).

In particular, on Windows, this is only required for fast-fail crashes because those are handled entirely outside the monitored application via WER (and no client-side hook can pull the last state of the global scope to persist into an error).

An alternative for you would be to examine the Breakpad backend (which doesn't flush on any platform). I really cannot say a lot about your use case, but you can have a look at the preliminary discussion here: https://docs.sentry.io/platforms/native/advanced-usage/backend-tradeoffs/

There are multiple options to counter this from our side as well:

• we could introduce an option to disable disk flushing, which means only SEH exceptions will retrieve a proper scope (which could be enough if you don't expect fast-fail crashes)
• we could move the scope updates into the crashpad interface, similar to how we did it with attachments recently, which would mean we'd get rid of scope sync via disk entirely, but there would still be IPC overhead
• we could defer scope persistence, meaning we only enqueue your scope mutation, and a background worker does persistence and disk flushing
• we could optimize the hot-path, but that seems comparatively inconsequential since we still would need to sync to disk at some point

The first three sound sensible as optimizations for that path.

However, it would be interesting to know more about your use case for changing the context on every frame, since the context and every scope item primarily exist to enrich events, and the external flushing is entirely about crash events (since regular events are generated inside the application, they can access the current scope state at any point).

It might be sufficient to disable disk flushing or make it more explicit, since you may not rely on per-frame data for enriching/identifying fast-fail crashes.

Another option would be to use local scopes (example for adding attachments to local scopes) exclusively within your frames, which are applied to specific events and not flushed like global scope mutations:

sentry-native/include/sentry.h

Lines 1642 to 1645 in 075b3bf

	/**
	* Creates a local scope.
	*/
	SENTRY_API sentry_scope_t *sentry_local_scope_new(void);

sentry-native/include/sentry.h

Lines 1656 to 1662 in 075b3bf

	/**
	* Sends a sentry event with a local scope.
	*
	* Takes ownership of `scope`.
	*/
	SENTRY_API sentry_uuid_t sentry_capture_event_with_scope(
	sentry_value_t event, sentry_scope_t *scope);

And configure that local scope with any sentry_scope_* interface.

In the context of a single-process crash collection workflow, is this file lock needed? It would be preferable to have a mode of operation where this file was just created once or ideally not at all.

There is no file lock involved when flushing the scope to disk. The file lock is used solely during initialization when a run directory is chosen for the current process or when old runs are cleaned up.

CC: @jpnurmi, @mujacica

[supervacuus]

Hi @JoshuaMoelans,

The only thing we additionally do is create a short-lived lock on the scope, but we do not touch any file handles.

We do! When closing the mutation lock, we also invoke the backend flusher, which Crashpad currently uses on Windows and macOS.

[jeremy-wildlight]

Thanks @supervacuus for all the helpful context! We had recently upgraded from an earlier version of the library which apparently didn't flush the file so we never noticed. To answer your question, we had previously updated the scope whenever interesting game state had changed (player position, map, and so on) which could really be "any time". The thinking was that updating the context "just in time" (i.e. in a top-level exception handler or before an assert) might be error-prone since the application is in an inconsistent state at the time of a crash/exception/assert.

Some other thoughts:

• On Windows at least, creating the file handle in the first place is often the expensive part as opposed to dirtying a page -- would it be possible to cache the file handle in memory?
• It might be useful to have an option to update the context in a "best effort" manner, without incurring an actual flush.
• IPC into the external handler may be relatively cheap if we write to shared mapped page (we'd still pay for some global synchronization mechanism)

Now that we know this is the current behavior though, we will probably opt to schedule these context updates on a different thread in user-space in cases where the timing is critical. In the meantime, it might be useful for endusers to advertise info about the perf characteristics of the various native API calls to evaluate tradeoffs between scopes, contexts, or other mechanisms.