Web Vitals are captured incorrectly

[lforst]

Description

It seems like the SDK is reporting completely wrong webvitals: Trace

• TTFB is completely off
• FCP after LCP

Looking through some traces in Sentry this did not seem to be an issue in v7.

[Lms24]

Hmm I'm not sure if both are related. Some observations so far:

• TTFB:
  • We changed ttfb reporting in #fix(metrics): use web-vitals ttfb calculation #11185 and fix(tracing): [v7] use web-vitals ttfb calculation #11231. Maybe this change is related
  • In

    sentry-javascript/packages/browser-utils/src/metrics/browserMetrics.ts

    Lines 344 to 361 in 5a709df

    	['fcp', 'fp', 'lcp'].forEach(name => {
    	const measurement = _measurements[name];
    	if (!measurement || !transactionStartTime || timeOrigin >= transactionStartTime) {
    	return;
    	}
    	// The web vitals, fcp, fp, lcp, and ttfb, all measure relative to timeOrigin.
    	// Unfortunately, timeOrigin is not captured within the span span data, so these web vitals will need
    	// to be adjusted to be relative to span.startTimestamp.
    	const oldValue = measurement.value;
    	const measurementTimestamp = timeOrigin + msToSec(oldValue);
    	// normalizedValue should be in milliseconds
    	const normalizedValue = Math.abs((measurementTimestamp - transactionStartTime) * 1000);
    	const delta = normalizedValue - oldValue;
    	DEBUG_BUILD && logger.log(`[Measurements] Normalized ${name} from ${oldValue} to ${normalizedValue} (${delta})`);
    	measurement.value = normalizedValue;
    	});

    , we do not iterate over ttfb although the comment mentions it explciitly. Not sure yet if oversight or on purpose. I asked @AbhiPrasad if he knows more about this. Could just be that the comment is outdated and this is a false flag 😅
• LCP:
  • In the given trace the values are fairly close to each other. Maybe this is one of these cases where we'd report an intermediate/initial LCP value which would change later after the transaction ended. I'm really not sure though. Do we have examples where there is a larger gap between LCP and FCP and LCP < FCP?

[md384]

I have also found that web-vitals data for my site are seemingly incorrect. We have long recorded web vitals data using the web-vitals package, and we have noticed that the values from that do not line up with ones reported by Sentry (with v7 too) - we're currently on 8.22.0.

Below shows the difference of LCP, TTFB and FCP between sentry reported values (left) and ones from the web-vitals (right) package (webvitalscomp_XXX metric).

p75

p50

LCP seems to be the most off - significantly different in many cases.

Individual examples (self-hosted so can't link examples)

On individual traces, here is an example of the sentry values being different to our custom metric and also not lining up with the spans in the trace

Here can see the difference of start time of the request and the start time of the first paint span is 1723035794.5266 - 1723035793.3029 = 1.22s which is still lower than our custom metric but closer than the sentry reported one.

This also shows LCP > FCP which incorrect as mentioned in the original post.

We have many examples of individual traces like this.

[Lms24]

Hey @md384 thanks for providing this data!

I can see a couple of things that might explain the discrepancies:

• we collect all web vitals at the time when we end our pageload span. This might be too early, resulting in only capturing an intermediate value for LCP. This could be the reason why our reported LCP values tend to be shorter than your recorded LCP values.
  • it doesn't really explain why we get FCP > LCP values though. I've seen it before, leading me to believe that:
• Our start time + vital offset calculation must be off somehow. This also might explain the TTFB issue that was mentioned in this issue.

I'm curious: How do you use the web-vitals library? Specifically, do you set reportAllChanges: true in the onFCP and related callbacks.

[Lms24]

Unrelated to my comment above but adding it here:

A further example of incorrect LCP/FP/FCP values: https://sentry-sdks.sentry.io/performance/trace/10e33722186ea42a792504e313d192e5/?field=title&field=event.type&field=project&field=user.display&field=timestamp&name=All+Events&node=txn-e507fc916a5f4d3f8bba7a1a9fb1032d&node=txn-b7ddd0be9dc04146be202f3865bf3cc6&project=4504796902588416&query=&sort=-timestamp&source=discover&statsPeriod=10m&timestamp=1723115339&yAxis=count%28%29

[md384]

Hey @md384 thanks for providing this data!

I can see a couple of things that might explain the discrepancies:

• we collect all web vitals at the time when we end our pageload span. This might be too early, resulting in only capturing an intermediate value for LCP. This could be the reason why our reported LCP values tend to be shorter than your recorded LCP values.

  • it doesn't really explain why we get FCP > LCP values though. I've seen it before, leading me to believe that:

• Our start time + vital offset calculation must be off somehow. This also might explain the TTFB issue that was mentioned in this issue.

I'm curious: How do you use the web-vitals library? Specifically, do you set reportAllChanges: true in the onFCP and related callbacks.

We do not set reportAllChanges. FWIW, I have seen some instances where web-vitals reported values with FCP > LCP but seems less often than sentry reports.

One other thing, is that the metric counts for my custom metric is consistently lower than the sentry count

We do have a browser guard (typeof window !== 'undefined' && Boolean(window?.navigator?.platform)) to only report the custom metric for browsers, but perhaps there is something there.

[Lms24]

@md384 which version of the web-vitals library are you using?