-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug?]: (OpenTelemetry) As of Redwood 6, a single active span appears to be wrapping the entire API server lifetime #9043
Comments
Thanks @pvenable, I'll try to reproduce this and see if I can fix it this week. |
Hey @pvenable, could you help me a little? How are you starting your OpenTelemetry SDK when you use |
@Josh-Walker-GM Here's some info I hope will be helpful. If this doesn't answer your question, please let me know and I'll see what I can do. We have three "touchpoints" with opentelemetry in our Redwood app:
import * as opentelemetry from '@opentelemetry/api';
const getTraceparent = () => {
const output: { traceparent?: string } = {};
opentelemetry.propagation.inject(opentelemetry.context.active(), output);
return output.traceparent;
}; I believe what's happening here is that the I'm not familiar with why this active span was added to the CLI, but if the span itself is indeed necessary, my hunch is that were it not "active" it might not cause this issue. I don't have many other ideas yet other than requesting a way to opt out of this outer span. Thanks, and please let me know if you have any other questions. |
Thanks so much for the detailed response, very helpful! I am still hoping to understand better how/when your SDK was created/invoked. I think this would be related to:
I think on the docs for OTel they tend to recommend something like |
Hey, @Josh-Walker-GM -- thanks for sticking with this.
Alright, I'll share a somewhat redacted version of this. Imports: import { credentials, Metadata } from '@grpc/grpc-js';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import { Resource } from '@opentelemetry/resources';
import {
BatchSpanProcessor,
ConsoleSpanExporter,
NodeTracerProvider,
} from '@opentelemetry/sdk-trace-node'; (In case it's relevant, note that we don't generally use A export const createProvider = () => {
const provider = new NodeTracerProvider({
resource: Resource.default().merge(
new Resource({
// Omitted: `service.name` and `env` properties
})
),
});
const exporter = createExporter();
provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();
return provider;
}; An internal const createExporter = () => {
const metadata = new Metadata();
// Omitted: `metadata.set(...)` a couple values here
return new OTLPTraceExporter({
credentials: credentials.createSsl(),
metadata,
// Omitted: `url: 'grpc://...'`
});
}; This is referenced only from const extraPlugins: Plugin[] = [];
if (/* Omitted: conditions under which we would initialize our plugin */) {
const provider = createProvider();
const plugin = customOpenTelemetryPlugin(provider);
extraPlugins.push(plugin);
} And I can't think of anything else we're doing that would constitute starting or invoking the SDK. We've never passed anything like I hope something I've shared here is useful. Let me know if you think there's any other info that might help. Thanks! |
I had a chance to look into this again this morning. One of the first things I tried was passing I'm not sure whether this issue should be closed, because from my perspective there's still unexpected behavior here (even if I can opt out), but if anyone else runs into this, |
Sorry for being slow with this one! I'd keep it open because this is not the behaviour I want to support. Your debugging seems to have pretty much nailed the cause so I'll try and find some time to sort out a fix. |
I took a look at this tonight and I could kinda reproduce this. I'll need to have a think about the best way to fix it. Thanks for dealing with this in the meantime with the |
What's not working?
We're using a customized opentelemetry plugin in our app that starts an active span for each GraphQL operation. It's similar to the bundled Redwood opentelemetry plugin, but for various reasons we have our own version. (Some relevant history/context here and here.) This has generally worked great for us throughout the 4.x and 5.x release cycles.
We recently deployed Redwood v6 and then noticed that our previously per-operation traces are now all being bundled under a parent trace that appears to span the entire lifetime of our API server -- rather than a distinct trace per operation.
I suspect the source of this new parent trace is
redwood/packages/cli/src/index.js
Lines 120 to 122 in 865c908
If my analysis is correct, can I prevent this behavior somehow? Or is there another change in v6 that's likely to have caused this?
How do we reproduce the bug?
No response
What's your environment? (If it applies)
yarn rw serve api
Are you interested in working on this?
The text was updated successfully, but these errors were encountered: