Error executing async csv_from_savedobject report #62986
Comments
theY4Kman
changed the title
joelgriffith
added
bug
|
@theY4Kman we can definitely do a better job of this, but IIRC your JSON body needs some options (settings) in order to function properly. We'll take a look and see if there's a more intuitive way for declaring these runtime errors. |
|
The same call worked in 7.4.2. The I believe this error is related to the dummy request / export class KibanaRequest<
Params = unknown,
Query = unknown,
Body = unknown,
Method extends RouteMethod = any
> {
// ...
constructor(
request: Request,
// ...
) {
this.url = request.url;
this.headers = deepFreeze({ ...request.headers });
// ...
this.route = deepFreeze(this.getRouteInfo(request)); // <---- from here
this.socket = new KibanaSocket(request.raw.req.socket);
this.events = this.getEvents(request);
}
// ...
private getRouteInfo(request: Request): KibanaRequestRoute<Method> {
const method = request.method as Method;
const { parse, maxBytes, allow, output } = request.route.settings.payload || {}; // <------ source of error here
const options = ({
authRequired: request.route.settings.auth !== false,
tags: request.route.settings.tags || [],
body: ['get', 'options'].includes(method)
? undefined
: {
parse,
maxBytes,
accepts: allow,
output: output as typeof validBodyOutput[number], // We do not support all the HAPI-supported outputs and TS complains
},
} as unknown) as KibanaRequestRouteOptions<Method>; // TS does not understand this is OK so I'm enforced to do this enforced casting
// ...
}
}Plus, the dummy request won't have |
|
Oooo, though, it would be hella sweet to have errors like those saved to the reporting job — right now, the only info is either |
|
I looked at the Fixing that uncovered another issue: the executor was unable to find the index pattern referenced in the saved object. While debugging, I found that when executing the job, the saved objects client was not scoped to the user or their space. I once again looked at the Here are the changes I made: https://github.com/PerchSecurity/kibana/compare/v7.6.2..fix/7.6.2-csv_from_savedobject Judging from the previous return type of the |
Indeed this ability has gone out of maintenance. Originally, we had automated functional tests to validate this feature, but the test repeatedly was hit by flaky test failures, and skipped: #37471. Apologies for this! One question though: why not use the |
|
I'm adding the feedback label to this PR. We plan on totally removing the |
|
Hi @theY4Kman just want to see if you have feedback about why you are not using the Discover POST URL for the CSV exports. I'm planning on removing the "async csv_from_savedobject" code and route in #71031 The only reason I can think of choosing this API over Discover CSV export is this export doesn't take the search results through the field formatters and provides raw data. However, that kind of inconsistency is hard to support, and I believe most users would say that's a bug. If that is important, perhaps that should be an option for Discover CSV export. |
|
My primary motive is to initiate the generation of reports externally. The "async csv_from_savedobject" endpoint was perfect for this: it allowed the reporting job to be initiated without waiting on the response (or failing if a reverse proxy timed out the request); and additionally, it only required the saved object type & ID. If #71031 allows a job to be initiated and return a job ID, regardless of how long it takes to process the job, it'd likely be adequate for our purposes: creating hands-off, periodic reports. As for why I'm not using Discover CSV: it's my understanding that the Discover POST URL includes the raw Elasticsearch query; however, the saved search saved object does not include this raw query — it's some other internal format from Kibana. Things like filters, etc. So, in order to submit Discover CSV reports externally, I'd have to duplicate / reverse engineer Kibana's filter translation, which seems brittle, easily outdated, and hard to debug. Submitting dashboard jobs is hard enough, where we've gotta synthesize a reasonable height to submit with the job. (The print layout is too sparse, and surprising to users who expect what they see when generating reports from within Kibana.) |
|
@theY4Kman your problem statements echo the same concerns as the Reporting Services team which tries to provide an API to export CSV the same as it would show in a browser. To export this data, we need the same query that the browser uses to search. Historically in Kibana code, the source of generating that query has been in browser-side Angular code that can't be shared with the server. The solution was to pass the entire query in the API to generate a request. Obviously that is not ideal for all the reasons you've pointed out. The Reporting Service code for generating a CSV export based on Saved Object ID isn't "perfect" and still requires a lot of side-by-side catching up, and bloating of API params over time. We're also handling and passing through JSON that we don't understand why is needed. The solution are are looking for is already on the @elastic/kibana-app-arch radar here: #65069. As I've been working with that team and asking questions about how to fix things in Reporting, I've been tracking their efforts to un-bundle their logic from Angular and get us to a point where a plugin like Reporting can share their source of data access on the server. I hope that helps! Sorry that it will still take some time to get to a better API for CSV export. |
Kibana version: 7.6.2
Elasticsearch version: 7.6.2
Server OS version: Docker image
Browser version: —
Browser OS version: —
Original install method (e.g. download page, yum, from source, etc.): Docker
Describe the bug: After submitting a CSV from Saved Object reporting job with
/api/reporting/v1/generate/csv/saved-object/search:xyz, the executor fails withTypeError: Cannot read property 'settings' of undefinedSteps to reproduce:
failed, and error visible in server logs.Expected behavior: Reporting job should be executed successfully, and change its state to
completed.Provide logs and/or server output (if relevant):
Any additional context:
The text was updated successfully, but these errors were encountered: