Reporting URL from Discover has no relative time range mode

[PhaedrusTheGreek]

When exporting data as CSV from the Discover application, the job parameters use an absolute time range that was co-erced into the POST URL:

http://localhost:5601/api/reporting/generate/csv?jobParams=(conflictedTypesFields:!(),fields:!(_id,_index,_score,_type,search.columns,search.description,search.hits,search.kibanaSavedObjectMeta.searchSourceJSON,search.sort,search.title,search.version,type,updated_at),indexPatternId:%27918bb7f0-a497-11e8-8fe1-3991a3b1b556%27,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!()),docvalue_fields:!(updated_at,url.accessDate,url.createDate),query:(bool:(filter:!(),must:!((match_all:()),(range:(updated_at:(format:epoch_millis,gte:1534808045652,lte:1534811645652)))),must_not:!(),should:!())),script_fields:(),sort:!((_score:(order:desc))),stored_fields:!(%27*%27),version:!t),index:%27*%27),title:%27New%20Saved%20Search%27,type:search)

Time range portion appears to be: range:(updated_at:(format:epoch_millis,gte:1534808045652,lte:1534811645652))

In contrast, PDF export from Dashboard appears to have the desired behavior:

http://localhost:5601/api/reporting/generate/printablePdf?jobParams=(browserTimezone:America%2FToronto,layout:(id:print),objectType:dashboard,relativeUrls:!(%27%2Fapp%2Fkibana%23%2Fdashboard%2F98aee040-a495-11e8-8fe1-3991a3b1b556%3F_g%3D(refreshInterval:(display:Off,pause:!!f,value:0),time:(from:now-1h,mode:quick,to:now))%26_a%3D(description:!%27!%27,filters:!!(),fullScreenMode:!!f,options:(darkTheme:!!f,hidePanelTitles:!!f,useMargins:!!t),panels:!!((embeddableConfig:(),gridData:(h:15,i:!%271!%27,w:24,x:0,y:0),id:!%2700f9e390-17e0-11e8-bf14-c9dbfc6e6269!%27,panelIndex:!%271!%27,type:visualization,version:!%276.3.1!%27),(embeddableConfig:(),gridData:(h:15,i:!%272!%27,w:24,x:24,y:0),id:!%2712d5aed0-cc60-11e7-aadb-7b129851f054!%27,panelIndex:!%272!%27,type:visualization,version:!%276.3.1!%27),(embeddableConfig:(),gridData:(h:15,i:!%273!%27,w:24,x:0,y:15),id:!%27248ac390-cc60-11e7-aadb-7b129851f054!%27,panelIndex:!%273!%27,type:visualization,version:!%276.3.1!%27)),query:(language:lucene,query:!%27!%27),timeRestore:!!f,title:!%27New%2BDashboard!%27,viewMode:view)%27),title:%27New%20Dashboard%27)

Time range portion appears to be: time:(from:now-1h,mode:quick,to:now)

Relative range is necessary when automating report generation from watcher, for example, if we want to generate a CSV and send to email every day, we need to have now-1d to now as the range.

As a workaround, the URL could be modified by hand like so:

http://localhost:5601/api/reporting/generate/csv?jobParams=(conflictedTypesFields:!(),fields:!(_id,_index,_score,_type,search.columns,search.description,search.hits,search.kibanaSavedObjectMeta.searchSourceJSON,search.sort,search.title,search.version,type,updated_at),indexPatternId:%27918bb7f0-a497-11e8-8fe1-3991a3b1b556%27,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!()),docvalue_fields:!(updated_at,url.accessDate,url.createDate),query:(bool:(filter:!(),must:!((match_all:()),(range:(updated_at:(gte:now-1d,lte:now)))),must_not:!(),should:!())),script_fields:(),sort:!((_score:(order:desc))),stored_fields:!(%27*%27),version:!t),index:%27*%27),title:%27New%20Saved%20Search%27,type:search)

However URLs should be copied as generated.

[MorrieAtElastic]

A user I was talking to also needs this fixed and needs to be able to generate CSV report URLs with relative time intervals for the sake of periodic reports generated through watcher.

[tsullivan]

Tracking this issue to make sure that as we improve the API for generating reports, we will support absolute + relative time ranges in the request.

cc @joelgriffith

[tsullivan]

This is WIP for the new CSV export API: #34571

[tsullivan]

as we improve the API for generating reports, we will support absolute + relative time ranges in the request.

This is still being tracked as not started. The new CSV Export API still only has time range params as from and to absolute dates.

The job parameters for CSV report creation require a lot of browser-side Kibana APIs. This is an important enhancement to work on, for side reasons: we have other potential improvements to do whenever the browser-side APIs are involved.

[tsullivan]

I looked into this further and found that Reporting is gathering the query within the jobParams via referencing Discover code. That is how we want the code ownership split.

Reporting calls searchSource.getSearchRequestBody which is how we keep the source of truth of the query outside of Reporting.

Somewhere in the generation of the query body, the relative time range that we want is getting converted to an absolute time range.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[jderose9]

Just wanted to add that I am also experiencing this issue and would prefer that the dynamic range is preserved.

[collsni]

Also experiencing this issue when generating .csv reports with Watcher.

[collsni]

Example of my work around:

#not fixed

https://XXXX/api/reporting/generate/csv?jobParams=(conflictedTypesFields:!(),fields:!(%27@timestamp%27,user.name,client.ip,client.domain,vpn.profile,vpn.client.type),indexPatternId:ee646450-5994-11ea-a5a9-190fcdaa6b56,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!(),includes:!(%27@timestamp%27,user.name,client.ip,client.domain,vpn.profile,vpn.client.type)),docvalue_fields:!((field:%27@timestamp%27,format:date_time)),query:(bool:(filter:!((match_all:()),(match_phrase:(event.code:%27722055%27)),(range:(%27@timestamp%27:(format:strict_date_optional_time,gte:%272020-03-03T20:09:58.036Z%27,lte:%272020-03-04T20:09:58.037Z%27)))),must:!(),must_not:!(),should:!())),script_fields:(),sort:!((%27@timestamp%27:(order:desc,unmapped_type:boolean))),stored_fields:!(%27@timestamp%27,user.name,client.ip,client.domain,vpn.profile,vpn.client.type),version:!t),index:%27filebeat-*%27),title:%27ASA%20VPN%20-%20Client%20Type%27,type:search)

#fixed

https://XXXX/api/reporting/generate/csv?jobParams=(conflictedTypesFields:!(),fields:!(%27@timestamp%27,user.name,client.ip,client.domain,vpn.profile,vpn.client.type),indexPatternId:ee646450-5994-11ea-a5a9-190fcdaa6b56,metaFields:!(_source,_id,_type,_index,_score),searchRequest:(body:(_source:(excludes:!(),includes:!(%27@timestamp%27,user.name,client.ip,client.domain,vpn.profile,vpn.client.type)),docvalue_fields:!((field:%27@timestamp%27,format:date_time)),query:(bool:(filter:!((match_all:()),(match_phrase:(event.code:%27722055%27)),(range:(%27@timestamp%27:(format:strict_date_optional_time,gte:now-1d,lte:now)))),must:!(),must_not:!(),should:!())),script_fields:(),sort:!((%27@timestamp%27:(order:desc,unmapped_type:boolean))),stored_fields:!(%27@timestamp%27,user.name,client.ip,client.domain,vpn.profile,vpn.client.type),version:!t),index:%27filebeat-*%27),title:%27ASA%20VPN%20-%20Client%20Type%27,type:search)

Replaced (range:(%27@timestamp%27:(format:strict_date_optional_time,gte:%272020-03-03T20:09:58.036Z%27,lte:%272020-03-04T20:09:58.037Z%27)))

With (range:(%27@timestamp%27:(format:strict_date_optional_time,gte:now-1d,lte:now)))

[tsullivan]

I investigating this in more depth in hopes of fixing it. Here is what I found:

• Reporting jobParams for CSV export contain the entire Elasticsearch query in order to export using the Reporting serverside framework
  • Note: this issue: [data.search.searchSource]: Expose SearchSource on server #65069 blocks us from simply using the ID of the saved search loaded in Discover
• Discover uses the data plugin in the browser to run the search. This part of the Data plugin seems to be the cause of the relative mode being converted to absolute parameters: https://github.com/elastic/kibana/blob/6a69e19/src/plugins/data/common/query/timefilter/get_time.ts#L50

I think a fix would be to wait for the necessary code for search generation is available on the server side so Reporting can use it properly. Then CSV export job params would just contain the saved search ID and the relative time filters *. The fact that relative time gets converted to absolute at the time of the search shouldn't matter.

• As well as browser time zone, in case a TZ is not set in Advanced Settings

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[sasubillis]

This feature is an absolute must to have. Without this feature, wasted many hours to simply edit the time range from absolute to relative.

[rbr101]

Any update on this, this is still not resolved.
Changing the copy POST url manually is really crap, scheduling csv reports not possible without custom scripting.

Please do something about it.

[tsullivan]

@rbr101 if you are seeing this in 7.16.x, the issue is here: #122575

There is a fix currently under review here: #123206

[rbr101]

@tsullivan

Yes I'm running 7.16.x, I can remember it worked before on version < 7.16.x.
So this was introduced again in 7.16. , for now I fixed it with an regex replace I my script.

Thanks for the update.