Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Directly use Zed lake query endpoint #1618

Closed
philrz opened this issue May 6, 2021 · 2 comments · Fixed by #1792 or #1836
Closed

Directly use Zed lake query endpoint #1618

philrz opened this issue May 6, 2021 · 2 comments · Fixed by #1792 or #1836
Assignees
Milestone

Comments

@philrz
Copy link
Contributor

philrz commented May 6, 2021

As described in brimdata/super#2680, the current API is allowing Brim to access data in pools via the legacy "search" endpoint, but there's functionality such as join from multiple pools that will only be possible when the app starts directly using the new Zed lake "query" endpoint.

@philrz philrz added this to the Data MVP1 milestone May 12, 2021
@philrz philrz changed the title Brim: Directly use Zed Lake Service API Brim: Directly use Zed lake query API Sep 14, 2021
@philrz philrz changed the title Brim: Directly use Zed lake query API Brim: Directly use Zed lake query endpoint Sep 14, 2021
@philrz philrz modified the milestones: Data MVP1, v0.26.0 Sep 14, 2021
@philrz philrz changed the title Brim: Directly use Zed lake query endpoint Directly use Zed lake query endpoint Sep 14, 2021
@philrz
Copy link
Contributor Author

philrz commented Sep 15, 2021

I took a shot at verifying the work thus far with Brim 0dcf10c by attempting this multi-pool join derived from the Zed docs:

from (
  fruit.ndjson => sort flavor;
  people.ndjson => sort likes;
) | inner join on flavor=likes eater:=name

This produced a backend error Upstream data source blocked by 'from operator'. I could see in Wireshark that the query being sent to the backend was:

{"query":"from '0x0dcfc31bd80817b052e0c8eb90a8297af928a8ea' | from (   fruit.ndjson => sort flavor;   people.ndjson => sort likes; ) | inner join on flavor=likes eater:=name | head 10000"}

@nwt proposed a fix for this:

Rather than prepend a from to the query, the app should set the head field in the request, like this.

{
 "query": "from (   fruit.ndjson => sort flavor;   people.ndjson => sort likes; ) | inner join on flavor=likes eater:=name | head 10000",
 "head": "0x0dcfc31bd80817b052e0c8eb90a8297af928a8ea"
}

@philrz philrz linked a pull request Sep 16, 2021 that will close this issue
@philrz
Copy link
Contributor Author

philrz commented Sep 16, 2021

Verified in Brim commit de44850.

Now with the benefit of the additional fix from #1836, we can execute the multi-pool join shown in the Zed docs from within the Brim app, as shown in the attached video:

Verify.mp4

In a discussion with the Dev team, it was confirmed that the current fix added in #1836 is only limited to queries where the from leads off the query entered by the user in the search box. That is, queries with from nested somewhere deeper inside the Zed are not expected to work yet.

Thanks @mason-fish!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants