Action Plan for Thanos User Interface and UI related APIs Improvements

[bwplotka]

This is going to be epic! 💪 We just had an amazing hour-long UI-focused community call with @d-ulyanov @adrien-f @geobeau @vankop Thanks All for joining!

I am still impressed by the quick response on our call for UI hero initiative ❤️ So many heroes! 💪

Meetings Notes ended up to be Thanos UI Design Doc, where we discussed goals, requirements and action plan for future of UI in Thanos. This issue is to track those. Everyone else: Please review, comment and give the feedback on this! Especially @thanos-io/thanos-maintainers ((: @juliusv your opinion would be nice as well 👍

UI Requirements & Goals

• Simple to maintain by the backend engineers. Simple React Code. (e.g. Prometheus React did not get very var so let's keep that).
  • If we want to reuse some Prometheus UI code, we need to somehow easy to track the modifications e.g. Navigation. Keep it as close as Prometheus!
• Benchmark/Integration test for UI to catch bugs early.
• Make sure it works for scale cases. (a large number of labels). UI should not lag.
  • Sometimes might involved backend API optimizations.
  • Some things are special only for Thanos. E.g. for Prometheus it's rarely the case you have billions of metric names. In Thanos, we can have easily.
    • Open question: Would Prometheus accept a PR to improve this, even scale is lower on Prometheus side? @juliusv
• It should be consistent with ALL components.
  • Flags / Configuration
  • Build Info
  • Idea: Discovery page for all APIs (our additional, maybe portable to Prometheus as well!)

Non-Goals

• Build Dashboards / Replace Grafana. Improve graphing majorly etc. That's out of scope.
• Make UI optional: Headless. Not having node JS as a dependency. It would be nice for local dev. We decided it's out of scope for now.

Action Plan

This is the current action plan. Please comment on this issue if you want to work on this! 🤗 I will update this description with up-to-date info!

• Finalize the base of React from Prometheus from Adrien. Just merge the graph. Help with Review wanted! (:
  • Assignee: @adrien-f
• Add React UI tests / benchmark with large number of label values etc.
  • Some of it @vankop
• Add fuzzy matching for large number of label values.
• Port Querier Store Page to React.
• Rename Bucket Viewer to Block Viewer.
• Port Block Viewer to React.
• Enrich Block viewer as mentioned in Improve Bucket Web UI view: Front-end hero wanted! 💖 #2149
• Create Base UI
• Add UI to Store Gateway
• Create Discovery page
• Federated Rule UI
  • Federated UI
  • Federated API: Assignees @s-urbaniak @bwplotka
    • Design Doc
    • feature branch: Merged Experimental Federated Rules API feature branch to master. #2200

[s-urbaniak]

ping @kyoto Thanos needs help upstream in terms of frontend work. Maybe this is something we could help here? :-)

[vankop]

I can add tests, fix bugs in terms of scaling if any..

As a plan for tests:

1. stub Thanos API responses with big amount of data (metric labels, what else?)
2. write integration tests using https://www.cypress.io/
   a. I think tests with happy path will be enough (e.g. open graph page -> type query -> press "execute" -> result displayed)
   b. maybe wrap all run in docker, so this will be easier for backend developers to run tests (also as I remember cypress has awesome feature to disable stubing requests as opt-in, so tests will run as e2e)

[juliusv]

Some things are special only for Thanos. E.g. for Prometheus it's rarely the case you have billions of metric names. In Thanos, we can have easily.
Open question: Would Prometheus accept a PR to improve this, even scale is lower on Prometheus side? @juliusv

I think it depends on how Thanos-specific that gets, and how much of a maintenance burden that feature would be.

[kushthedude]

Hey @bwplotka , I am interested in working on this issue for GSoC. I have setted up thanos on my local. Any suggestions or guidelines I need to know to get started.

[bwplotka]

@juliusv @adrien-f at some point you said that React version of Prometheus is still in development, so we should wait until more stable. @juliusv is this the case now? (:

We proposed this as our GSoC project as well with main mentor being @squat and me.

[bwplotka]

@kushthedude please follow GSoC procedure to apply.

[juliusv]

@bwplotka There's still stuff happening (additions and cleanups), but the new React UI is getting pretty settled. So this at least shouldn't stop this.

[heyAyushh]

Hi, I am a GSOC applicant and would love to work on this issue, I've worked a lot in react and can also reuse Prometheus UI code, just a need mentors to guide and command me.

[bwplotka]

GSoC proposal was done, so hopefully, we will have someone full time on this: https://github.com/cncf/soc#enriching-and-extending-thanos-uis-with-react-for-awesome-user-experience

Also please share the doc linked here in your countries and universities so we have diverse proposals. 🤗

[heyAyushh]

okay, I'll share it.
Thanks

[stale]

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

[bwplotka]

@prmsrswt is on it (:

[GiedriusS]

Some small observations after the initial import in #2412 have been merged.

I think we should work on these in the "old" UI:

• query: add tests for endpoints + prefix tests #2582
• query: --web.route-prefix makes UI inaccessible #2581

Ideally, these two would be done before the next release. This would be a nice thing but only if you want to do: #2578.

I think we should work on these next in the "new" UI:

• Figure out a plan on how we will update the Prometheus code in the future? How to separate it from all others? For the moment, let's try to keep the React components in a separate directory as much as possible and if there are mistakes in the Prometheus code then submit the fixes upstream. Worth thinking about it but not a blocker for now.

As @adrien-f said before:

The goal is to be able to just copy paste the source from upstream and let the Thanos directory do the rest to ease upgrades.

• Add a drop-down for selecting a resolution in the new UI;
• http://localhost:10902/new/graph?g0.expr=&g0.tab=1&g0.stacked=0&g0.range_input=1h should not give us a blank page with no stores. The result is null hence the page errors out and is blank but we should handle this case;
• Add a "stores" page by using the new /api/v1/stores endpoint.

I also have one response to @bwplotka 's concern in the original post:

Make sure it works for scale cases. (a large number of labels). UI should not lag.

Actually, the new UI has this code:

                    {matches
                      .slice(0, 100) // Limit DOM rendering to 100 results, as DOM rendering is sloooow.
                      .map(({ original, string: text }) => {                                                                

So it shouldn't lag. We could still improve the API itself to support pagination but at least the new UI won't try to show potentially thousands of metrics.

---

Edit: also found this small bug prometheus/prometheus#7241 upstream :P

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

This is in quite heavy progress! Do you mind @prmsrswt to share some update at some point (: Maybe there is someone interested in the progress 🤗

[onprem]

Yeah sure :)

We are steadily making progress in this front. Some of the work done till now includes

• We have merged the initial UI with /graph endpoint working. Thanks to @adrien-f 🎉 (ui: add new React UI from Prometheus #2412)
• We now have a Error boundary in the React UI, so we show a simple page with "Something went wrong" instead of just a blank page. (ui: Add error boundary in the React UI #2759)
• The PR to add /stores page is in review. You can track it's progress (and maybe help in review 😃) here ui: Add Stores page to React UI #2754

There were many small but important improvements too, for example we fixed the UI to work with --web.route-prefix and now we have an e2e test for it too.

Also, there's so much more in the store. Some immediate goals we are working towards are, having a Configuration page, where you can see the cli flags and values of various configuration files, and writing some e2e tests for --web.external-prefix as well, so we don't break them in future (hopefully).

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[onprem]

Guess this calls for another status update. 😉

• We have fully migrated the Query UI to React. It should now be on feature parity with the old UI.
• The initial work for migrating the Ruler UI is done (ui: Migrate Thanos Ruler UI to React #2865 )
• We are now working on migrating the Block viewer (or, bucket viewer) to React. This should solve many issues with the block viewer we have right now.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[bgilmore]

Hoping this is still in progress? Looking forward to the new block/bucket viewer.

[onprem]

@bgilmore Yes, it is.

The new block viewer already got merged and made it's way to the latest release. If you are on Thanos v0.15.0 you can try it out. It should already be on feature parity with the old block viewer and it has some additional improvements as well.

A few things we are planning to add in the block viewer in the future:

• Details about the block and index size
• Showing if a block is marked for deletion or not.
• Allowing some actions from the UI itself, like marking for deletion, downloading meta.json, etc.

[kunal-kushwaha]

@bgilmore Yes, it is.

The new block viewer already got merged and made it's way to the latest release. If you are on Thanos v0.15.0 you can try it out. It should already be on feature parity with the old block viewer and it has some additional improvements as well.

A few things we are planning to add in the block viewer in the future:

• Details about the block and index size
• Showing if a block is marked for deletion or not.
• Allowing some actions from the UI itself, like marking for deletion, downloading meta.json, etc.

Hey @prmsrswt @bwplotka, I'd like to work on this (ref: #3112) for the Community Bridge program. Can you point me towards the plans we have for the UI changes to be made? Discussions/Docs/Notes, anything would help. Thanks!