docs: ADR for pagination and repr of single taxonomy view api [FC-0030] #72

ChrisChV · 2023-08-11T17:56:19Z

This PR adds an ADR to define the representation of the tags and the pagination in the single taxonomy view api

Reference issue: openedx/modular-learning#75

openedx-webhooks · 2023-08-11T17:56:23Z

Thanks for the pull request, @ChrisChV! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

docs/decisions/0014-single-taxonomy-view-api.rst

pomegranited · 2023-08-14T02:06:05Z

docs/decisions/0014-single-taxonomy-view-api.rst

+**Cons**
+
+- The children would not have pagination, in the long run there may be cases in which
+  the branch has hundreds of children, and they would still all be brought.


If we need to mitigate this, we could enforce a "maximum number of tags per branch" during tag import.

I would not be sure about this, if for the MVP we are already thinking of importing large taxonomies, it would not work to set limits (even if they are large)

We can ask the product folks, but I don't think this restriction is going to work. This whole approach assumes that there are a lot of root tags and very few children per root, but I think the opposite will generally be more common: only a few root tags and potentially thousands of child or grandchild tags. Think of the tree of life used in Biology - it has only three root tags (Bacteria, Achaea, and Eukaryota) but there are over 1 million species in the taxonomy. Of course that's an extreme case, but I think that shape of tree is more common than many root tags and few children. See also LabXchange, or Amazon product categories, or anything like that.

Oh Yes, you are right, it makes sense. I think we can mitigate the "Cons" of "Get the branch in another call"

docs/decisions/0014-single-taxonomy-view-api.rst

pomegranited · 2023-08-14T02:18:48Z

docs/decisions/0014-single-taxonomy-view-api.rst

+
+- The children would not have pagination, in the long run there may be cases in which
+  the branch has hundreds of children, and they would still all be brought.
+


I agree with your chosen option @ChrisChV , that makes total sense.

There's two more use cases implied by the UI:

Sort A-Z or Z-A

This sorts the top-level tags, not the child tags underneath (cf figma comment).

Search for tags in a taxonomy

This one's a little more complicated since it could be done a couple of ways.

Options I see:

We support tag search on the backend, and return a subset of matching tags in the format proposed here.
It means a backend API hit every time someone searches for a tag, but will scale.

We constrain the number of tags allowed in a taxonomy for MVP, so that the API can return all the tags in one page.
This allows the frontend to handle tag search all on its own, which is performant, but doesn't scale.

@ChrisChV and @bradenmacdonald How do you think we should handle this?

@pomegranited What's the eventual limit on taxonomy size going to be, post-MVP? If it's fairly high I think we should just do the backend version now. If we're never going to allow huge taxonomies, let's do frontend.

@bradenmacdonald Yeah, I don't think there's an upper bound planned for post-MVP, and there's already huge taxonomies being discussed, as you noted. So I agree: using the backend for search is the safest (and simplest) option.

@bradenmacdonald @pomegranited Considering taxonomies like this, I also agree with the search option in the backend. But another concern arises, in the interface to tag an object, will all the tags be brought in as initially thought? I think this page should use this same view (with pagination) and the same search

@ChrisChV Given the huge taxonomies shown, I think we'll have to find a different approach for that page. We can't bring in all the tags as we initially thought. It would be too big.

I was thinking of using the same view for both screens, since the screen to tag an object must also be paginated, I'm going to ask UX about this

mphilbrick211 · 2023-08-22T15:34:50Z

Hi @ChrisChV! Is this pull request ready for review?

ChrisChV · 2023-08-22T15:48:15Z

Hi @mphilbrick211, yes it's ready for review. @bradenmacdonald It's ready for your review

bradenmacdonald

Hmm, I like most of your ideas but I'm not sure if this approach is going to work. Let's discuss with UX/product and see what they say.

docs/decisions/0014-single-taxonomy-view-api.rst

bradenmacdonald · 2023-08-22T17:20:50Z

docs/decisions/0014-single-taxonomy-view-api.rst

+**Cons:**
+
+- In the UI there is the functionality *Expand all*, another view would have to 
+  be made to handle this functionality in a scalable way.


I personally like the "Get the branch in another call" approach the most. It's simple: first load only the first page of root tags, and then the user can load additional pages of root tags or expand any root tag and load [the first page of] its children. And so on, recursively.

If the main con is

In the UI there is the functionality Expand all, another view would have to be made to handle this functionality in a scalable way.

I think we can easily work around that by disabling the "Expand all" option for taxonomies that have > 1,000 tags.

Likewise, for taxonomies with < 1,000 tags we can pre-load all the data from the whole taxonomy.

But don't go changing the ADR just yet - let's ask on Slack for input.

I asked on Figma.

I updated the document with what was discussed

bradenmacdonald · 2023-08-22T17:21:11Z

docs/decisions/0014-single-taxonomy-view-api.rst

+
+- In the UI there is the functionality *Expand all*, another view would have to 
+  be made to handle this functionality in a scalable way.
+- A user could make many calls; every time a parent is opened.


That's fine, these calls should be extremely fast and performant to serve.

bradenmacdonald · 2023-08-22T17:33:00Z

docs/decisions/0014-single-taxonomy-view-api.rst

+**Cons**
+
+- The children would not have pagination, in the long run there may be cases in which
+  the branch has hundreds of children, and they would still all be brought.


We can ask the product folks, but I don't think this restriction is going to work. This whole approach assumes that there are a lot of root tags and very few children per root, but I think the opposite will generally be more common: only a few root tags and potentially thousands of child or grandchild tags. Think of the tree of life used in Biology - it has only three root tags (Bacteria, Achaea, and Eukaryota) but there are over 1 million species in the taxonomy. Of course that's an extreme case, but I think that shape of tree is more common than many root tags and few children. See also LabXchange, or Amazon product categories, or anything like that.

bradenmacdonald · 2023-08-22T17:39:29Z

docs/decisions/0014-single-taxonomy-view-api.rst

+- List of root tags that can be expanded to show children tags.
+    - This list can be sorted alphabetically: A-Z (default) and Z-A
+- The user can expand all root tags.
+- The user can search for tags.


We need to clarify in this ADR how this search is going to work, and how it interacts with pagination.

My suggestion is something like this:

On the backend when serving this API, if there is a search term like "cat", filter all the tags to only those that contain the string "cat" as well as their ancestor tags.

If the total number that matched is less than ~200, send them all to the frontend in a JSON tree structure.

If the total number that matched is more than that, send only [the first page of] root tags to the frontend. Each root tag can be expanded by another API call, and/or additional pages of root tags can be loaded.

But let's discuss this with product and UX team before making too many changes here.

@bradenmacdonald I have updated the document with what was discussed.

I've been thinking, if we are going to return complete taxonomies that have less than 1000 tags, we can do a search in the frontend with these taxonomies. I have not added it to the document, because I think that many conditionals are already being added and it will complicate the implementation. What do you think?

Yes, I think so too. Although I also don't want to complicate things too much by having two different implementations of search, based on the taxonomy size. Maybe there's some way we can make the "< 1,000 so load all and search via frontend" just a subset / special case of the general version. Something like "if all tags are loaded into the frontend, do a local filter, else do a server-side search and refresh the tags shown" would work in both cases.

bradenmacdonald · 2023-08-22T17:41:12Z

@mphilbrick211 Is the "FC" / funded contribution label missing from this repo, or has it been removed in general? This is an FC PR.

mphilbrick211 · 2023-08-22T19:13:52Z

Hi @bradenmacdonald - looks like the label isn't in this repo.

For FC projects, would you mind please putting the FC-XXXX ID number in the PR title? That will also help me know which project it is.

bradenmacdonald · 2023-08-22T20:36:47Z

@mphilbrick211 Yep, we try to do that generally, sorry we missed it here. I've added it now and created the FC label too.

bradenmacdonald

Looks good to me, just some minor cleanups of the text. Are you happy with this @ChrisChV ?

docs/decisions/0014-single-taxonomy-view-api.rst

bradenmacdonald · 2023-08-28T21:55:09Z

docs/decisions/0014-single-taxonomy-view-api.rst

+It was taken into account that taxonomies commonly have the following characteristics:
+
+- It have few root tags.
+- It have a very large number of children for each tag.


Suggested change

- It have a very large number of children for each tag.

- It may have a very large number of children for each tag.

docs/decisions/0014-single-taxonomy-view-api.rst

bradenmacdonald · 2023-08-28T22:00:42Z

FYI @ormsbee , please review this ADR for pagination of tags if you are interested.

ChrisChV · 2023-09-11T18:08:28Z

Hi @bradenmacdonald, I think we can merge this ADR

openedx-webhooks · 2023-09-12T20:39:48Z

@ChrisChV 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

docs: ADR for pagination and repr of single taxonomy api

4c82871

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Aug 11, 2023

pomegranited reviewed Aug 14, 2023

View reviewed changes

ChrisChV added 2 commits August 15, 2023 14:46

docs: Added use cases

035e91e

docs: Added search tags decision

ebf505f

bradenmacdonald reviewed Aug 22, 2023

View reviewed changes

bradenmacdonald changed the title ~~docs: ADR for pagination and repr of single taxonomy view api~~ docs: ADR for pagination and repr of single taxonomy view api [FC-0030] Aug 22, 2023

bradenmacdonald added the FC Relates to an Axim Funded Contribution project label Aug 22, 2023

ChrisChV added 3 commits August 23, 2023 14:07

docs: Use cases updated

d62f053

docs: Update decision to get children in another view

b3fe842

docs: Add paginated search

940f8f8

bradenmacdonald approved these changes Aug 28, 2023

View reviewed changes

ChrisChV added 2 commits August 29, 2023 14:16

docs: Nits and updates

10a02a0

docs: nit

dc4b792

bradenmacdonald merged commit ea311e1 into openedx:main Sep 12, 2023

bradenmacdonald deleted the chris/single-taxonomy-view-ADR branch September 12, 2023 20:39

ormsbee mentioned this pull request Mar 24, 2025

add vscode pytest settings open-craft/openedx-learning#19

Closed

ormsbee mentioned this pull request Sep 4, 2025

version bump 0.28.0 open-craft/openedx-learning#20

Closed


		- The children would not have pagination, in the long run there may be cases in which
		the branch has hundreds of children, and they would still all be brought.

	- It have a very large number of children for each tag.
	- It may have a very large number of children for each tag.

docs: ADR for pagination and repr of single taxonomy view api [FC-0030] #72

docs: ADR for pagination and repr of single taxonomy view api [FC-0030] #72

Uh oh!

Conversation

ChrisChV commented Aug 11, 2023

Uh oh!

openedx-webhooks commented Aug 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChrisChV Aug 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mphilbrick211 commented Aug 22, 2023

Uh oh!

ChrisChV commented Aug 22, 2023

Uh oh!

bradenmacdonald left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bradenmacdonald Aug 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bradenmacdonald commented Aug 22, 2023

Uh oh!

mphilbrick211 commented Aug 22, 2023

Uh oh!

bradenmacdonald commented Aug 22, 2023

Uh oh!

bradenmacdonald left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bradenmacdonald commented Aug 28, 2023

Uh oh!

ChrisChV commented Sep 11, 2023

Uh oh!

openedx-webhooks commented Sep 12, 2023

openedx-webhooks commented Aug 11, 2023 •

edited

Loading

ChrisChV Aug 15, 2023 •

edited

Loading

bradenmacdonald Aug 22, 2023 •

edited

Loading