Composable, extendable and customisable decks

[axelboc]

Hey there, thanks for the update! Looks like you're making good progress. 🚀

I've been thinking about Brain Brew and what it could bring to Ultimate Geography in the long term, so I thought I'd share some long-term ideas. (@aplaice, this might interest you as well.)

---

The problem

Over the years, users of UG have asked for a number of extra fields to be added to the deck -- most notably IPA and audio pronunciation, but also languages, currencies, flag descriptions (vixillology), links to Wikipedia, etc.

These requests have so far been rejected or put on hold for various reasons, ranging from lack of user interest to maintenance burden (single CSV, long-term support, etc.)

IMO, the key to maintaining a good-quality open source project in the long term is to avoid scope creep - i.e. doing a few things really well rather than doing every possible thing poorly. So I've always been careful about keeping the deck as simple and uncluttered as possible. I prefer to have a vast majority of really happy users rather than to try-but-kinda-fail to satisfy everyone.

Brain Brew will obviously solve the CSV limitation, but the other issues remain... If a field has strong user interest and people are willing to maintain it in the long term, it would make sense to include it in the deck's repository, but then including the field in one or more note templates can still be problematic as it impacts every user. And this still leaves fields that don't have strong user interest or guaranteed long-term support...

If I understand correctly, the Personal Fields feature you've been working on will at least let users add more content to their deck while still allowing them to re-export or re-import it. This is a great step forward, but eventually, we'll need a leap, and I think Brain Brew can make that leap.

What we need

What we need is a way to make decks fully composable, extendable and customisable.

Users need the ability to compose, extend and customise a core deck in whichever way they want without having to actually modify that deck.

Let me start by explaining what I mean by those terms.

Composable

This is about allowing users to create, modify and remove templates from a deck.

A basic example of composability is UG's extended deck (confusing... 😄), which combines fields in two additional templates.

But there are plenty more use cases for composability. Who knows, perhaps somebody would like to have a template with capitals on the front and flags on the back? 🤷 Or perhaps they're sick of seeing Flag similarities and want to remove the field from the flag templates. Or perhaps they'd like to turn the Country info field into a hint field?

Going even further, perhaps a user has extended the deck with another field (see next section) and they'd like to create a template with this field, or modify one of the templates to include the field as additional information?

Extendable

This is about adding new content - i.e. fields, for instance currencies or IPA. Pretty straightforward.

Customisable

This is about customising the styles of the templates (font, color, spacing, etc.) People may find some fonts easier to read, or they may want to add icons to help them understand the cards faster.

---

A note on translated decks

Translated decks are perfect examples of both extendability and composability: new fields that are combined into new templates. (Technically, the templates in UG are the same for all languages, but that's just because Anki DM doesn't support translating templates.)

---

How do we get there?

Extension repositories

The way I see it, the first thing we need is a way to store "extended" fields in separate repositories. Yes, perhaps even translated fields!

The advantage of having "extension" repositories separate from the main deck's repository is clear: each "extension" repository would be free to live its life, to have its own maintainers, its own opinions (e.g. stylistic choices, simplified vs traditional Chinese characters, etc.), and so on. If an extension repository were to no longer be maintained or to become too opinionated, people could fork it, create an alternative version of it, or simply stop using it. Either way, the core deck would remain unimpacted.

Deck configuration

Then we need a deck configuration schema that allows:

• referencing a core deck's repository,
• referencing any number of extension repositories,
• adding/removing/replacing the core deck's templates (either with custom templates or with templates provided by an extension repository),
• customising the core deck's styles.

Ideally, the configuration schema would be "Brain Brew-agnostic", in the sense that it would not contain configuration specific to Brain Brew. Some sort of high-level format that could be processed by any tool.

People would share their deck configurations in their own repos, in Gists, or whatever.

Brain Brew support

Obviously Brain Brew would need to support this deck configuration format. 😄

Tooling

Users would then need an easy way to generate decks from shared configurations.

Installing Python, setting up Brain Brew, copying a deck configuration file, running the build command, importing in Anki, etc. It's way too much for most users. We need tooling to help, perhaps a desktop app. That's long, long term... 😄

---

What do you think?

First, what do you think about the general idea/plan/whatever the hell this was...? 🤣 Am I making any sense? Do you think extension repositories are the way forward? It felt good to try to identify which issues extendability, composability and customisability could resolve for us deck maintainers...

Obviously, this whole thing makes no sense if Brain Brew can't support it. It's already built for composability, so I guess the complexity is more around extendability. Do you think referencing and pulling content from other repositories is doable? I mean, the idea would really be for Brain Brew to run potentially outside of the core deck, in a folder without any content but a configuration file. Is this even plausible?

[aplaice]

Firstly, @ohare93 I'm extremely sorry that I still haven't had time to have a look at and test Brain-Brew!

---

@axelboc I think that composability, extensibility and customisability would all be extremely useful.

Do you think extension repositories are the way forward?

I think that they're a really interesting idea and could be great! I'm not sure if they're the way forward, but they seem better than the main alternatives I can think of (having all the "alternatives" in the core deck, as "optional" data/configurations, or of having the alternatives as forks of the core deck, that'd have to be constantly rebased after any update in the core).

Obviously I have no idea whether they'd work with Brain Brew. (I also don't want to pile even more pressure and expectations onto @ohare93...)

Tooling
Users would then need an easy way to generate decks from shared configurations.
Installing Python, setting up Brain Brew, copying a deck configuration file, running the build command, importing in Anki, etc. It's way too much for most users. We need tooling to help, perhaps a desktop app. That's long, long term... smile

In principle, it could mostly work as an Anki add-on, since Anki presumably bundles most of the relevant Qt libraries needed for a functional GUI.

However, as you say, it's very, very long term, and probably out-of-scope here since it'd involve modifying/replacing/merging with CrowdAnki.

[ohare93]

Wow, what an extensive and well thought out first issue for my little project here! 😉 👌 lots to reply to, and all of it good (even the 🥧 in the sky parts! 😄) I am usually not on the receiving end of 1000 word paragraph replies, but I am happy to receive such! (And apologies for the length of my own reply 😭)

Replies to random parts

IMO, the key to maintaining a good-quality open source project in the long term is to avoid scope creep - i.e. doing a few things really well rather than doing every possible thing poorly. So I've always been careful about keeping the deck as simple and uncluttered as possible. I prefer to have a vast majority of really happy users rather than to try-but-kinda-fail to satisfy everyone.

I fully agree 💯%! Though I have desired more in the past (Edinburgh of the Seven Seas 😭) and still argue that the extended little islands are relevant, I can concede that having these as part of an offshoot deck as part of this extendable project you propose is a much better solution in the end 👍

---

Extension repositories

The way I see it, the first thing we need is a way to store "extended" fields in separate repositories. Yes, perhaps even translated fields!

The advantage of having "extension" repositories separate from the main deck's repository is clear: each "extension" repository would be free to live its life, to have its own maintainers, its own opinions (e.g. stylistic choices, simplified vs traditional Chinese characters, etc.), and so on. If an extension repository were to no longer be maintained or to become too opinionated, people could fork it, create an alternative version of it, or simply stop using it. Either way, the core deck would remain unimpacted.

A great idea, and one that I have been dabbling in myself lately. Have you considered using git submodule to add a git repo to a git repo? I will soon start using this to separate my Danish and Esperanto language learning cards into their own repos, but combine / separate them using Brain Brew in a parent repo. This way the data is silo'd and separate, while still being usable for Brain Brew (it is as long as Brain Brew can find the given filepath to read/write!)

This type of style could work for what you are proposing, but may not be necessary if the future goal of a version of Brain Brew that can read directly from the repos themselves is achieved.

Federation of UG

😏

In this hypothetical future, where there are a small Federation of UG deck translations, extensions, and improvement repositories, would you consider creating a Git Organisation for all of the "official" ones to live under? What to deem "official" is up for debate, though I'd argue that translations of the core deck easily fit. An organisation would have many benefits:

• Easier for people to find translations / extensions / improvements (any other categories? 😅)
• Can assign "owners" of the repos (such as the people doing the translating/updating) and simply "@deckowners" or some such for a breaking change to the core deck, rather than having to have an ever expanding list
• Easy permission system for trusted users to be able to fix issues on any/all repo
• More community feeling and cohesion. When a change is made to one deck, other organisation members are much more likely to see than if we're a bunch of satellite randos not caring what goes on where. If that change impact their deck, or is a better way to do things, then people can react 👍
• Central place to discuss potential offshoot decks/ideas, or meta changes to all the decks in the Projects panel of an Organisation. Would also keep the issues of the Core deck neat and tidy.

Just a thought I had from you proposing this, not a deal breaker 😁

---

If I understand correctly, the Personal Fields feature you've been working on will at least let users add more content to their deck while still allowing them to re-export or re-import it.

It will allow any user of CrowdAnki to select any field in any import as "Personal" so that the value will not overwrite an existing entry. The value of a "Personal" field will still be taken for new cards though.

In this way deck managers can choose to add a default value to a card, by setting that value or not in their data in Brain Brew. Also they can will soon be able to set a field a Personal on the CrowdAnki import file itself, so that it auto defaults to that on import. The user can still change it to not Personal, but this will minimise the accidental incorrect import. (This feature cannot be implemented in Brain Brew until it is pulled into CrowdAnki first)

More info available on my Patreon post here. (I don't shill in your repo, but I sure can shill in mine! 😉)

---

Needs

Alright, let me summarize the things you believe (plus some of my own) are needed to make a great system going forward. Though I will remove the categorisation of "Composable, Extendable, and Customisable" in favour of my preferred way to look at this: Deck Parts.

In order to be able to untangle this mess of scope and create a more Federated approach to Anki deck moderation you state (and I agree) that we will need to give the users full choice over the following items upon Import:

1. Note Models

1. Card Types (aka "templates" as they contain the front and back template). "Composable" in your system.
2. Note styling. "Customisable" in your system.
3. Note Fields. "Extendable" in your system.

For any of the card types users should be able to:

• Create
• Modify
• Remove

❗ Apart from Note Styling! The css here is used for every Card Type on a Note type. Though I do have a general idea about programmatically being able to set styling separately on each card by the use of ".card1" anki css and simply combining all the individual parts to make a whole, but it is ofc much simpler to just use the one css as that is how Anki is designed 😅

2. Notes

Users should be able to take extra notes into the deck, plain and simple. If someone wants to extend the deck with their offshoot repo of American states but still fit into the Ultimate Geography framework Ultimate Geography then that should be possible.

The deck as it currently stands could even be broken down into categories like Independent Countries, Territories, Seas/Oceans, etc 🤔 Users could then have an option for which to include. Though this "chose what you want to import" functionality will be covered in one of my PRs for CrowdAnki to filter the import based on the tags 👍

3. Media

Notes encompass most the media of the deck. Though media can also be referenced and used in a Note Model (in card template or styling) as long as the file begins with an underscore. Not naming it so will result in a working card, but the media file will not be referenced correctly and thus will not export with a note and may be found in the "Check Media" option as unused.

Regardless of where the media file is referenced, it is worthwhile to mention that it would be good to be able to replace the media of the deck with other media 🤔 This would be an option taken by the guy just today who raised the issue about media format for maps not being SVGs 😅 and I'm sure there are other use cases.

4. Deck Headers

The stuff that makes a deck itself: the name, config/review options, and description. It would be good for others to be able to translate the deck description and name. However I am currently trying to push for these to be optional default values used only if you do not already have the cards. I improved this already with the "Do Not Move Existing Cards" PR to CrowdAnki, however if new cards are imported then the Deck Headers are used to make a new deck with which to place them. Again, I hope my Import Options CrowdAnki PR can fix this issue by giving the user the option of where to put the cards, if no Deck Headers are desired/available.

I hope that in the future a CrowdAnki deck could be imported without any Deck Headers and that the user would just be prompted on import what the do with the new and existing cards. I am working on that as an option in CrowdAnki just now.

---

Brain Brew's Current Functionality

I am very happy to report that each of the requirements stated above are already functional in the current version of Brain Brew 🎉

Now let me list the caveats 😆

1. Note Models are currently all or nothing: each is packaged with the fields, styling, and templates. Though technically one can achieve the above requirements, but there would be massive duplication of data for small changes 😭 This is a current top priority (see Ongoing Work below)
2. Csvs are the only non CrowdAnki format currently supported (Yaml on the way).
3. Media from Note Models is not fully functional yet
4. Media is not settable per deck generation yet

And of course all that says nothing about the fact that one must still do this from the command line 😅 though if one were to create a few split up repos of UG, as discussed in the Federation section above, and simply have them locally on your computer then currently you could use Brain Brew to read data from and/or write data to each to build and manage a deck!

Obviously, this whole thing makes no sense if Brain Brew can't support it. It's already built for composability, so I guess the complexity is more around extendability.

Extending generated decks with more fields is already working 👍 See this Yaml Builder Config file here. If one wished to add more notes then simple add another "csv_file_mapping". To add more fields add another "derivative" under that, and make sure each of the fields in your csv are listed in "csv_column_to_fields" under "note_mode_mappings" has a partner matching field (or put it in the "personal_fields" list. See my Brain Brew Starter Project for a working version with comments.

I realise all of this may not have been clear of obvious due to the complexity of the task and my unclear communication about it in the past. I have invested some time into the documentation (though full code documentation will be later as major refactoring is occuring now), so if there is anything else that is unclear do let me know and I will try and make a writeup to explain it 👍

Ongoing Work

1. Personal Fields [Waiting for PR in CrowdAnki to be accepted]
2. Refactoring Deck Parts into Yaml files, and minimising their requirements (no file written if not needed)
3. Yaml Source file (easier after the above is complete, and more complicated than you would have originally thought!)
4. Note Model generation and extraction

After these key tasks are done I will most likely invest some time into a proper demo of Brain Brew, as I have not had an official first publication, outside of a disjointed Discord chat and posting on the UG thread 😅 The above changes will make things muuuch easier for others to understand and contribute their own changes to the project, and I have high hopes that the lofty goals will attract likeminded helpful souls who simply wish for their specific Source type to be implemented and so they do it themselves 😁

---

Brain Brew in the future

@aplaice

In principle, it could mostly work as an Anki add-on, since Anki presumably bundles most of the relevant Qt libraries needed for a functional GUI.

Brain Brew could indeed be an Anki add-on, as I have made it quite minimally dependent. I also (from adding the visual config window to CrowdAnki) have some experience with making the UI itself, and could simply generate the current builder config files from a ui options, instead of the user filling them out beforehand just now. This would maintain the use of Brain Brew as a Maintainers command line tool, while allowing normal users to benefit too.

However, as you say, it's very, very long term, and probably out-of-scope here since it'd involve modifying/replacing/merging with CrowdAnki.

I do not foresee a full replacement of CrowdAnki. It just does too much. Though if in the future there comes constant challenge/denial to future PRs going through then I suppose I could see myself forking and building ontop of that 🤔 begrudgingly... 😞 CrowdAnki does have an MIT license, so that wouldn't be an issue.

I would much prefer a model where Brain Brew is it's own add-on, but has a dependency on CrowdAnki also being installed. In this system Brain Brew would simply call the functions of CrowdAnki using Anki Hooks. So if a user chose to import a deck in some way using Brain Brew, then it would do all the transformation of Csv/Yaml/etc into a CrowdAnki file in a temporary folder, and behind the scenes CrowdAnki would be called by Brain Brew with that file as the target. On cleanup the temp files would be removed. Similar for export into one of these files, Brain Brew would instruct CrowdAnki to make the json file in a temp repo, and then it would work it's magic to convert to the other location.

Tooling
Users would then need an easy way to generate decks from shared configurations.

Installing Python, setting up Brain Brew, copying a deck configuration file, running the build command, importing in Anki, etc. It's way too much for most users. We need tooling to help, perhaps a desktop app. That's long, long term... smile

If I can implement the above changes, and have Brain Brew native in Anki then such a future is indeed possible! 😁 at that point each deck or offshoot need only provide a working file of any format that Brain Brew can read, and it will combine them all together as stated in the config file 👍 I hope we can get there together!

Random ideas

1. Translate Note Models by replacing the css styling only, and simply replacing the content of specific containers with ids? 🤔
2. Use TiddlyWiki as an interactive source format option? (I am in love with TiddlyWiki now) This would be instead of (or together with) Csv/Yaml.
   • It is free
   • Is just a html file, and can he hosted on Github pages
   • Can be added to or taken from with a REST api
   • Could allow for easy linking of related countries, and the like

---

Summary

Good ideas mate @axelboc I am fully onboard! To infinity and beyond! 😁

@aplaice

Firstly, @ohare93 I'm extremely sorry that I still haven't had time to have a look at and test Brain-Brew!

No worries, mate. I'm taking my time, so feel free to take yours! 😁

At present I am aiming for 8 hours of Brain Brew work a week, during the weekends, as I have a full time job. I hope by my shitty Patreon page you can see I am indeed committed to the project 😁

And though I seriously greatly enjoy writing these fucking biblical length replies, as it greatly clarified my own thoughts on these topics, it does sap a few hours of precious time I could be using to code! 😅 😆 so let's all agree to smaller replies going forward? 😉 on that note, feel free to open any further more specific Issues on this repo, even if they are just questions about a specific topic or functionality. There aren't any users yet, so there'll be no pollution of the issues space! 😁

[axelboc]

let's all agree to smaller replies going forward?

I'll do my best... 🤣

Have you considered using git submodule to add a git repo to a git repo?

I've never used submodules, so I'm not hugely familiar with them. The idea is to get the content of all the child repos in subfolders when you clone the parent repo, is that right? I guess it would simplify things for Brain Brew, but I feel like it wouldn't be very flexible.

Perhaps it's a bit too far-fetched, but I picture more something like a package manager: you specify which repos you're interested in and it downloads them in a temp/untracked folder.

---

Actually, scratch that, the ideal scenario for me would be to no longer need a deck manager in the UG repo at all! I know, crazy 😄 With inspiration from some of your comments, here is what I'm envisioning:

• UG and each extension/translation repo includes a YAML file at the root describing the repo in a way that Brain Brew understands (so it knows what the repo contains and where to get things).
• Brain Brew comes as an Anki addon that allows importing a Francken-deck from a config file (or GUI). The add-on's import config allows specifying which core repo and which extensions to import and how.
• On import, Brain Brew downloads the content of the repos specified in the import config/GUI, does its own thing to generate a CrowdAnki JSON file, and hands over the file to CrowdAnki to perform the actual import (as you've suggested could work).
• As the owner for the UG shared deck, my process no longer involves running a command within the repo itself and importing the generated ZIP file in Anki. Instead, I use the Brain Brew add-on with a very simple config to directly import the core UG repo (i.e. the standard, English deck).
• I then publish the shared deck on Anki Web as usual, and then export the deck as an APKG and attach it to the release notes on GitHub.
• I no longer attach ZIPs for the extended decks and translations to the release notes. It is up to users to import those decks themselves with the Brain Brew add-on (which is much simpler for them too, since they don't have to download/extract/import the ZIP).
• To make it easy on users to import popular alternative decks, we publish pre-made import configurations in the core UG repo and list them in the repo's root YAML file. When a user enters the URL of the repo in Brain Brew, the add-on auto-populates a drop-down menu with the pre-made import configs.
• The add-on keeps a history of the import configs used, so users can upgrade decks easily. (We could even imagine notifying the user when an update is available.)

---

In this hypothetical future, where there are a small Federation of UG deck translations, extensions, and improvement repositories, would you consider creating a Git Organisation for all of the "official" ones to live under?

YES, yes, a hundred times yes! 💯 It's been in my mind for a while now, and this is what actually triggered me to write this issue.

I want the core deck to remain highly stable and "protected", but I don't want this policy to get in people's way! I want UG to become an ecosystem with a life of its own! If the vision I shared above comes to be (at least in part), then a GitHub organisation will be the next logical step, for sure.

The deck as it currently stands could even be broken down into categories like Independent Countries, Territories, Seas/Oceans, etc

Totally, since it wouldn't prevent me from importing all of these "core categories" in order to publish the standard English shared deck.

[ohare93]

I'll do my best...

😁 failure is highly probable, but we can at least try! 😆

Actually, scratch that, the ideal scenario for me would be to no longer need a deck manager in the UG repo at all! I know, crazy With inspiration from some of your comments, here is what I'm envisioning:
...

That is a bold future indeed! 😁 I had not even considered that as a possibility, as Brain Brew is a management tool in my eyes. But if each user has access to it in an easy pre-configurable fashion then individual deck generation as you stated is quite simple 🤯 I think you may be onto something here, and that may make me consider an native Anki add-on sooner than I expected 🤔 "Sooner" != soon though 😅 at least not without help

YES, yes, a hundred times yes! It's been in my mind for a while now, and this is what actually triggered me to write this issue.

I want the core deck to remain highly stable and "protected", but I don't want this policy to get in people's way! I want UG to become an ecosystem with a life of its own! If the vision I shared above comes to be (at least in part), then a GitHub organisation will be the next logical step, for sure.

Noice 👍 So the work you and @aplaice have been doing to trim down the UG deck will hopefully become the new-UG federation's core deck 😁 Perhaps if this is the goal do you still plan on releasing the trimmed deck as the next version? 🤔 though this eventual goal may take a while to realise, perhaps it is worth the wait just to skip all the issues with people desiring different versions? 😅

[axelboc]

Perhaps if this is the goal do you still plan on releasing the trimmed deck as the next version?

Yes, I'm hoping to work on the pre-release this week-end. I'm open to suggestions as to what to do with the removed cards, though. I'll continue the discussion in anki-geo/ultimate-geography#307

[axelboc]

I think you may be onto something here, and that may make me consider an native Anki add-on sooner than I expected

Awesome to hear, by the way. 😊 Looking forward to helping out where I can!

[axelboc]

Just thought I'd clarify that this whole ideal future discussion doesn't change the fact that we should most definitely replace Anki DM with Brain Brew in UG as soon as you think Brain Brew is ready for it. IMO, the crucial short term goal is still to make translations more manageable.

[ohare93]

Just thought I'd clarify that this whole ideal future discussion doesn't change the fact that we should most definitely replace Anki DM with Brain Brew in UG as soon as you think Brain Brew is ready for it. IMO, the crucial short term goal is still to make translations more manageable.

Sounds great, mate 👍 I assume you are still not particularly keen on split csvs though? 😏 if so, I will wait until Yaml is working as a source option 👍 if not then I can probably make the PR next weekend. With the caveat that template changes will need to be made in Anki until I implement card template generation.

Perhaps updating my in-progress PR so that you and others could play with for a while to see how it works and try to find if there's anything that's missing or broken? 🙂 I will try and document things as best as possible, but am also open to a call or live chat at that stage to walk whoever through the generation process, and answer any "what if I wanted to change X" questions 😅

[axelboc]

I think it's worth waiting for YAML, but a proof-of-concept PR with split CSVs still sounds like a good idea, since I'm sure there's plenty to learn and discuss. Looking forward to it!

[Cardosaum]

I was curious about this issue @ohare93 mentioned. and I have to say...
What a great thread!

I really loved the idea of federated decks, it's a game changer.

Although he said it's a little project, I wouldn't say that. 😉