Tracking: Turing Way

[rowanc1]

We had a wonderful conversation this morning with the TuringWay team on how best to support the TuringWay team in their adoption of JupyterBook v2, which will be using mystmd. There were a lot of exciting possibilities that we talked about in terms of creating an ecosystem of content that the TuringWay could expose (figure, equations, glossaries), and that be included, linked to and cross-referenced by other books, tutorials, and communities in the wider open-science ecosystem. There are also some current deficiencies in MyST MD at the moment, one of the core ones is around internationalization. We got a really good demo from @BatoolMM on how that process is working, and some of the sticking points on how to bring content together to have a unified experience between languages. The localization also must include right-to-left languages, which will require a number of style updates and improvements across the components.

Resources for translations:

• https://turingway-arabic.netlify.app/welcome.html
• https://github.com/TWTranslation/the-turing-way/tree/main/book/translation

Example of Turing Way content deployed in MyST

• https://rowanc1-turingway.curve.space/ (this is deployed using Curvenote)

Some cool things

Updates that need to happen

• Glossary Compatibility with RST Glossaries #618
• Improve the upgrade process 🆙 Upgrade Jupyter Books in myst init #1223 (in progress)
• Internationalization and localization (this is a big task that we need to break down, what is necessary is a lot clearer to me notw after @BatooIMM's demo) Add support for translations / internationalization / localization of page elements #166
• Host multiple languages in a single theme
• PR: fix admonitions with Note --> note (that can be upstreamed right away)
• Way to turn off @username references on a page (e.g. changelogs) or have those fallback to a configured list of IDs and improve, e.g. our GitHub person cards.
• Discuss and demo different ways to build out the infrastructure for the book (and tradeoffs between the different approaches)
• Idea (curious on your take): Add source attribution information to images/figures #1305
• Improved header / footer information

There are probably a number of other things that will come out as we work towards these goals, I encourage members of the TuringWay team to jump in here and let us know how we can support you. Please try the preview above and get a sense of where we are at. One of the cool wins is that the performance of navigating between pages should be improved quite a bit. There are also some abbreviations that will work and increase accessibility etc. for various pages.

I know that my team and @agoose77 do have some time to help with specific things, and some issues are in progress this week, so hopefully when we meet again in a few weeks there will be some progress to show!

@JimMadge @da5nsy @BatoolMM @agoose77 @sgibson91

We have had some previous conversations with the Environmental DataScience Book as well:

https://github.com/alan-turing-institute/environmental-ds-book

@acocac

[da5nsy]

Amazing, thank you so much for getting the ball rolling on this @rowanc1. Really excited to see where we can go with this. And happy to assist where we can!

Linking to the issue over on TTW where we've been tracking our own attempts/progress on enabling multi-language support: the-turing-way/the-turing-way#3255

[agoose77]

@rowanc1 was this meeting recorded in any fashion? I had some pre-conceived ideas around what TTW is doing regarding Crowdin, etc, and I know a bit about the space of tools. My read is that there are many ways to do this, so any guidance on the solution that TTW is converging on would be useful context.

[da5nsy]

Hi @agoose77!
Not recorded I'm afraid, but I'm happy to answer (or redirect) any questions you have.

@BatoolMM is the person to ask about translation, or have a look here at our existing documentation.

And then from an infrastructure perspective this is the latest. In summary, there's currently no obvious out-of-the-box tools that we could use to deploy a multi-language site. We might be able to get something working with RTD, but honestly we would prefer an internal process that didn't rely on an external build service (so that we would be able to host anywhere).
(@JimMadge, @sgibson91, @bsipocz - correct me if I've misunderstood!)

[rowanc1]

@fwkoch just made a change that takes turing way from >1min to build to 17 seconds. 🚀

[JimMadge]

Just leaving some of my thoughts/opinions,

Hosting isn't the same as the build. However, hosting that we have considered/used (Netlify, RTD) do place restrictions on the build.
Ideally, I would prefer that we control the build and just copy the output somewhere to host.

Multilingual support seems surprising difficult given how many languages are commonly used.
A lot of the solutions we have looked at seem to have inherited the idea of translations from software, where you have either

• A set of messages with .po files for each language
• A primary language with translations for each other language

Neither of these feel like they suit us. We don't want to enforce the idea of a primary or default language. We don't really see the task as reproducing the same information in each language.

Maybe we want something more like Wikipedia, where different languages are fairly independent but if there are pages on the same topic in two languages, you can switch between them.

This really isn't my area so I have no idea how difficult or possible that is 😆

[da5nsy]

Hi folks!
Just adding a note to say that I tried following the instructions at https://next.jupyterbook.org/upgrade to test upgrading TTW and totally failed.
the-turing-way/the-turing-way#3958
Might be a Windows thing, or just me goofing something up, but I don't know.

[agoose77]

@da5nsy thank you for trying out Jupyter Book and reporting back! I'll take a look at this within the next day.

[da5nsy]

We're going to work through the error logs from the-turing-way/the-turing-way#4019 tomorrow at a collab cafe. If any folks from mystmd would like to join they'd be super welcome!

[acocac]

I've experimented JB2 for EDS book since last month - Happy to join the collab session to have a look at the error logs too!

[rowanc1]

It would be awesome to have a some of the exploration of this documented here so that we can improve things! I think that @agoose77 will be there, however I can't make it. Good luck!! 🚀

[agoose77]

We had a look at some error logs. There were a lot of hard-failure errors in @sgibson91's local build that look suspiciously like resource-exhaustion errors during the HTML build specifically.

We also found a lot of errors pertaining to genuinely missing links.

I observed:

1. We should perhaps support caching link resolution (i.e. "it resolved") for a number of days
2. We should enable cache persistence of myst builds with the GHA generated by myst init.
3. Our DOI heuristic might be too aggressive: Examples of discrimination in AI include [racism in predictive policing](https://www.technologyreview.com/2020/07/17/1005396/predictive-policing-algorithms-racist-dismantled-machine-learning-bias-criminal-justice/) is treated as a DOI with 1005396
4. Our wiki heuristic doesn't understand permalinks to wiki content: https://en.wikipedia.org/w/index.php?title=Tidy_data&oldid=962241815 fails
5. We end up with a lot of errors RE Waiting for response from http://localhost:3000/remote-collab-guidelines.json that ultimately crash
6. We also see ECONNRESET errors
7. Perhaps we want to be able to ignore errors in a particular file e.g. automatic link fragment generation
8. Do we support {cite}`Corporate Governance and Ethics<Dessain2011governance>`

[rowanc1]

DOI fixed in continuous-foundation/doi-utils#15

The ECONNRESET is likely #1775 if you are building html. Especially for turing way we hit the web server with all requests at once; this should be looped in a promise limit. That should fix 5/6.

[rowanc1]

This should fix 5/6:

• 🔄 Fetch with Retry for HTML build #1793