Docusaurus 2

[yangshun]

Update 03/19/19

We have started a Roadmap/Feature requests page thanks to the awesome folks at Canny.io. Have a go at it on our WIP v2: https://v2.docusaurus.io/feedback/

---

These are some of the problems I'm seeing in Docusaurus now and also how we can address them in v2. A number of the ideas here were inspired by VuePress and other static site generators.

In the current JavaScript-based static site generators ecosystem, there are the following libraries:

• https://vuepress.vuejs.org/
• https://www.gatsbyjs.org
• https://github.com/nozzle/react-static
• https://github.com/phenomic/phenomic
• https://docsify.js.org/
• https://github.com/lord/slate
• https://middlemanapp.com
• https://github.com/jnordberg/wintersmith
• https://github.com/jaredpalmer/razzle
• https://github.com/pedronauck/docz

Useful libraries

• https://github.com/c8r/x0
• https://github.com/mdx-js/mdx
• https://github.com/jonschlinkert/gray-matter

My proposal for D2 is that Docusaurus be responsible for documentation content, routing, translation and versioning, leaving layout and styling to the end user. Docusaurus would provide a default theme and a few common layouts that the user can use, or ignore if they want to build their layout from scratch. It's not a good idea for Docusaurus to maintain the entire layout and styling as these stuff are hard to make improvements without breaking existing users' code.

A Google Doc where we go into more details can be found here

D2 Design Principles

• Easy to learn but most things are still achievable by users, even if it takes them more code and more time to write. No abstractions beat bad abstractions, and we don't want users to have to hack around the wrong abstractions. Mandatory talk: Minimal API Surface Area
• Users won't feel overwhelmed when looking at the directory of a D2 project. It should look familiar to users familiar building web apps and websites and is easy to build on top of.
• We should not lock-in our users to use our plugins or our CSS. Certain lower-level infra level stuff like React Loadable, React Router are fine, but not higher level ones, such as choice of Markdown engines, CSS frameworks, CSS methodology.
• The separations of concerns between each layer of our stack should be clear - well-abstracted and modular.

D1 Problems and Proposed Solutions

Infra

Current Problems

• Routing logic is separated and resides within both server and generation code, leading to duplicated logic and harder to ensure that code working in server mode also work in generation mode.
• Routing is not declarative. Hard to understand what possible routes there are in Docusaurus.
• Build pipeline is imperative and quite tangled.
• Does not allow adding local vendor CSS and JS (feature request: local project files in siteConfig.scripts & siteConfig.stylesheets  #534)
• New features are mostly enabled via siteConfig - siteConfig will bloat over time and become unmaintainble.
• Current architecture is a bit tangled and makes it hard to introduce plugins.
• Many components relies on process.cwd() too much (mostly for siteConfig), making it very hard to test.
• LiveReload does a full-page reload, and page reloads for changes of all files.

Suggested improvements

• Use a module bundler like webpack (or Metro if possible, since it's also by Facebook).
• Static pages can be generated using (https://github.com/markdalgleish/static-site-generator-webpack-plugin)
• Assets should be fingerprinted (should be trivial with webpack)
• Use webpack-dev-server during development which enables hot reloading - only reload the modules which changed.
• Rearchitect architecture to introduce hooks into the development and build phase so that a plugin system is possible.
• Keys casing are not consistent - camelCase in siteConfig and blog front matter but snake_case in docs front matter.

HTML/CSS

Current Problems

• Raw CSS without any preprocessor. Hacky variable substitution approach using regexp.
• Docusaurus controls much of the styles and layout, which makes it prone to breaking end users code who customize.
• All CSS is compiled into one file.
• HTML/CSS is self-written and doesn't follow CSS framework best practices. Typography is not great (margins and paddings are not consistent)/. Partially fixed in CSS revamp - Improve typography and overall styling #757.
• Hard to do theming on top of current CSS structure. Limited to changing of colors and fonts.
• Site layout is not customizable - EDIT: Just found out it is (Add the ability to provide custom layout #128).
• Docs/blog layout is not customizable.
• No 404 page.

Suggested improvements

• Layout and styling should be controlled by theme and user. Docusaurus should just provide the data (content, translation, versioning), props and a default theme. This is the approach taken by many static site generators as well, such as Jekyll, Gatsby and VuePress. VuePress allows ejecting of the default theme for further layout customization.
• Follow CSS frameworks best practices - Use rem for units, allow specifying margin width, etc.
• Allow user to select their own CSS preprocesser.
• Allow setting styles and scripts for specific pages only. Use case - some sites like Babel, Prettier and Reason have playgrounds where they only want to load specific scripts and styles.
• Make Docusaurus pages/components more customizable (Make Docusaurus Components more extensible #249)
• Add a 404 page.

JavaScript

Current Problems

• JS code in pages is currently not being transpiled by Babel. We have to write ES5 in code that will be sent to browsers.
• React is used as a templating engine for static content; we're not using any of its view library features. Interactivity has to be added through script tags and dangerouslySetInnerHTML.

Possible improvements

• If the user adds interactivity to their React component/page, we should include React library on the page and hydrate the component after the page has been rendered.

API

Current Problems

• CLI commands should be using docusaurus <command> (instead of docusaurus-command) which is more conventional (see Yarn and Git).
• Docusaurus-provided components are required using a relative path which looks a bit odd and may break if we change the library's internal directory structure.

Possible improvements

• Rework the CLI commands. Should be just a matter of renaming and shifting them into docusaurus core. This also solves the problem of having to publish docusaurus-init as a separate package.
• Export docusaurus components on the main npm file itself. Add even more common components (we can consider creating pages as components as well).

Blog

Current Problems

• NA

Possible improvements

• Allow for commenting support (via a plugin and not siteConfig as how it is currently done)
• Allow customizing of date.
• Draft mode for posts
• Tags, Categories

Markdown

Current Problems

• Only reads YAML, does not support different kinds of front matter.
• Unable to embed content from other Markdown files.

Possible improvements

• Add reference style linking.
• Add page tabbing (Add page tabbing infrastructure #45), probably via a Remarkable plugin.
• Allow importing markdowns in other markdowns.
• Line highlighting within code blocks.
• Support embedding React components in markdown.
• Emoji support.
• Check that internal page links and anchors work fine.
• Support special block tags like in VuePress for tip/warning/danger markup.

Search

Current Problems

• We're using Algolia as the only form of search now. Algolia search is a third-party service that requires internet and doesn't work behind VPN/offline.

Possible improvements

• I really like Algolia search as it's useful and so simple to use. But we could also come up with alternative offline search mechanisms to get around the limitations mentioned above. At my previous company, I added offline search for a static site via Lunr.js. It works by looking through the markdown files and generated a static index of the content during build time. The searching and rendering of search results is purely done on the client side. It roughly works - https://engineering.grab.com/search.html?q=react

These are my two cents. Feel free to chime in (:

[ahmadalfy]

+100 for moving the layout and styling outside Docusaurus' scope.

[JoelMarcey]

Thanks so much for staring the planning process here @yangshun. These are wonderful.

I will say a request for a local/offline search option should be considered for v2 as well -- we have had that request multiple times as well.

Also, I feel it is important to reiterate this -- no matter what features we add for v2, we always want to make sure we adhere to the overall mantra of Docusaurus which is "easy to create a documentation website", with the emphasis on "easy". It is always a tricky balance to add a bunch of features and keep things as simple as possible.

[endiliey]

This is exciting. webpack 🎉.

Extra things that I think will be great to have

• Easy way to support redirect urls (301 Redirect, Not 404)

/blog/my-old-title   /blog/an-even-better-title
/docs/en/android-setup /docs/en/getting-started

• Easy way to import from other documentation sites generator (e.g: gitbook)
• User can download documents (maybe in pdf) 😄

Questions

• Are we going only accept bug fixes for Docusaurus v1 ? Any PRs opened against v1 that are not bug fixes will be closed and features will only be accepted for v2
• Given that this will be a lot of work. When ? 😃

[yangshun]

I will say a request for a local/offline search option should be considered for v2 as well -- we have had that request multiple times as well.

Added a section about search.

Also, I feel it is important to reiterate this -- no matter what features we add for v2, we always want to make sure we adhere to the overall mantra of Docusaurus which is "easy to create a documentation website", with the emphasis on "easy". It is always a tricky balance to add a bunch of features and keep things as simple as possible.

Yes! No worries we'll stick to this. It might just be troublesome for v1 users to migrate to v2 due to some potentially backward-incompatible changes we're making.

[JoelMarcey]

Whether we do this for 1.3+ or v2, we should also considering standardizing on one or both of the following:

• yarn or npm in the documentation. Not both.
• Depending on whether we do the above, remove the other's .lock file. Decide if we want .lock files at all.

[caabernathy]

Let's also think about integrating autogen systems into an upcoming version - see #78

[JoelMarcey]

Adding Docusaurus format support to hh-apidoc is not a Docusaurus v2 feature necessarily, but we can consider as something to help grow the use of it.

hhvm/hh-apidoc#29

[fredemmott]

Ultimately I think it should be hh-apidoc's responsibility to provide documentation/metadata in a good format for docusuarus when wanted - but for user experience, it would be good if docusaurus were aware of it and would suggest/configure running it as part of the docusaurus build process rather than needing a separate predecessor step

[prakhar-goel]

+1 for offline search using lunrjs or similar. We are using Docusaurus for internal company documentation and therefore cannot use default Algolia search. Ability to search through markdown files would be a great add on to the current version.

[kevinbarabash]

It would be nice if #852 could be addressed as part of v2.

[chenglou]

I really hope that Docusaurus can keep its initial simplicity; v2 sounds nice but there are several things I'm slightly worried about. By simplicity, I mean both the interface and implementation (I'll explain why I care about the latter, aside from being an occasional contributor/debugger myself, at the end). Docusaurus' rather solid, that and the crowdin support are the two reason why we switched away from other ultra ambitious generators with a complex webpack pipeline that failed to handle three digits number of markdown files...

Specifically:

Routing logic is separated and resides within both server and generation code, leading to duplicated logic and harder to ensure that code working in server mode also work in generation mode.

Yeah this is definitely a pain point we felt on the user side! I've noticed/filed/fixed a few of these.

Routing is not declarative. Hard to understand what possible routes there are in Docusaurus.

Not too sure what this means; isn't it glanceable in sidebars.json?

Current architecture is a bit tangled and makes it hard to introduce plugins.

Making it pluggable seems to always come with a tradeoff (relevant). I'm imagining that when I read the supposedly new siteConfig, I might see a list of clean but super opaque require calls instead of a js object config? This is what happened with other static generators. The complexity is hidden by all the third-party plugin requires but there's always some that behave weirdly and try to over-compose, like webpack configs.

LiveReload does a full-page reload, and page reloads for changes of all files

It's been quite fast from my experience; it's less of our business from the user side, but I'd love to see the Docusaurus codebase staying simple. It would be nice if siteConfig.js's changes are picked up though!

Use a module bundler like webpack

We switched to Docusaurus to avoid some of the more hardcore webpack and ecosystem problems. At the worst time, I was cherry-picking and deploying things manually each push. cc @rickyvetter who'd concur.

Actually, the watcher mode and the startup time has regressed quite a bit these days, but I didn't have the heart to point it out because Docusaurus has been serving us well 😅. It'd be nice to have it fast again... in fact, that's probably one of my biggest request. And I don't mean overengineer the pipeline to get a fast lazy startup; I mean the "good old days" where the startup was simple enough to stay within 1-2s. Webpack will blow past that budget.

Raw CSS without any preprocessor. Hacky variable substitution approach using regexp.

Most other CSS preprocessors aren't much of a step up from basically regexes lol. Given CSS' syntax, a single regex pass isn't actually that hacky; you might replace some words in comments but that's pretty much it. The one from Docusaurus has been blazing fast so far, so that's nice. But given that a preprocessor is opt-in, and given its limited scope, I think letting userland customize it sounds fine. Aka we don't pay the price for not using them, aside from maybe one or two checks in Docusaurus codebase.

Docusaurus controls much of the styles and layout, which makes it prone to breaking end users code who customize.

We're such folks, for all 3 sites. But I'd rather keeping up with the very occasional small layout breaking changes (very quick to check the css changes from changelog and fix them) than having them overly customizable (the point being that such customizability will probably affect Docusaurus' current setups).

All CSS is compiled into one file.

That has been very fine so far! Kinda like how non-nonsense it is.

HTML/CSS is self-written and doesn't follow CSS framework best practices. Typography is not great (margins and paddings are not consistent)

Yeah fixing those would be great!

No 404 page.

Oh yeah, a custom URL redirect json config would be sweet. We made our own for our sites by essentially putting dummy placeholder redirect pages.

Anyway, the gist is that, we really appreciated Docusaurus' non-nonsense toolchain, and whatever depth-related improvements (like even better page layout) are super welcome. Reaching too much for breath and becoming essentially another "website compiler" seems a bit much; after all, we do want to just focus on the documentation, and no matter how hard folks try to compiler-fy websites to regain the ease of use, simplicity and performance, we've not seen such effort that hasn't leaked hard into userland so far. Won't point fingers though. Also, sorry if some of the spirit I've summarized here wasn't part of Docusaurus' initial goal; we might happen to have piggy backed on top of it.

Thanks a lot for your continued work 😃

[muuvmuuv]

+1 for Gatsby.js. This would make Docusaurus so flexible and modular

[sunnylqm]

Can we have a roadmap for v2 like this? react-navigation/react-navigation#4434 @yangshun

[JoelMarcey]

@chenglou Thank you so much for your thoughtful feedback. I am in 100% agreement with your overall premise that Docusaurus needs to be true to its original charter of "easy and simple" for open source websites. I believe the v2 leads, @endiliey and @yangshun, understand that and believe that too.

The v2 folder has been added to the repo so we can keep tabs on the progress of the implementation.

[yangshun]

A more detailed v2 document can be found here. Comments are welcome!

Would like to give interested parties a heads up that v2's progress is delayed due to manpower constraints on our end (no one full-time working on the project). We hope to have a beta ready by mid 2019, dogfood on Facebook's Docusaurus sites, and launch in Q4 2019.

[JoelMarcey]

That said, if anyone is interested in joining our effort as a direct contributor to v2 (or maintainer of v1), let us know. We would love the help.

[ymschaap]

@yangshun @JoelMarcey I've been following the project for a while now. Have 10+ year experience in front-end development, lots of React+webpack work and would be able to commit time to join the effort on getting 2.0 launched.

[JoelMarcey]

@yangshun, @endiliey -- what is the best way for @ymschaap and @HenryStevens (who I chatted with offline) to get started helping with v2?

[endiliey]

This might need more discussion. But one of the useful thing that can be done for those who wanted to help with v2 is to actually test & play around with the on-going v2 implementation

There is an instruction on how to run it locally.

Edit: Alternatively, join us on discord

[afrankel-sfdo]

didn't see this on the list: per language assets. useful when images contain information that needs to be translated.

[emanic]

Perhaps I can explain more specifically what I am looking to do and you can see if this is something you may want to add to v2, but more importantly I would appreciate some guidance on how I might accomplish it with v1. We have versioned docs and need to display different content to users based on which version of the docs they are looking at. We want to be able to identify what should be replaced in the markdown using something like this {{variable}}. In the immediate term, we need this for download paths inside of code blocks so that they can copy and paste code that gets them the correct client for their version. Longer term, we want to show different chunks of texts and even different navigation based on the version. I saw the discussion here #395, unfortunately, this does not satisfy my use case. I understand that some of this would be in a remarkable plugin (though none exist at present that we can use), but also, we need to be able to set the strings, navigation, content chunks via React. Any tips would be appreciated.

[CGick]

I am migrating my companies product documentation to Docusaurus, and a lot of our pages are referenced by other pages. It would be nice to have a reference mapping file in which I can add all of the link ids and the link address they map to, then in my docs pages just enter the link id and have it pull the link address from the mapping file. If this is currently possible could someone please let me know how this is done.

[joetidee]

Is there a rough timeline for the beta and full releases of v2?

[Berkmann18]

Is there any estimated release time or something?

[iamandrewluca]

Proposal: Evergreen documentation

I don't think this should be implemented into Docusaurus, but to keep in mind.

From times to times it happens that docs remain behind code.
Would be great to have a validator that will test if docs reflect latest code changes. I think it could be used as a prebuild step.

This package allows to validate documentation files
https://github.com/Originate/text-runner

Are there other alternatives?

[endiliey]

Is there a rough timeline for the beta and full releases of v2?
Is there any estimated release time or something?

There is no rough estimated timeline, but I guess we're pretty close to beta at least.

If you want to see it shipped faster, try it today https://v2.docusaurus.io/ and give feedback.
You can also contribute by sending pull request, sponsoring us on Open Collective and/or chat with us on Discord. We need help 😉

We don't want to release unready version, that's why we opt-ed to stay in alpha as much as we can to ensure its stable and we "do it right". Actualy, many are using alpha versions for production. It has matured, docsearch is using it, CRA is using it and actually we're planning to migrate Redux's site soon to use the alpha version. Breaking changes has been very easy so far, as long as you know basic React and JavaScript.

Some people are adventurous enough as well with the customization. Example:

https://vector.dev/

https://uniforms.tools/

[Berkmann18]

What's the timeframe for v2 coming out?

[JoelMarcey]

@Berkmann18 Hi. As you may or may not know, a generally stable alpha of v2 is out now - https://v2.docusaurus.io. Sites, including those from Facebook Open Source, are using it. The reason it is not "fully released" yet is that it is missing a key piece of functionality that is currently in v1 -- localization support. That is being worked on now. If you don't need that, then you might find that v2 already serves your needs.

To answer your question though - v2 replacing v1 (i.e., v1 is deprecated) hopefully sometime in the next couple of months.

cc @yangshun @slorber

[Berkmann18]

@JoelMarcey In that case, I guess I'll wait then as the project where we use Docusaurus relies on translations provided via Crowdin.

[honi]

Is there any documentation regarding Infima theming variables or an ETA for Infima source code release?

[slorber]

@honi apart the website and inspecting, source is not yet available. FB policy is not really to publish projects that are not ready. But Infima is probably not very far. We'll see how to make this available soon.

[uditalias]

I created this simple CSS gist that shows how you can add Line Numbers to code blocks.

https://gist.github.com/uditalias/3969a20ef759bf3a8757499429b30063

You can see it in action here: https://www.injex.dev/docs/api/core/decorators/inject

[thibaudcolas]

Following from #789 (comment), are there good issues to subscribe to for people who are waiting on localization support? I see #3524 for the Jest website, which has big localization requirements I assume. Are there any others?

I’m considering Docusaurus for a big documentation site, localization support is a hard requirement, it’s unclear how far ahead Docusaurus v2 is with this in particular.

[slorber]

@thibaudcolas sorry for not communicating much about progress.

Let's track progress in the i18n RFC issue where I'll mention important updates about i18n.

I've just added a comment to answer your questions:
#3317 (comment)

Let me know in this issue if our i18n system is good enough for you.

[slorber]

The core features of Docusaurus 2 have been implemented and we moved to beta recently, so let's close this issue