RFC: React Native as a monorepo

[kelset]

This proposal is something that me and @tido64 (and @cortinico to some extent) have been brainstorming for a little while - after a recent problem with react-native-codegen and RN68.

As per title, basically this proposal is not much as in introducing something completely new, but more to discuss a solution to an existing problem. To put simply, the current shape of the react-native codebase hosted in GitHub is lacking structure and consistency and this is leading to a number of issues.

You can read more details in the text itself - here's the file view for easier reading experience.

To see proposal in graph form, expand this.

Feel free to add comments and questions! I'd love to see this turn into something actionable as soon as possible, since it addresses a number of system problems with the repo - it will require some coordination and collaboration, but I'm sure Meta and MSFT can collaborate to make it happen :)

This will also unlock/make easier to revisit other RFCs, such as @empyrical's #49

---

sidenote: this pr also adds a folder to keep images in, assets/, and adds to the .gitignore the pesky macos DS_Store

---

last updated on Sept 2022 to reflect the latest status.

[cortinico]

Thanks for sending this over @kelset
I've did a first pass and left some comments/nits

[cortinico]

I'd add Metro or React as well here

[kelset]

I purposefully avoided adding Reactjs because it seems to suffer from some of the issues I list in the RFC - here's a quick example:

• https://github.com/facebook/react/blob/main/packages/react/package.json -> v18.1.0
• https://github.com/facebook/react/blob/main/packages/react-devtools-inline/package.json -> 4.26.6
• https://github.com/facebook/react/blob/main/packages/react-noop-renderer/package.json -> 16.0.0
• https://github.com/facebook/react/blob/main/packages/jest-mock-scheduler/package.json -> 0.1.0
• https://github.com/facebook/react/blob/main/packages/eslint-plugin-react-hooks/package.json -> 4.5.0

For Metro, as far as I can see they seem to have addressed the naming and semver issues a the way fairly similar to the one we are proposing here (for example, see facebook/metro@fcff382) but they still falls under the category of "opaque repos" like RN (as far as I can see, it follows the same patterns as RN, like for when a PR needs to be merged: facebook/metro#819 ).

That said, I'll add a reference further down in the RFC to the fact that RN could probably learn from how Metro has been setup - only concern there is that it relies on Lerna (see https://github.com/facebook/metro/blob/94e79e0778c8b41224b54c24e48b01e9c53e426a/package.json#L33 ) and Lerna has been officially discontinued (nvm it looks like they are moving stewardship).
(EDIT: added)

[mikehardy]

Just as a supporting comment, my experience supporting the firebase community leads me to lean strongly to "locked versioning" for easy of user support / issue triage as it eliminates much of the versioning confusion - firebase ecosystem has transitioned fully to that style (if you'll allow that a gradle "bill of materials" fits the bill on android - since we use the BoM - though their components do have independent versioning underneath). It really helps a lot, so having "one true version" with automated release policy would be nice from my perspective. (https://invertase.io/blog/react-native-firebase-versioning)

If some packages are higher velocity than others leading to faster versions and worries about release eng time, my counters are "more automation" and "versions are free". I recognize "more automation" isn't free, but within reason it should be possible, if faster revving packages are an issue.

Not sure if there are other counterarguments but may well be, happy to learn if so, especially if they really are insurmountable and mean locked versioning is a non-starter in some cases

[empyrical]

One approach I was thinking about for this initial monorepo restructuring was a codemod script of some sort - so you don't have to keep rebasing a huge pull request keeping up with the main branch, you just need to maintain a script which restructures the repo/applies needed patches until it's decided the changeover is ready.

There's a lot of internal stuff at Meta that would be effected by changing the structure of the RN repo too so this effort will need involvement from someone there. A codemod script approach would probably also be useful because once it's to the point where tests are passing on OSS with the new structure, someone at Meta could modify/add on to the script to change over things internally too.

And hopefully it could be useful for helping people rebase their pull requests to the new structure when the change happens, too.

[cortinico]

One approach I was thinking about for this initial monorepo restructuring was a codemod script of some sort - so you don't have to keep rebasing a huge pull request keeping up with the main branch, you just need to maintain a script which restructures the repo/applies needed patches until it's decided the changeover is ready.

That's a good point. I still think that the major challenge at this stage is letting this setup works with the internal infrastructure. Having one of your draft PR @empyrical imported internally as a first run could help us at least draft a list of the necessary pieces to make this happen.

[cortinico]

• Most of it needs to be tackled in "one fell swoop"

Following up on this RFC as I would like to see how we can actionate it.
Specifically, I think we should try to split it into smaller chunks of work:

1. NPM Package renaming & Version Alignment If possible, we should look into renaming and updating versions of all the packages inside the packages/ folder. Version numbers here are up for debate (either 1000.0.0 like react-native or the upcoming stable).
2. Automatic publishing Once we have all the version aligned, we should extend the CircleCI config to run publishing of all the packages inside the packages/ folder during the release process (alongside the publishing of the react-native package). Ideally the current bump job that gets executed once we cut a branch, should be responsible of all the bumps of all the packages.

If we manage to achieve the first two items here, we will already solve a lot of the versioning problems, plus we would not require manual publishing of first party packages anymore.

Once we're done with it, we can look into:

3. Refactor of the root package Essentially move the root package.json into packages/react-native. This is the item where I expect most of the internal work would be needed.

This last item is also essentially blocked by facebook/metro#1 as we won't be able to run RN-Tester anymore otherwise.

Do we feel the first two items are actionable in isolation? Or do we feel we can split this work further or differently? I just want to make sure this doesn't get stale as it looks too much of a monolithic work.

[kelset]

Do we feel the first two items are actionable in isolation? Or do we feel we can split this work further or differently? I just want to make sure this doesn't get stale as it looks too much of a monolithic work.

I think we could start looking at #2 first, put it in place, then do #1.

And yeah #3 can/should be tackled separately

[cortinico]

I think we could start looking at #2 first, put it in place, then do #1.

Yes the publishing (.2) could be done also first. The only gotchas there is make sure we don't re-publish older versions of packages which are already published (as they will fail to do so). Still is something we can split and do separately.

Adding a couple of feedbacks/thoughts on the overall RFC:

• Before we do anything, it would be great to understand how the monorepo is going to be managed. Are we going to use Lerna or any other monorepo mgmt system? If yes, it would be great to have a bit of evaluation of them before we jump into solution mode.

• An open problem which is still unresolved, and is unique of react-native is storage of Native Artifacts (Android's .aar and Tarballs for iOS) that can't fit inside the NPM package due to size constraints. In general it would be great to find a solution to this problem as well as those artifacts are currently thigthly coupled to NPM packages. I don't want to increase the scope of this RFC, so we should probably move the discussion somewhere else (to another RFC?). Just keep in mind that those artifacts will have to be handled if we move things around.

[tido64]

• Before we do anything, it would be great to understand how the monorepo is going to be managed. Are we going to use Lerna or any other monorepo mgmt system? If yes, it would be great to have a bit of evaluation of them before we jump into solution mode.

If I may, I would like to suggest the same-ish set up as https://github.com/microsoft/rnx-kit. We've used many solutions in the past that we weren't happy with, but we have had zero complaints with Nx for monorepo management and Changesets for versioning so far.

• An open problem which is still unresolved, and is unique of react-native is storage of Native Artifacts (Android's .aar and Tarballs for iOS) that can't fit inside the NPM package due to size constraints. In general it would be great to find a solution to this problem as well as those artifacts are currently thigthly coupled to NPM packages. I don't want to increase the scope of this RFC, so we should probably move the discussion somewhere else (to another RFC?). Just keep in mind that those artifacts will have to be handled if we move things around.

How much storage do we get from GitHub? Could we store binary blobs here and download them on demand?

[cortinico]

How much storage do we get from GitHub? Could we store binary blobs here and download them on demand?

GH Release would be a good solution. The problem is that we would like to have a solution that works also for nightly without having to publish a new Github Release every day as it will create a lot of noise in the repo.

[tido64]

How much storage do we get from GitHub? Could we store binary blobs here and download them on demand?

GH Release would be a good solution. The problem is that we would like to have a solution that works also for nightly without having to publish a new Github Release every day as it will create a lot of noise in the repo.

Are you referring to the number of releases or that latest would point a nightly? I imagine this tick box should at least address the latter:

[cortinico]

Are you referring to the number of releases or that latest would point a nightly?

To the number of releases. I'm afraid we could end up in situation like this:
https://github.com/adoptium/temurin18-binaries/releases

Having hundreds of releases to crawl + generating yet another notification for users subscribed to the repo. Maybe I'm overestimating the impact of this and going with Pre-release for nightlies would also work.

[kelset]

I agree with Nico here, I am not a fan of having too many GH releases - let's maybe consider an alternative, like publishing them as a new npm package separately? ex. @react-native/hermes-artifacts?

[cortinico]

like publishing them as a new npm package separately

The root issue here is sizing. The Hermes tarball is ~400Mb and can't stay in a NPM package. Splitting is 'doable' but would add a lot of custom infra to re-compose the artifacts so really suboptimal

[kelset]

So basically it'd be better to have something like a "bucket", sourceforge style thing where to store them. Right. Need to think about this.

[tido64]

The root issue here is sizing. The Hermes tarball is ~400Mb and can't stay in a NPM package. Splitting is 'doable' but would add a lot of custom infra to re-compose the artifacts so really suboptimal

What does that tarball include? hermes-engine is only 20 MBs.

[cortinico]

So basically it'd be better to have something like a "bucket", sourceforge style thing where to store them. Right. Need to think about this.

Essentially. An S3 bucket would work but anything where we can store asset and access them via URL could work. Worth noting that CircleCI assets might already solve this issue. Artifacts have a default retention of ~30 days, which I believe is acceptable for nightlies: https://circleci.com/docs/artifacts#artifacts-and-self-hosted-runner or we can host them somewhere else.

What does that tarball include? hermes-engine is only 20 MBs.

Debug symbols are making the tarball explode both on Android & iOS. The iOS tarball here is 466Mb: https://github.com/facebook/react-native/releases/tag/v0.69.2
The Android equivalent had to have Prefab libs stripped as they come with Debug symbols by default and would have bumped the NPM package to ~300Mb: https://github.com/facebook/react-native/blob/ccdf9ac9853601a81ff21f8f42f5bd866dd4de75/ReactAndroid/hermes-engine/build.gradle#L52-L54

This is only for Hermes. All the RN libraries are shipped without Debug symbols due to size constrains, making for a far from optimal debugging experience IMHO.

[renchap]

I am not familiar with RN's infrastructure, but I think you can take inspiration from the way Homebrew manages their pre-built binaries (Bottles) using Github's package registry. I am not an expert on it, but I remember reading some threads on the topic and that their solution has been advised and validates by Github

They are using Github Packages to host them. Github Packages are not tied to releases or a repo per default (they belong to a user or org), so they wont pollute the Releases page, but they can optionally be linked to a repo.

The trick is to upload the artifact as an OCI image, a container with only one layer. OCI images are a group of tgz archives and the content does not matter.

They use Skopeo for it, like this:

/usr/bin/skopeo copy --all oci:artifact--version docker://ghcr.io/homebrew/core/artifact:version

From what I understand, the artifact--version directory needs to be compliant with OCI's image layout , but this basically means writing a few JSON files and putting your tarballs at the correct location.

Once uploaded, the artifacts can be downloaded using a stable URL, and discovered using GH package API, without authentication. Github can cache them efficiently as all artifacts have a public and static URL once uploaded.

This is probably a different discussion than this RFC, but I think this would elegantly solve the artefact distribution problem without having to manage external system or credentials. If needed, I can spent some time investigating this further and work with a React Native team member on the implementation.

Some links:

• feature announcement (for external taps): https://brew.sh/2020/11/18/homebrew-tap-with-bottles-uploaded-to-github-releases/
• all the core packages then moved to it: Default download to GitHub Packages Homebrew/brew#11070
• a sample Github action uploading an artifact: https://github.com/Homebrew/homebrew-core/runs/7698895866?check_suite_focus=true
• Github announcement of the container registry, advertising Homebrew's usage for their pre-build packages: https://github.blog/2021-06-21-github-packages-container-registry-generally-available/#dont-just-take-our-word-for-it

[kelset]

thanks @renchap for the great insight! Worth looking into it more :)

---

Just wanted to add a comment here to quote some things that @yungsters mentioned in his PR (facebook/react-native#34401) so that we won't lose track:

• Should I separately configure Flow (and include flow-bin in devDependencies)? ESLint? Jest? TypeScript? Babel?
• How should I verify that @react-native/event-emitter references in react-native work correctly, without having to publish merge a pull request (which might break things), publish @react-native/event-emitter, and then re-test locally? I think that's what Yarn Workspaces is supposed to be for… but what are the right steps to verify?
• My intuition is that npm packages should be compiled to run in as many environments as possible, but for something like this… is that really necessary? It's a nice to have, but React Native provides some nice assumptions (e.g. Metro support for ES Modules and Flow). Should I be manually producing X.js that has Flow removed via Babel, but also include X.js.flow? It would be easier if X.js could just be the same file in the react-native repository source.

We should probably enhance the RFC with a section along the lines of "expectation on packages in the /packages folder" or something with some guidelines for when adding a new one or extracting one from react-native itself. This would also help with scenarios like the one recently mentioned in @necolas' RFC #496 (comment) of extracting Animated to its own package (that was previously attempted by @EvanBacon here facebook/react-native#28892)

[cortinico]

I am not familiar with RN's infrastructure, but I think you can take inspiration from the way Homebrew manages their pre-built binaries (Bottles) using Github's package registry. I am not an expert on it, but I remember reading some threads on the topic and that their solution has been advised and validates by Github

I haven't followed up on this yet, but I'd like to mention that this is a great solution and might actually be the silver bullet for our problems! I've reached out to @renchap and we'll try to come up with a solution around this.

I'd say for the time being, let's trying to keep the conversation here focused on Monorepo (folder structure, versioning, releasing, etc.). We'll be sharing a separate discussion on packaging/artifacts in the future.

[kelset]

Adding an extra note here, we will likely have to do some deep-dives across the react-native repo and figure out if there are rogue packages hidden within it. By complete chance I just found out about this extra package.json living deep within the ReactCommon folder... https://github.com/facebook/react-native/blob/main/ReactCommon/hermes/inspector/tools/msggen/package.json

this type of scenario should not be present anymore when the goal of this RFC is completed (a hidden, non-published, package.json hidden deep within the source code)

[necolas]

This would also help with scenarios like the one recently mentioned in @necolas' RFC #496 (comment) of extracting Animated to its own package (that was previously attempted by @EvanBacon here facebook/react-native#28892)

And it would likely help us avoid Animated becoming more coupled to React Native internals, as has recently happened with the introduction of ReactNativeFeatureFlags into the module and the integration of TurboModule calls in a way that assumes native-only use of Animated. 3rd party offerings like Reanimated 2 avoid these issues and consider web support along the way.

[kelset]

Merged 🎉

Thanks everyone who helped pushed this proposal through the finish line - it's been a massive endeavour and I want to give a special thanks to @cortinico @hoxyq and @tido64 for all their work 👏

And thanks to everyone who commented, suggested, pointed things out!

...and now we couuuuld go back to @empyrical's RFC #49 🤣 😉