Support Universal Modules in turtle in order to enable customizable builds

[terribleben]

Goals

1. The default exp build behavior should do the same thing it does now, provide all modules. (we might change this later.)
2. For advanced users, there should be a way to customize their build to omit unneeded or unwanted native code. I'm guessing probably a key in app.json.

• Use case 1: Thin builds. The user just wants the smallest ipa/apk possible and does not plan to use more APIs in an over-the-air update, so they know they can omit many of our native modules.
• Use case 2: Omit specific modules. The user just wants to remove one or two specific modules (for example, they don't want the "Uses Ads" flag to show up on their Google Play listing, so they remove just the ads API. How to disable expo build-in ads, including Admob and FacebookAds, since that cause warning installing apk. This issue is very important to all chinese! expo#1789 )

3. We should still avoid running on-demand xcodebuild whenever possible in order to save time and resources.

Prerequisites

• Our SDK needs to be (at least partially) modular. We're shipping the first set of Universal Modules within the Expo runtime in SDK 29. So far, only a small handful of our modules have been converted to universal modules. I don't think we need to wait for 100% universal modules before proceeding with this turtle project.
• To simplify the dependency picture, we can ship standalone apps with single-SDK support. We should do this before we introduce further complexity. Ticket here: https://github.com/expo/universe/issues/2366

Turtle details

Mostly TBD. We probably need to construct a matrix of build caches (instead of having just a single build per release) on the server. The turtle agent can check to see if it has a cached artifact for the user's module configuration, and if not, build one. We may also need to stop running the build our CI machines and do it on the turtle machines instead? There is a (probably old, outdated) description of this here: https://github.com/expo/universe/issues/2032

On iOS, optional module dependencies are expressed with cocoapods. I think on Android they are gradle deps. So the agent will probably need to generate the correct Podfile/gradle file in order to assemble a build.

Lastly, certain parts of the native bundle configuration step will change depending on used modules. As one simple example, we only need to configure NSCameraUsageDescription for builds which use the camera. That means we might need a way for each module to express its own "configure step" in JS.

[dsokal]

I think the time has come. I'll put my focus on adding support for customizable Turtle builds.
I'd like to propose a solution for this problem that I've already discussed with @sjchmiela.

First of all, I need to note that things have changed since creating this issue. We have turtle-cli right now and it enables users to build their standalone apps on their own computers / CI services. This is a huge change because users don't need to depend on our servers stability. Also, they can skip waiting in a queue, because there isn't any queue when building with turtle-cli.

Implementing this feature is more challenging with iOS builds than with Android builds. What's the difference? Turtle builds Android standalone apps from sources and it takes ~12 minutes to complete, whereas on iOS we have a prebuilt (on our CI) shell app and then we are only configuring some stuff, creating an archive and signing the app. An iOS build is less time consuming, the average build time is around 3 minutes, but pre-building (aka compiling the sources) shell app takes ~30 minutes. In order to add support for customizable iOS Turtle builds we'd need to skip the process of pre-building the shell app and it would cause significant increase of the app build time. If a user wanted to choose unimodules to include/exclude from the build he would need to wait at least 30 minutes + time for waiting in the queue. It means the iOS builds queue wouldn't be empty anymore unless we add more VMs (and therefore more physical MacPro machines) and it's expensive in terms of paying more for the infrastructure.

What's my idea then? We have turtle-cli and we can leverage this tool for enabling users to build their customized standalone apps. A user would still have a possibility to use Expo servers for building the app, but the app would have all unimodules in it. This is better for us because we wouldn't need to come up with any complicated infrastructural solution (aka "a matrix of build caches"). What if a user would like to build his customized standalone app on our servers after all? Well, I know that Expo is preparing for taking payments from users, so maybe we can enable customizable iOS Turtle builds in some premium plan. Thanks to that, we wouldn't need so much more resources to handle both regular and customizable builds and waiting in the queue wouldn't be so bad (because there wouldn't be too much long builds). Of course, this doesn't apply to Android builds, because we don't have a pre-building phase there. Users would be able to build customized standalone apps on Expo servers without any charges because it doesn't change anything too much.

What do you guys think about that? @ide @ccheever @jakubste

[tcdavis]

Yaaaaay, that's super exciting!
To clarify, this is just intended to cover optionally excluding / including our own expo-modules, not allowing users to include their own?

Re: Doing this in turtle-cli first, I'm a little hesitant. The main goal of accepting payments is to shut down this line of thinking: (from https://www.reddit.com/r/reactnative/comments/9v94q9/expo_questions/)

\1. How is Expo funded?
This is a key question for us. It's really un-transparent about how Expo is funded.
For example, the expo push notification and CDN stuff is presumably not cheap to run.
The reason this is important is that if I'm building an app based on their infrastructure I don't want to have to re-write it in a few months.

So we need the payment plan to be something people look at and say:

• Sure, I'll pay for that
• Ok, I see how this makes sense as a business

Things like in app purchases and smaller bundle sizes are deal breakers for a lot of people, so this absolutely should make its way to expo-cli, but turtle-cli should be following turtle services, not the other way around.

Needing 10x capacity on mac stadium is pretty rough though. Is the complexity of the matrix solution just in needing a matrix at all, or would it help to reduce the matrix to something like:

• minimal hello world
• base (what we are building currently)
• base + (matrix of modules X you need to detach for X to pass apple review)

I think it would probably make sense to do this the naive, uncached way and have this as a premium only feature for turtle services as well. I don't see it making sense in the free tier without caching though.

[ide]

From a maintenance and quality-of-service perspective, I think investing in Turtle CLI and having our builder service be "Turtle as a service" will help us ensure that Turtle CLI continues to work well. There might be some features that only our service supports or features that only Turtle CLI supports, but the more we can achieve parity and one codebase, the better our tools/services are going to be.

I think it would probably make sense to do this the naive, uncached way and have this as a premium only feature for turtle services as well. I don't see it making sense in the free tier without caching though.

I agree with this.

For 30 minutes of macOS time, CircleCI charges about $1 and I assume MacStadium is similar. Given that builds may take even longer and we have engineering overhead as well, I think structuring our premium pricing plans so that custom builds cost a few dollars each is reasonable.

One thing not mentioned in this conversation is custom native modules. I foresee many issues with them and would greatly prefer to support only Expo-authored unimodules in our build service. This avoids many problems: much higher reliability by working with known code, much less frustration for developers needing to debug native code, no need to upload code from the developer's computer, much less chance of remote code execution (especially bad on Macs since we don't have good isolation between builds), etc. So for this reason, making ExpoKit and Turtle CLI great so people can use custom native modules on their own computers is also really important.

[dsokal]

Well, I guess I might haven't said it precisely. I never wanted to allow users to include their own modules. I agree with you that we should only add support for including / excluding expo-unimodules.

@tcdavis you said that it doesn't make sense to add any feature to Turtle CLI prior to adding it to Turtle. Well, since it's the same codebase and we don't need to add these things separately, it doesn't apply here and we can make the releases independently.

Also, I think that caching shell apps with a certain subset of modules after a user builds an app with this subset is not worth doing because it sometimes happens that we deploy Turtle (and change a shell app) few times in a day. I'd go with doing this the naive way.

So if I understand you guys correctly, we're in agreement that:

• we're doing this the naive way - without caching custom shell apps
• custom builds with expo-cli will be available in "a premium plan"

Did I get it right?

[wkozyra95]

Short description of how will changes in all repos work together

• expo-cli ([expo-cli][xdl] support android custom builds expo-cli#528)
  read package.json and send it with a manifest to the server when publishing
• www (https://github.com/expo/universe/pull/3504)
  extract dependencies field from package.json and add it to manifest
• turtle (android custom builds #47)
  resolve unimodules using dependencies field in manifest and packages directory in expo/expo, the final list of unimodules is passed to xdl build function.
• xdl([expo-cli][xdl] support android custom builds expo-cli#528)
  create directory enabled-modules in shellapp and symlinks to packages for every package from list.
• expo/expo ([android] custom builds expo#4025)
  autoinstall all packages form enabled-modules directory using script from unimodules/react-native-unimodules
• react-native-unimodules (generate android pacakge list unimodules/react-native-unimodules#28)
  • add support for maven dependencies
  • generate java file with list of pacakges BasePackageList.java

[wkozyra95]

released in sdk33