TS compilation perf: faster objectUtil.addQuestionMarks

[jussisaurio]

I'm not sure what the precise reasons for this being faster are, but consistently benchmarking in my project about 50% more type instantiations with the original version vs. the one proposed in this commit; plus the compilation time is 20% longer in the original.

EDIT: check out this issue comment from a tsc maintainer

Before:

Files:                        1251
Lines of Library:            39145
Lines of Definitions:       126004
Lines of TypeScript:         11878
Lines of JavaScript:             0
Lines of JSON:                   0
Lines of Other:                  0
Identifiers:                183217
Symbols:                    285778
Types:                       99275
Instantiations:            7734511
Memory used:               340757K
Assignability cache size:    34242
Identity cache size:          1483
Subtype cache size:            724
Strict subtype cache size:      99
I/O Read time:               0.03s
Parse time:                  0.38s
ResolveModule time:          0.10s
ResolveTypeReference time:   0.01s
ResolveLibrary time:         0.01s
Program time:                0.61s
Bind time:                   0.16s
Check time:                  2.53s
I/O Write time:              0.00s
printTime time:              0.02s
Emit time:                   0.02s
Total time:                  3.32s

After:

Files:                        1251
Lines of Library:            39145
Lines of Definitions:       126008
Lines of TypeScript:         11878
Lines of JavaScript:             0
Lines of JSON:                   0
Lines of Other:                  0
Identifiers:                183224
Symbols:                    286106
Types:                       86715
Instantiations:            5187916
Memory used:               334963K
Assignability cache size:    34946
Identity cache size:          1479
Subtype cache size:            724
Strict subtype cache size:      99
I/O Read time:               0.03s
Parse time:                  0.38s
ResolveModule time:          0.11s
ResolveTypeReference time:   0.01s
ResolveLibrary time:         0.01s
Program time:                0.62s
Bind time:                   0.18s
Check time:                  2.04s
I/O Write time:              0.00s
printTime time:              0.02s
Emit time:                   0.02s
Total time:                  2.86s

[netlify]

✅ Deploy Preview for guileless-rolypoly-866f8a ready!

Name	Link
🔨 Latest commit	50dcc45
🔍 Latest deploy log	https://app.netlify.com/sites/guileless-rolypoly-866f8a/deploys/66243bc507a4ce0008bad9a6
😎 Deploy Preview	https://deploy-preview-2845--guileless-rolypoly-866f8a.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[victorhnogueira]

interesting, how did you do this benchmark? how can I reproduce?

[jussisaurio]

interesting, how did you do this benchmark? how can I reproduce?

I'm basically just benchmarking it against one of the TS APIs at work that uses Zod quite heavily. Would probably be good to have some public zod-kitchen-sink type performance benchmark project though...

I'm having trouble getting any perf difference between these two implementations on just a few ZodObjects, but in my real world project it's quite clear as you can tell from the numbers cited in the OP

[jussisaurio]

interesting, how did you do this benchmark? how can I reproduce?

Script to generate a bunch of zod object schemas and .omit() & .extend() modifications for each:

import fs from "fs";

// Step 1: Possible Zod types
const possibleZodTypes = ["z.string()", "z.number()", "z.boolean()"];
const possibleChainMethods = ["", ".optional()", ".nullable()", ".nullish()"];

// Step 2: Generate a random string for keys and variable names
function generateRandomString(length) {
  const charset = "abcdefghijklmnopqrstuvwxyz";
  let result = "";
  for (let i = 0; i < length; i++) {
    const randomIndex = Math.floor(Math.random() * charset.length);
    result += charset[randomIndex];
  }
  return result;
}

// Step 3: Generate a random Zod schema
function generateRandomZodSchema() {
  const numberOfKeys = Math.floor(Math.random() * 30) + 1; // 1-30 keys

  let schema = `z.object({`;
  const keys = [];
  for (let i = 0; i < numberOfKeys; i++) {
    const key = generateRandomString(8); // Key name of 8 chars
    keys.push(key);
    const randomTypeIndex = Math.floor(Math.random() * possibleZodTypes.length);
    const randomChainMethodIndex = Math.floor(
      Math.random() * possibleChainMethods.length
    );
    const randomType = possibleZodTypes[randomTypeIndex];
    const randomChance = Math.random();
    if (randomChance > 0.98) {
      const { schema: nestedSchema } = generateRandomZodSchema();
      schema += ` ${key}: ${nestedSchema},`;
      continue;
    }
    const type = randomType + possibleChainMethods[randomChainMethodIndex];

    schema += ` ${key}: ${type},`;
  }

  schema += " })";
  return {
    schema,
    keys,
  };
}

// Step 4: Write the generated schemas to a file
function writeToFile(
  filepath = "randomZodSchemas.ts",
  numberOfSchemas = 100,
  numberOfOmits = 10,
  numberOfExtends = 10
) {
  let allSchemas = 'import * as z from "zod";\n\n';
  for (let i = 0; i < numberOfSchemas; i++) {
    const variableName = generateRandomString(7);
    const { schema, keys } = generateRandomZodSchema();
    allSchemas += `export const ${variableName} = ` + schema + ";\n";

    for (let i = 0; i < numberOfOmits; i++) {
      const omitSchemaVariableName = generateRandomString(7);
      const omitKeys = keys
        .slice(0, keys.length - 1)
        .filter(() => Math.random() > 0.5);
      allSchemas += `export const ${omitSchemaVariableName} = ${variableName}.omit({
      ${omitKeys.map((key) => `"${key}": true`).join(",\n")}
    });\n\n`;
    }
    for (let i = 0; i < numberOfExtends; i++) {
      const extendSchemaVariableName = generateRandomString(7);
      const extendKeys = Array(3)
        .fill(0)
        .map(() => generateRandomString(7));
      allSchemas += `export const ${extendSchemaVariableName} = ${variableName}.extend({
      ${extendKeys.map((key) => `"${key}": z.string()`).join(",\n")}
    });\n\n`;
    }
  }

  fs.writeFile(filepath, allSchemas, (err) => {
    if (err) {
      console.error("Error writing file:", err);
    }
  });
}

writeToFile("src/randomZodSchemas.ts");

Generate a file, then import * as randomSchemas from './thatFile.ts' and run npx tsc --noEmit --extendedDiagnostics on it. This reproduces the perf difference quite clearly.

Importantly: a meaningful amount of perf degradation only occurs when using .omit() and .extend(), but when you do, the difference is stark.

[jussisaurio]

Turns out it was even possible to remove the requiredKeys helper with had yet a another small positive impact on performance; github user Andarist landed on this kind of solution earlier this year in #2620 !

Beyond this PR, it might be a good idea to build a separate compilation performance regression test suite for Zod. Something like the type generation script above (albeit cleaned up) might serve as a starting point.

[jussisaurio]

I made a really braindead-simple benchmarking repo, some test runs here:

#2839 (comment)

[psychedelicious]

On zod 3.22.4, this patch breaks a ton of types.

[jussisaurio]

On zod 3.22.4, this patch breaks a ton of types.

Could you provide some example types so I can modify the PR accordingly?

Those examples could be added as type level regression tests.

[psychedelicious]

Could you provide some example types so I can modify the PR accordingly?

Those examples could be added as type level regression tests.

Sure. The issues are in schemas composed of a union of nested, extended schemas. I'm sure there's a more compact schema that can reproduce the issue without needing to bring this ginormo thing in.

[psychedelicious]

I apologize, your PR works fine. I cherry-picked your commits onto upstream master and used that build successfully.

My problem was that I patched the build artefact directly with pnpm patch. That broke not only zod types, but also a ton of non-zod types with nested objects/records. I suppose there is a weird interaction with pnpm patch and TS. Spooky!

[jussisaurio]

Was able to reduce the # of type instantiations some more by baking in flattening to addQuestionMarks; that type was always wrapped with flatten so inlining it was possible

[jussisaurio]

Although apparently it fails on newer TS versions... the project version is 4.5.x

EDIT: reverted.

[colinhacks]

Thanks! I'd added some additional tests in generics.test.ts and it took some fiddling to get this to pass. The zod-ts-perftest repo is so useful! Thanks for putting that together. Got it down to ~3m instantiations from ~6m. 🚀

I also threw a simplified version of extendShape into this PR as well.

Amazing stuff @jussisaurio!!!

[colinhacks]

This has landed in Zod 3.23.

https://github.com/colinhacks/zod/releases/tag/v3.23.0

[colinhacks]

And...it broke JSDoc #3437

Here's where I'm at in my investigations:

// slow (60% more instantiations), but preserves JSDoc
type extendShape2<A extends object, B extends object> = Pick<
  A,
  Exclude<keyof A, keyof B>
> & B;

// fast, but JSDoc is lost
type extendShape1<A extends object, B extends object> = {
  [K in keyof A | keyof B]: K extends keyof B
    ? B[K]
    : K extends keyof A
    ? A[K]
    : never;
};

// fast & preserves JSDoc! doesn't reduce object to simplest form
type extendShape3<A extends object, B extends object> = {
  [K in Exclude<keyof A, keyof B>]: A[K];
} & B;

[jussisaurio]

Iteration 1:

  export type extendShape<A extends object, B extends object> = {
    [K in keyof A]: K extends keyof B ? never : A[K];
  } & {
    [K in keyof B]: B[K];
  };

This seems to preserve the JSDocs at least with the example given in #3437, and also seems to create (slightly) less type instantiations as well.

However, it fails a few existing tests, eg.

test("test inferred merged type", async () => {
  const asdf = z.object({ a: z.string() }).merge(z.object({ a: z.number() }));
  type asdf = z.infer<typeof asdf>;
  util.assertEqual<asdf, { a: number }>(true); // fail
});

Iteration 2:

  export type extendShape<A extends object, B extends object> = {
    [K in keyof A as K extends keyof B ? never : K]: A[K];
  } & {
    [K in keyof B]: B[K];
  };

This passes all existing tests, however increases instantiations by 8.5% (380k vs 350k in my test-case using zod-ts-perftest). This may be a tolerable compromise?