Skip to content

Commit

Permalink
Remote runtime controller (#5939)
Browse files Browse the repository at this point in the history
* Squashed commit of the following:

commit 7d0f2d6
Author: Somhairle MacLeòid <smacleod@cloudflare.com>
Date:   Thu May 30 13:35:53 2024 +0100

    Update index.ts

commit 4fac979
Author: Somhairle MacLeòid <smacleod@cloudflare.com>
Date:   Thu May 30 13:10:39 2024 +0100

    Create flat-days-punch.md

commit 2c7bc56
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Thu May 30 12:52:32 2024 +0100

    Remove cirular dependency between logger and env var factory which Vitest/Vite can't understand

commit e4f2efb
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Thu May 30 02:48:47 2024 +0100

    lint

commit c678631
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Thu May 30 02:35:29 2024 +0100

    restore logging & enable tests

commit 4a87a51
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Thu May 30 01:53:33 2024 +0100

    bindings passthrough + persistence path

commit f1ff546
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Thu May 30 01:25:26 2024 +0100

    handle empty routes array in getInferredHost

commit cc1c2f0
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Thu May 30 01:20:08 2024 +0100

    ignore https options in LocalRuntimeController (handled in Proxy

commit 87b5a98
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Thu May 30 00:56:21 2024 +0100

    starting to get tests passing

commit 6b6204d
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Wed May 29 23:06:26 2024 +0100

    Separate preview sessions and tokens

commit 6912a49
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Wed May 29 21:33:17 2024 +0100

    fix build + lint + types

commit c9d70b6
Merge: db6eada 8033afb
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Wed May 29 21:14:51 2024 +0100

    Merge branch 'startDevWorker-milestone-2-local' into remote-runtime-controller

commit db6eada
Merge: 10a9495 90d6be7
Author: Somhairle MacLeòid <smacleod@cloudflare.com>
Date:   Wed May 29 21:09:43 2024 +0100

    Merge branch 'startDevWorker-milestone-2-local' into remote-runtime-controller

commit 10a9495
Merge: 0842071 be715bd
Author: Samuel Macleod <smacleod@cloudflare.com>
Date:   Wed May 29 20:57:04 2024 +0100

    Merge branch 'startDevWorker-milestone-2-local' into remote-runtime-controller

commit 0842071
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Tue May 21 16:16:48 2024 +0100

    chore: pull in changes from LocalRuntimeController PR

commit a2a6ba1
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Tue May 21 15:15:12 2024 +0100

    update startDevWorker types
    from LocalRuntimeController PR

commit 6e38a28
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Sun May 19 21:25:28 2024 +0100

    fix: make unstable_dev respect experimentalDevenvRuntime

commit 82ce0a1
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Sun May 19 21:50:37 2024 +0100

    fix: unstable_dev with node v18
    by defaulting .ip to 127.0.0.1

    + enable unstable_dev (previously: wrangler-dev-api-app) fixture tests to run in ci

commit 341d212
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Sun May 19 20:32:48 2024 +0100

    implement --experimental-devenv-runtime
    for conditional use of DevEnv-{Local|Remote}RuntimeController

commit db353b2
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Sun May 19 20:08:12 2024 +0100

    chore: refactor bundle ref back to normal variable

commit dd55a24
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Sun May 19 19:58:57 2024 +0100

    fix: auth + accountId selection from React flow with RemoteRuntimeController

commit c0c88f2
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Sun May 19 19:39:26 2024 +0100

    use an actual onBundleComplete callback
    instead of relying on useEffect + deps tracking

commit 2f8b98c
Author: Rahul Sethi <5822355+RamIdeas@users.noreply.github.com>
Date:   Fri May 17 16:57:20 2024 +0100

    wip: RemoteRuntimeController + integration

* Refactor errors to use shared definition

* Rename flag

* Remove extraneous console logs

* Fix spacing

* Actually teardown

* Address comments

* Fix flags

* Remove ip overriding

* Hardcode IP in unstable_dev

* Cleanup unstable options

* Error on invalid text_blobs

* Teardown on change from remote -> local and vice-versa

* add volta

* e2e test for switching runtime race conditions

* Update snapshots

* immutable config

* more testing

* file url
  • Loading branch information
penalosa authored Jun 5, 2024
1 parent cfde058 commit 21573f4
Show file tree
Hide file tree
Showing 27 changed files with 1,062 additions and 392 deletions.
5 changes: 5 additions & 0 deletions .changeset/flat-days-punch.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"wrangler": patch
---

feat: Adds the experimental flag `--x-dev-env` which opts in to using an experimental code path for `wrangler dev` and `wrangler dev --remote`. There should be no observable behaviour changes when this flag is enabled.
8 changes: 4 additions & 4 deletions fixtures/dev-env/tests/index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -181,8 +181,8 @@ function fakeReloadComplete(
pathname: `/core:user:${config.name}`,
},
userWorkerInnerUrlOverrides: {
protocol: config?.dev?.urlOverrides?.secure ? "https:" : "http:",
hostname: config?.dev?.urlOverrides?.hostname,
protocol: config?.dev?.origin?.secure ? "https:" : "http:",
hostname: config?.dev?.origin?.hostname,
},
headers: {},
liveReload: config.dev?.liveReload,
Expand Down Expand Up @@ -569,7 +569,7 @@ describe("startDevWorker: ProxyController", () => {
`,
config: {
dev: {
urlOverrides: {
origin: {
hostname: "www.google.com",
},
},
Expand All @@ -585,7 +585,7 @@ describe("startDevWorker: ProxyController", () => {
...run.config,
dev: {
...run.config.dev,
urlOverrides: {
origin: {
secure: true,
hostname: "mybank.co.uk",
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,11 @@
"license": "ISC",
"author": "",
"scripts": {
"dev": "node --no-warnings ./src/wrangler-dev.mjs"
"dev": "node --no-warnings ./src/wrangler-dev.mjs",
"test:ci": "npm run dev"
},
"dependencies": {
"wrangler": "workspace:*"
},
"volta": {
"extends": "../../package.json"
Expand Down
File renamed without changes.
File renamed without changes.
8 changes: 8 additions & 0 deletions packages/wrangler/e2e/__snapshots__/dev.test.ts.snap
Original file line number Diff line number Diff line change
@@ -1,9 +1,17 @@
// Vitest Snapshot v1, https://vitest.dev/guide/snapshot.html

exports[`basic js dev: 'wrangler dev --remote --x-dev-env' > can modify worker during wrangler dev --remote --x-dev-env 1`] = `"Hello World!"`;

exports[`basic js dev: 'wrangler dev --remote --x-dev-env' > can modify worker during wrangler dev --remote --x-dev-env 2`] = `"Updated Worker! value"`;

exports[`basic js dev: 'wrangler dev --remote' > can modify worker during wrangler dev --remote 1`] = `"Hello World!"`;

exports[`basic js dev: 'wrangler dev --remote' > can modify worker during wrangler dev --remote 2`] = `"Updated Worker! value"`;

exports[`basic js dev: 'wrangler dev --x-dev-env' > can modify worker during wrangler dev --x-dev-env 1`] = `"Hello World!"`;

exports[`basic js dev: 'wrangler dev --x-dev-env' > can modify worker during wrangler dev --x-dev-env 2`] = `"Updated Worker! value"`;

exports[`basic js dev: 'wrangler dev' > can modify worker during wrangler dev 1`] = `"Hello World!"`;

exports[`basic js dev: 'wrangler dev' > can modify worker during wrangler dev 2`] = `"Updated Worker! value"`;
126 changes: 126 additions & 0 deletions packages/wrangler/e2e/dev-env.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
import shellac from "shellac";
import dedent from "ts-dedent";
import { beforeEach, describe, expect, it } from "vitest";
import { CLOUDFLARE_ACCOUNT_ID } from "./helpers/account-id";
import { makeRoot, seed } from "./helpers/setup";
import { WRANGLER_IMPORT } from "./helpers/wrangler";

// TODO(DEVX-1262): re-enable when we have set an API token with the proper AI permissions
describe("switching runtimes", () => {
let run: typeof shellac;
beforeEach(async () => {
const root = await makeRoot();

run = shellac.in(root).env(process.env);
await seed(root, {
"wrangler.toml": dedent`
name = "dev-env-app"
account_id = "${CLOUDFLARE_ACCOUNT_ID}"
compatibility_date = "2023-01-01"
`,
"index.mjs": dedent/*javascript*/ `
const firstRemote = process.argv[2] === "remote"
import { unstable_DevEnv as DevEnv } from "${WRANGLER_IMPORT}";
const devEnv = new DevEnv()
let config = {
name: "worker",
script: "",
compatibilityFlags: ["nodejs_compat"],
compatibilityDate: "2023-10-01",
dev: {
remote: firstRemote,
auth: {
accountId: process.env.CLOUDFLARE_ACCOUNT_ID,
apiToken: process.env.CLOUDFLARE_API_TOKEN
}
}
};
let bundle = {
type: "esm",
modules: [],
id: 0,
path: "/virtual/esm/index.mjs",
entrypointSource: "export default { fetch() { return new Response('Hello World " + (firstRemote ? 'local' : 'remote') + " runtime') } }",
entry: {
file: "esm/index.mjs",
directory: "/virtual/",
format: "modules",
moduleRoot: "/virtual",
name: undefined,
},
dependencies: {},
sourceMapPath: undefined,
sourceMapMetadata: undefined,
};
devEnv.proxy.onConfigUpdate({
type: "configUpdate",
config,
});
devEnv.runtimes.forEach((runtime) =>
runtime.onBundleStart({
type: "bundleStart",
config,
})
);
devEnv.runtimes.forEach((runtime) =>
runtime.onBundleComplete({
type: "bundleComplete",
config,
bundle,
})
);
// Immediately switch runtime
config = { ...config, dev: { ...config.dev, remote: !firstRemote } };
bundle = {...bundle, entrypointSource: "export default { fetch() { return new Response('Hello World " + (firstRemote ? 'local' : 'remote') + " runtime') } }"}
devEnv.proxy.onConfigUpdate({
type: "configUpdate",
config,
});
devEnv.runtimes.forEach((runtime) =>
runtime.onBundleStart({
type: "bundleStart",
config,
})
);
devEnv.runtimes.forEach((runtime) =>
runtime.onBundleComplete({
type: "bundleComplete",
config,
bundle,
})
);
const { proxyWorker } = await devEnv.proxy.ready.promise;
await devEnv.proxy.runtimeMessageMutex.drained();
console.log(await proxyWorker.dispatchFetch("http://example.com").then(r => r.text()))
process.exit(0);
`,
"package.json": dedent`
{
"name": "ai-app",
"version": "0.0.0",
"private": true
}
`,
});
});
it("can switch from local to remote, with first fetch returning remote", async () => {
const { stdout } = await run`$ node index.mjs local`;
expect(stdout).toContain("Hello World remote runtime");
});
it("can switch from remote to local, with first fetch returning local", async () => {
const { stdout } = await run`$ node index.mjs remote`;
expect(stdout).toContain("Hello World local runtime");
});
});
51 changes: 27 additions & 24 deletions packages/wrangler/e2e/dev-with-resources.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,12 @@ import { killAllWranglerDev } from "./helpers/wrangler";
beforeEach(killAllWranglerDev);
afterEach(killAllWranglerDev);

const RUNTIMES = [{ runtime: "local" }, { runtime: "remote" }] as const;
const RUNTIMES = [
{ flags: "", runtime: "local" },
{ flags: "--remote", runtime: "remote" },
{ flags: "--x-dev-env", runtime: "local" },
{ flags: "--remote --x-dev-env", runtime: "remote" },
] as const;

// WebAssembly module containing single `func add(i32, i32): i32` export.
// Generated using https://webassembly.github.io/wabt/demo/wat2wasm/.
Expand All @@ -21,9 +26,8 @@ const WASM_ADD_MODULE = Buffer.from(
"base64"
);

describe.each(RUNTIMES)("Core: $runtime", ({ runtime }) => {
describe.each(RUNTIMES)("Core: $flags", ({ runtime, flags }) => {
const isLocal = runtime === "local";
const runtimeFlags = isLocal ? [] : ["--remote"];

e2eTest(
"works with basic modules format worker",
Expand All @@ -50,7 +54,7 @@ describe.each(RUNTIMES)("Core: $runtime", ({ runtime }) => {
}
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
let res = await fetch(url);

Expand Down Expand Up @@ -92,7 +96,7 @@ describe.each(RUNTIMES)("Core: $runtime", ({ runtime }) => {
});
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
let res = await fetch(url);
expect(await res.text()).toBe("service worker");
Expand Down Expand Up @@ -145,7 +149,7 @@ describe.each(RUNTIMES)("Core: $runtime", ({ runtime }) => {
}
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.json()).toEqual({
Expand Down Expand Up @@ -174,7 +178,7 @@ describe.each(RUNTIMES)("Core: $runtime", ({ runtime }) => {
`,
});
const worker = run(
`wrangler dev ${runtimeFlags} --inspector-port=${inspectorPort}`
`wrangler dev ${flags} --inspector-port=${inspectorPort}`
);
await waitForReady(worker);
const inspectorUrl = new URL(`ws://127.0.0.1:${inspectorPort}`);
Expand All @@ -200,7 +204,7 @@ describe.each(RUNTIMES)("Core: $runtime", ({ runtime }) => {
}
`,
});
const worker = run(`wrangler dev ${runtimeFlags} --local-protocol=https`);
const worker = run(`wrangler dev ${flags} --local-protocol=https`);
const { url } = await waitForReady(worker);
const parsedURL = new URL(url);
expect(parsedURL.protocol).toBe("https:");
Expand All @@ -227,20 +231,18 @@ describe.each(RUNTIMES)("Core: $runtime", ({ runtime }) => {
`,
});
// TODO(soon): explore using `--host` for remote mode in this test
const worker = run(
`wrangler dev ${runtimeFlags} --local-upstream=example.com`
);
const worker = run(`wrangler dev ${flags} --local-upstream=example.com`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.text()).toBe("http://example.com/");
}
);
});

describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
describe.each(RUNTIMES)("Bindings: $flags", ({ runtime, flags }) => {
const isLocal = runtime === "local";
const runtimeFlags = isLocal ? "" : "--remote";
const resourceFlags = isLocal ? "--local" : "";
const d1ResourceFlags = isLocal ? "" : "--remote";

e2eTest(
"exposes basic bindings in service workers",
Expand Down Expand Up @@ -273,7 +275,7 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
});
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.json()).toEqual({
Expand Down Expand Up @@ -305,7 +307,7 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
});
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.text()).toBe("3");
Expand Down Expand Up @@ -333,14 +335,15 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
"src/index.ts": dedent`
export default {
async fetch(request, env, ctx) {
console.log(await env.NAMESPACE.list())
const value = await env.NAMESPACE.get("existing-key");
await env.NAMESPACE.put("new-key", "new-value");
return new Response(value);
}
}
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.text()).toBe("existing-value");
Expand Down Expand Up @@ -390,7 +393,7 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
`,
});

const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.text()).toBe("<h1>👋</h1>");
Expand Down Expand Up @@ -446,7 +449,7 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
}
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.text()).toBe("existing-value");
Expand Down Expand Up @@ -494,16 +497,16 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
`,
});

// D1 defaults to `--local`, so we deliberately use `runtimeFlags`, not `resourceFlags`
await run(`wrangler d1 execute ${runtimeFlags} DB --file schema.sql`);
// D1 defaults to `--local`, so we deliberately use `flags`, not `resourceFlags`
await run(`wrangler d1 execute ${d1ResourceFlags} DB --file schema.sql`);

const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
const res = await fetch(url);
expect(await res.json()).toEqual([{ key: "key1", value: "value1" }]);

const result = await run(
`wrangler d1 execute ${runtimeFlags} DB --command "SELECT * FROM entries WHERE key = 'key2'"`
`wrangler d1 execute ${d1ResourceFlags} DB --command "SELECT * FROM entries WHERE key = 'key2'"`
);
expect(result).toContain("value2");
}
Expand Down Expand Up @@ -538,7 +541,7 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {
}
`,
});
const worker = run(`wrangler dev ${runtimeFlags}`);
const worker = run(`wrangler dev ${flags}`);
const { url } = await waitForReady(worker);
await fetch(url);
await worker.readUntil(/✉️/);
Expand All @@ -558,7 +561,7 @@ describe.each(RUNTIMES)("Bindings: $runtime", ({ runtime }) => {

describe.each(RUNTIMES)("Multi-Worker Bindings: $runtime", ({ runtime }) => {
const isLocal = runtime === "local";
const _runtimeFlags = isLocal ? [] : ["--remote"];
const _flags = isLocal ? [] : ["--remote"];

// TODO(soon): we already have tests for service bindings in `dev.test.ts`,
// but would be good to get some more for Durable Objects
Expand Down
Loading

0 comments on commit 21573f4

Please sign in to comment.