feature: engine extension #13718
Open
feature: engine extension #13718
+11,708
−1,575
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Begins the major architectural refactoring to enable external execution engines. Core Architecture: - Split ExecutionEngine into Discovery/Instance pattern - Introduce _discovery flag for new pattern - Create EngineProjectContext as limited ProjectContext subset - Enable loading engines from external URLs - Rename LaunchedExecutionEngine → ExecutionEngineInstance Quarto API Foundation: - Create @quarto/types package with TypeScript definitions - Start implementing QuartoAPI interface - Begin building type system for external engines Co-Authored-By: Claude <noreply@anthropic.com>
Ports Julia engine to the new ExecutionEngineDiscovery/Instance pattern while continuing to build out the Quarto API. Julia Engine: - Port julia to ExecutionEngineDiscovery/Instance interfaces - Explore moving julia out of core (attempted, reverted, restored with override) - Lay groundwork for julia as bundled extension Quarto API Expansion: - Add quarto.jupyter namespace (capabilities, daemon management) - Add quarto.path.* APIs (resource, runtime, dataDir) - Add quarto.format utilities and complete Format type - Add quarto.system APIs for process execution - Add mapped string utilities - Bundle types package for distribution - Pass quarto API to init(), claimsFile, populateCommand - Rename asLaunchedEngine → asEngineInstance throughout Co-Authored-By: Claude <noreply@anthropic.com>
Completes porting of core engines to ExecutionEngineDiscovery pattern and finishes core Quarto API functionality. Jupyter Engine Port: - Port jupyter to ExecutionEngineDiscovery interface (7-part series) - Update kernel management and cell execution for new pattern - Fix uses of legacy ExecutionEngine in jupyter code Knitr Engine Port: - Port knitr to ExecutionEngineDiscovery interface - Update R Markdown engine to work with new pattern API Completion: - Add system.onCleanup and path.dataDir to API - Move _discovery flag to engine implementations - Reorganize types by function for better structure - Clean up type exports and build configuration - Remove quarto.markdown, relocate asYamlText and breakQuartoMd - Port percent.ts to use quarto API - Use mapped types for FormatIdentifier Co-Authored-By: Claude <noreply@anthropic.com>
Completes the architecture migration by cleaning up legacy code and making the check command extensible for external engines. Julia as Bundled Extension: - Use ported julia engine from extension (not core) - Julia now serves as reference implementation for external engines Legacy Code Removal: - Remove legacy ExecutionEngine adapter pattern - Remove deprecated ExecutionEngine interface and _discovery flags - Add quartoRequired version checking for execution engines Check Command Extensibility: - Make quarto check dynamically extensible via ExecutionEngineDiscovery - Move check logic into individual engine implementations - Load external engines in cmd.ts before validation - Add quarto.system.checkRender() API - Fix issue where check was discovering CWD as project API Polish: - Fix jupyter API message function signatures in quarto-types - Add console namespace to Quarto API (spinner, output utilities) - Add inputFilesDir and additional Jupyter methods Co-Authored-By: Claude <noreply@anthropic.com>
Implements tooling and infrastructure for managing engine extensions as git subtrees, enabling them to be maintained in separate repositories while bundling them with Quarto releases. Git Subtree Infrastructure: - Add 'quarto dev-call pull-git-subtree' command to update subtree extensions from their source repositories - Support git subtree directory structure in bundled extensions - Fix pull-git-subtree to search current branch only with debug logging - Remove re-authoring logic from pull-git-subtree Julia Bundled Extension: - Move julia engine to bundled extensions (first bundled engine) - Update tests to account for julia-engine bundled subtree extension Build Fix: - Fix quarto-bld missing V8 experimental regexp engine flag Co-Authored-By: Claude <noreply@anthropic.com>
Final touches to the engine architecture and improvements to Julia engine testing. Architecture Polish: - Fix Windows dynamic import by converting paths to file:// URLs - Make engineCommand dynamic to support external engines - Correct relative path handling - Refactor project initialization and fix julia test cleanup Julia Testing: - Refactor julia tests to use proper Deno test steps - Improve test structure and cleanup Co-Authored-By: Claude <noreply@anthropic.com>
Single-file renders now initialize a ProjectContext with config, enabling
both engine and metadata extensions to work without requiring _quarto.yml.
Key changes:
- Single-file ProjectContext now has config = { project: {} } instead of
undefined, making it consistent with multi-file projects
- Pass full RenderOptions to singleFileProjectContext for extension services
- Export mergeExtensionMetadata() and resolveEngineExtensions() from
project-context.ts for use in single-file path
- Call both functions in singleFileProjectContext() following same order
as regular projects
- Use context.dir consistently for extension discovery
Extension output-dir handling:
- When extensions contribute output-dir metadata, set forceClean to match
the behavior of --output-dir flag, ensuring proper cleanup of the
temporary .quarto directory (addresses #9745, #13625)
- Warn users when extension contributes output-dir, showing the path and
whether cleanup will occur (respects --no-clean flag)
Co-Authored-By: Claude <noreply@anthropic.com>
Completes the external engine ecosystem with user-facing tools and organizational improvements. Git Subtree Organization: - Separate git subtree extensions into extension-subtrees/ directory to distinguish from other bundled extensions - Update subtree extension directory handling and tests - Remove bundled subtree extensions from input to pandoc to avoid duplicate processing Engine Template: - Add 'engine' extension template for quarto create - Include Discovery/Instance pattern boilerplate - Fix hardcoded "example" references in engine template - Improve engine template and error handling - Add cell language prompt for engine configuration - Add engine extension type to basic creation tests Metadata Display: - Filter bundled engines from metadata display output - Suppress expected/noisy engine metadata while preserving user's custom engines Co-Authored-By: Claude <noreply@anthropic.com>
this was a red herring in my investigation, but this is the correct thing to do will squash into the proper commit later.
that reaches out of quarto-cli this caused quarto-latexmk not to build correctly
yet another instance of #9737
this was causing dependencies to blow up where quarto-latexmk was trying to load all execution engines
Co-Authored-By: Claude <noreply@anthropic.com>
Implements MVP of `quarto dev-call build-ts-extension` command that: - Type-checks TypeScript extensions against Quarto API types - Bundles extensions into single JavaScript files using esbuild - Provides default configuration for extensions Key features: - Auto-detects entry point (config, single file, or mod.ts) - Config resolution (user deno.json or default from share/extension-build/) - Type checking with Deno's bundled binary - Bundling with esbuild's bundled binary - --check flag for type-check only Distribution setup: - Static config templates in src/resources/extension-build/ - Copied to share/extension-build/ during packaging - Works in both dev mode and distribution mode Test extension: - Minimal test in tests/smoke/build-ts-extension/ - Implements ExecutionEngineDiscovery interface - Demonstrates type-checking and bundling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…odes Problem: import-map.json pointed to ./quarto-types.d.ts which doesn't exist in dev mode (src/resources/extension-build/). Solution: Follow Lua filter transformation pattern: - Source file in src/resources/ uses dev-mode path - Transform during packaging for distribution Changes: - import-map.json now points to ../../../packages/quarto-types/dist/index.d.ts (works in dev mode from src/resources/extension-build/) - Added updateImportMap() function in prepare-dist.ts that: - Runs after supportingFiles() copies all resources - Transforms @quarto/types path to ./quarto-types.d.ts for distribution - Updates all Deno std versions from src/import_map.json - Loops over all imports dynamically (not hardcoded list) - Updated README.md to document dev vs distribution behavior Result: Default config now works in dev mode without committing generated artifacts to src/resources/. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Implements --init-config flag that generates a minimal deno.json with
an absolute importMap path pointing to Quarto's bundled configuration.
Features:
- Checks if deno.json already exists and errors with helpful message
- Creates minimal config with compilerOptions and absolute importMap path
- Uses resourcePath() to get correct path in dev and distribution modes
- Exits without building (config generation only)
- Shows helpful customization guidance for users
Example output:
✓ Created deno.json
Import map: /usr/local/share/quarto/extension-build/import-map.json
Customize as needed:
- Add "quartoExtension" section for build options
- Modify "compilerOptions" for type-checking behavior
Generated deno.json points to Quarto's shared import-map.json which
provides @quarto/types and Deno std library imports with correct versions.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Remove vendored quarto-types approach and restructure engine template to match the build-ts-extension specification. Changes: - Remove backward compatibility quarto-types copy from prepare-dist.ts - Remove vendoring logic from artifacts/extension.ts - Move TypeScript source from _extensions/ to src/ in engine template - Update _extension.yml to reference .js output instead of .ts - Update imports to use @quarto/types via import map - Add build instructions to README - Fix deno.json to include "deno.ns" in lib array (both default and --init-config) Engine extensions now follow the spec: - Source in src/ - Build to _extensions/<name>/<name>.js using build-ts-extension - Reference types via import map (no vendored copies) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Replace manual directory scanning with expandGlobSync to find _extension.yml
files, properly supporting nested organization structures like
_extensions/org-name/extension/_extension.yml.
Changes:
- Use expandGlobSync("_extensions/**/_extension.yml") pattern
- Support both flat and nested extension directory structures
- Simplify logic by removing manual directory iteration
- Improve error messages with cleaner relative path display
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Remove Deno std entries from import map since esbuild cannot bundle external URL schemes (jsr:, npm:, https:). Add externals support for marking imports as external (not bundled). Changes to quarto-cli: - Simplify import-map.json: only @quarto/types remains - Simplify updateImportMap(): just path transformation, no version syncing - Add externals?: string[] to quartoExtension interface - Implement --external flag handling in bundle function - Add default externals to deno.json: ["jsr:*", "npm:*", "https:*", "http:*"] Rationale: - esbuild binary only bundles relative file paths - External URL schemes must be resolved at runtime by Deno - Import map now serves single purpose: @quarto/types aliasing - Extension authors use full JSR URLs in source code - Default externals cover all non-bundleable schemes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Switch from esbuild to deno bundle for better import map support and
simpler tooling. deno bundle natively handles JSR/npm/https imports
and does type-checking + bundling in one step.
Changes:
- Replace esbuild with deno bundle in bundle() function
- Remove typeCheck() function (deno bundle does both)
- Restore import map entries for Deno std libraries
- Restore updateImportMap() version syncing in prepare-dist.ts
- Remove externals from quartoExtension interface (not needed)
- Add validateExtensionYml() to warn about path mismatches
- Simplify main action: --check uses deno check, normal build uses deno bundle
Benefits:
- Import maps work natively (no external URL workarounds)
- Extension authors can use bare imports ("path" vs "jsr:@std/path@1.0.8")
- One tool instead of two (simpler, faster)
- Native support for jsr:, npm:, https: imports
- Version consistency with Quarto maintained via import map
- ~60 lines of code removed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Adds support for specifying the entry point as a command-line argument: quarto dev-call build-ts-extension [entry-point] Entry point resolution priority (highest to lowest): 1. [entry-point] command-line argument 2. quartoExtension.entryPoint in deno.json 3. Single .ts file in src/ directory 4. src/mod.ts if multiple files exist This allows building extensions with multiple TypeScript files without requiring a deno.json just to specify the entry point. Updated error messages to suggest both CLI argument and deno.json options. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Renamed the deno.json configuration section from "quartoExtension" to
"bundle" for better semantic alignment with the underlying deno bundle
command.
Mapping to deno bundle CLI options:
- bundle.entryPoint → [file] positional argument
- bundle.outputFile → --output <output> (✅)
- bundle.minify → --minify (✅)
- bundle.sourcemap → --sourcemap[=<sourcemap>] (✅)
The "bundle" name is more intuitive and accurately reflects that these
are bundling configuration options that map to deno bundle CLI flags.
Updated all references:
- Interface definition: DenoConfig.bundle
- All config.bundle references throughout
- Documentation strings
- Error messages with example JSON
Example deno.json:
{
"bundle": {
"entryPoint": "src/main.ts",
"outputFile": "_extensions/my-ext/main.js",
"minify": true,
"sourcemap": "linked"
}
}
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Refactored output path resolution to support three modes:
1. Just filename (no path separators):
{ "bundle": { "outputFile": "julia-engine.js" } }
→ Infers directory from _extension.yml location
→ Result: _extensions/julia-engine/julia-engine.js
2. Full path (has path separators):
{ "bundle": { "outputFile": "_extensions/custom/out.js" } }
→ Uses path as-is
→ Result: _extensions/custom/out.js
3. Omitted:
{ "bundle": {} }
→ Infers both filename (from entry point) and directory
→ Result: _extensions/julia-engine/julia-engine.js
Implementation changes:
- Split into two focused functions:
- inferFilename(entryPoint): derives "foo.js" from "src/foo.ts"
- inferOutputPath(outputFilename): finds _extension.yml and joins with filename
- Updated bundle() to detect filename-only vs full-path
- Each function has single responsibility, no coupling
This provides flexibility while maintaining convention over configuration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
…rces/extension-subtrees/julia-engine'
…m commit 2fb2adf6e git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 2fb2adf6ef7b24272ff76a54ee0b51b5f6962a66
gordonwoodhull
force-pushed
the
feature/engine-extension-3
branch
from
ecb4468 to
9aedc82
Compare
Moves the build-ts-extension command from the hidden dev-call namespace to the public call command namespace, making it accessible to users. Changes: - Moved src/command/dev-call/build-ts-extension/ to src/command/call/build-ts-extension/ - Removed .hidden() flag from command definition - Updated error messages to reference 'quarto call build-ts-extension' - Added command registration to src/command/call/cmd.ts - Removed command registration from src/command/dev-call/cmd.ts The command is now available as: quarto call build-ts-extension [entry-point] 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…engine-extension-3
…m 2fb2adf6e..db03cb0c5 db03cb0c5 add checkInstallation stub git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: db03cb0c58661064db5bd6c1c083b828c8276749
…e context Fixes the issue where `quarto check julia` was failing with "Invalid value 'julia'" even though Julia is available as a bundled engine extension. The problem: When running commands like `quarto check julia` outside a project directory, no project context exists, so bundled engine extensions were never loaded. The solution: Create a zeroFileProjectContext() that provides just enough structure to discover and register bundled engine extensions when no actual project or file exists. The function initializeProjectContextAndEngines() now falls back to this zero-file context when no project is found. Also changed initializeProjectContextAndEngines() to return Promise<void> instead of Promise<ProjectContext | undefined>, since the name suggests initialization rather than retrieval and both callers ignore the return value. Now `quarto check julia` works both inside and outside project directories. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Make error messages more helpful by: - Only showing deno.json configuration when deno.json already exists - This discourages users from creating deno.json when they don't need it - Use relative paths in examples instead of absolute paths - Order solutions from simplest to most complex (CLI arg, then rename, then config) Changes: - "No src/ directory" error: Show CLI argument first, only show deno.json config if it exists - "Multiple .ts files" error: Show CLI argument first, then rename to mod.ts, then deno.json (if exists) - "Multiple extensions" error: If no deno.json, explain this isn't supported and suggest --init-config; if deno.json exists, show how to configure outputFile - Use relative paths in outputFile example (strip absolute path prefix) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- llm-docs/quarto-api.md: Step-by-step guide for adding functionality to the Quarto API (type definitions in @quarto/types and implementation in quarto-api.ts), including usage patterns for internal and external engines - llm-docs/error-messages.md: Style guide for writing idiomatic error messages with proper newline placement (end messages with \n, never start with \n or use empty calls)
Engine extensions now require quarto call build-ts-extension to bundle TypeScript before rendering. Updated test to build before attempting to render the created extension.
When a test fails on Windows, exit immediately instead of continuing through remaining tests in the bucket. This matches Linux behavior (bash -e flag) and keeps test failures at the end of the log where they're easy to find, instead of buried under 1000+ lines of subsequent test output.
also, load builtin engine extensions
…m db03cb0c5..7eaf7bc3d 7eaf7bc3d claude: use quarto api for logging git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 7eaf7bc3d77aa439805baa9d59df96633825fa48
…engine-extension-3
…m 7eaf7bc3d..9bb6ee655 9bb6ee655 artifact git-subtree-dir: src/resources/extension-subtrees/julia-engine git-subtree-split: 9bb6ee655388780db187cb48bdd8e34aed89148d
…engine-extension-3
…prove error messages Check configEntryPoint before validating src/ directory existence, allowing non-extension projects to specify custom entry points. Clarify error message when outputFile lacks a path by suggesting ./ prefix as an alternative to creating _extensions structure. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add test that verifies build-ts-extension correctly bundles TypeScript engine extensions. Test validates that imports from the import map (like @std/path) are properly bundled into the output JavaScript file. Remove deno.json from test to verify auto-detection of entry point from src/ directory works correctly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Wrap log.info/warning/error in arrow functions instead of direct references in quartoAPI object. This prevents the bundler from creating top-level async initialization code.
When functions are assigned directly at module level (e.g., `{ info, error }`), the bundler assumes they might be accessed immediately and inserts top-level await calls to initialize underlying Deno extensions. Wrapping them in arrow functions defers access until the functions are actually called, avoiding the top-level await issue.
Fixes prepare-dist workflow error:
await init_quarto_api();
^
SyntaxError: Unexpected reserved word
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
replaces #13701
I want to see ci succeed before merging!