OpenAPI Preprocessor

Preprocesses our OpenAPI specs to prepare them for generation via Speakeasy (our API client library generator).

Spec Directories

This repository manages several types of OpenAPI specifications in different directories:

Directory	Purpose
source_specs/	Original OpenAPI specification files provided as input to the transformation process. These are the source of truth for our API definitions.
generated_specs/	Transformed OpenAPI specs with server URL subpaths moved to individual API paths. These files are consumed by Speakeasy to generate client libraries.
overlayed_specs/	Specs with various overlays applied (see Overlays section below). The overlays add additional metadata or modifications needed for specific purposes.
merged_code_samples_specs/	Specs with code samples merged from multiple sources. These enhanced specs provide examples for documentation and developer usage.
modified_code_samples_specs/	Specs with code samples post processed (correcting issues with the generated code samples).
final_specs/	Final, fully proccessed OpenAPI specifications that combine code samples from multiple sources that have been post-processed for correctness. These files are consumed by the gh-pages site and ultimately make their way to developers.glean.com.

Overlays

The overlays directory contains OpenAPI Specification overlay files used to modify the base specs:

File	Description
info-name-overlay.yaml	Modifies the API title and adds Speakeasy naming metadata to improve SDK generation.
strip-headers-overlay.yaml	Removes specific authentication headers from the API specification that shouldn't be exposed in the generated SDKs.
speakeasy-modifications-overlay.yaml	Contains extensive Speakeasy-specific modifications to improve SDK generation, including method name overrides and grouping information.

These overlay files use the OpenAPI Specification Overlay format (RFC9535) to apply targeted modifications to the base specs without changing the original files. Speakeasy uses these overlays during the SDK generation process.

Merged Code Samples Specs

The merged_code_samples_specs directory contains OpenAPI specifications with embedded code samples:

File	Description
glean-client-merged-code-samples-spec.yaml	Client API specification enhanced with code samples in multiple programming languages. Used for documentation and SDK generation.
glean-index-merged-code-samples-spec.yaml	Indexing API specification enhanced with code samples in multiple programming languages. Used for documentation and SDK generation.

These files are generated by Speakeasy after external repositories have run their workflows and uploaded code samples to the Speakeasy registry. They provide developers with working code examples for each API endpoint.

Overlayed Specs

The overlayed_specs directory contains merged OpenAPI specifications that combine multiple API definitions:

File	Description
glean-merged-spec.yaml	A comprehensive merged spec containing both the Client and Indexing APIs in a single document. This provides a unified view of all Glean APIs and includes Speakeasy-specific extensions (x-speakeasy-*) for code generation.

These merged specs are used for generating consistent client libraries across multiple APIs and provide a single source of documentation.

Modified Code Samples Specs

This takes the specs from merged_code_samples_specs and runs pnpm transform:merged_code_samples_specs to correct issues with the generated code samples. This is done in the .github/workflows/generate-code-samples.yml GHA workflow.

Final Specs

The final_specs directory contains the end product of all of the processing being done. These specs are copied into the published GitHub Pages site (in the docs/specs/final directory), and then used by gleanwork/glean-developer-site.

Overview Diagram

flowchart TD
    %% Main directory nodes
    source_specs["source_specs"]:::dir
    generated_specs["generated_specs"]:::dir
    overlayed_specs["overlayed_specs"]:::dir
    merged_code_samples_specs["merged_code_samples_specs"]:::dir

    %% Workflow nodes
    transform_yml["transform.yml"]:::wf
    generate_code_samples_yml["generate-code-samples.yml"]:::wf
    deploy_pages_yml["deploy-pages.yml"]:::wf

    %% External resources
    overlays((Overlays)):::ext
    speakeasy_registry["Speakeasy Registry"]:::reg
    glean_developer_docs["glean-developer-docs<br>(GitHub Pages)"]:::ext

    %% External repos - arranged in a 2x2 grid for clarity
    api_python["api-client-python"]:::ext
    api_typescript["api-client-typescript"]:::ext
    api_go["api-client-go"]:::ext
    api_java["api-client-java"]:::ext

    %% STAGE 1: Transform source specs
    subgraph Stage1[STAGE 1: Source Transformation]
        source_specs --> transform_yml --> generated_specs
        generated_specs --> overlays --> overlayed_specs
    end

    %% STAGE 2: External repos process specs
    subgraph Stage2[STAGE 2: Client SDKs]
        overlayed_specs --> api_clients

        subgraph api_clients[API Client Repos]
            api_python & api_typescript & api_go & api_java
        end

        %% All API clients upload samples to registry
        api_python & api_typescript & api_go & api_java -->|speakeasy run| speakeasy_registry
    end

    %% STAGE 3: Generate code samples and deploy
    subgraph Stage3[STAGE 3: Publish]
        speakeasy_registry --> generate_code_samples_yml --> merged_code_samples_specs
        merged_code_samples_specs --> deploy_pages_yml --> glean_developer_docs
    end

    %% Styling definitions
    classDef dir fill:#FAFCFF,stroke:#0057FF,stroke-width:1px,color:#0057FF;
    classDef wf fill:#EDF3FF,stroke:#0057FF,stroke-width:2px,color:#0057FF;
    classDef ext fill:#F5FFE8,stroke:#D9FF1F,stroke-width:1px,color:#222;
    classDef reg fill:#000000,stroke:#FFD700,stroke-width:1px,color:#FFD700,font-style:italic;
    classDef section fill:#f5f5f5,stroke:#cccccc,color:#333333;

    %% Style the stages
    classDef stage fill:#f5f5f5,stroke:#cccccc,color:#333333,stroke-dasharray: 5 5;
    class Stage1,Stage2,Stage3 stage;

Loading

Workflows

This repository uses several GitHub Actions workflows to process the OpenAPI specifications:

Workflow	Purpose	Trigger
transform.yml	Transforms source specs and runs the glean-api-specs Speakeasy source to generate API specs.	Push to main (source_specs changes), schedule, manual
generate-code-samples.yml	Runs Speakeasy to generate merged code samples specs.	Manual only
deploy-pages.yml	Deploys specs to GitHub Pages.	After generate-code-samples completes, manual

The processing follows this sequence:

1. Transform specs from source files
2. External repositories consume these specs and generate their code samples
3. We manually trigger code sample generation which pulls from Speakeasy registry
4. GitHub Pages deployment publishes all specs for browsing

Transformation Process

The transformation script processes OpenAPI YAML specification files by:

1. Reading files from the source_specs/ directory
2. Moving the server URL subpath to each individual API path
3. Writing the transformed files to the generated_specs/ directory

This script performs the following transformations:

• Before: servers.url = "https://{domain}-be.glean.com/rest/api/v1", path = "/activity"
• After: servers.url = "https://{domain}-be.glean.com/", path = "/rest/api/v1/activity"

Output

Transformed files are saved to the generated_specs/ directory:

• generated_specs/client_rest.yaml
• generated_specs/indexing.yaml
• generated_specs/admin_rest.yaml

Usage

To run the transformation:

pnpm run transform

Development

To set up the development environment:

1. Clone the repository

2. Install dependencies:

   pnpm install

3. Run the transformation:

   pnpm run transform

4. Run tests:

   pnpm test