Skip to content

Deploying Open VSX

Jump to bottom
amvanbaren edited this page Oct 24, 2024 · 13 revisions

Architectural Overview

Overview diagram

The central element is the server application, which is available in the openvsx-server Docker image. It is a Spring Boot application and needs an application.yml file to configure the deployment. Open VSX does not provide any facility to deploy the other components (database, search engine etc.) because there are numerous ways how the infrastructure can be set up. Using Kubernetes is one option, but that is not mandatory.

The database holding all metadata of published extensions is a PostgreSQL instance. Up until version 0.17.0 all files are stored as binary data in the database in case no additional file storage is used. Though this setup is supported by Open VSX, it considerably increases storage and network throughput of the database. From version 0.18.0 files are stored on the local file system. For larger scale deployments using an external file storage is recommended. Currently Azure Blob Storage and Google Cloud Storage are supported as external storage providers.

Elasticsearch is used as default search engine for search queries from the web UI. As an alternative, you can choose to search via database queries, which will likely result in worse performance, or to completely disable the search functionality. In the latter case, all other Open VSX features are still available.

User authentication is done with OAuth. Currently only GitHub is supported as OAuth provider.

Getting Started

You can quickly spin up an Open VSX server and webui using the DockerFile below. Additionally you need a PostgreSQL instance and an Elasticsearch instance. To use the user and admin sections of the webui, you need to configure a GitHub OAuth app. For troubleshooting and further configuration, see issue #703.

DockerFile

ARG OPENVSX_VERSION

# Builder image to compile the website
FROM ubuntu as builder

WORKDIR /workdir

RUN apt-get update \
  && apt-get install --no-install-recommends -y \
    bash \
    ca-certificates \
    git \
    curl \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

# See https://github.com/nodesource/distributions/blob/main/README.md#debinstall
RUN curl -sSL https://deb.nodesource.com/setup_14.x | bash - \
  && apt-get install -y nodejs

RUN npm install --global yarn@1.*

ARG OPENVSX_VERSION
ENV VERSION=$OPENVSX_VERSION

RUN git clone --branch ${VERSION} --depth 1 https://github.com/eclipse/openvsx.git /workdir
RUN /usr/bin/yarn --cwd webui \
  && /usr/bin/yarn --cwd webui build \
  && /usr/bin/yarn --cwd webui build:default

# Main image derived from openvsx-server
FROM ghcr.io/eclipse/openvsx-server:${OPENVSX_VERSION}

COPY --from=builder --chown=openvsx:openvsx /workdir/webui/static/ BOOT-INF/classes/static/

Building Open VSX Docker image

The command gets the latest release tag name and uses it to build an openvsx image from the DockerFile in the current working directory.

export OPENVSX_VERSION=`curl -sSL https://api.github.com/repos/eclipse/openvsx/releases/latest | jq -r ".tag_name"`

docker build -t "openvsx:$OPENVSX_VERSION" --build-arg "OPENVSX_VERSION=$OPENVSX_VERSION" .

Configuring application.yml

The Open VSX server is configured using an application.yml file. See the Spring Boot documentation for more information on this configuration format.

The server application will automatically load the configuration file if you put it into the directory /home/openvsx/server/config of the server image. There are several ways to do this, e.g. you can extend the Docker image or use a Kubernetes ConfigMap.

You can use all configuration properties offered by Spring to set up your deployment. In particular, you need to configure a datasource to connect the application with the database.

Open VSX Configuration Properties

This section describes the special configuration properties supported by the Open VSX server.

Registry

Property ovsx.registry.version
Type string
Default
Compatibility Since 0.15.0

The version of the running Open VSX registry instance.

Publishing

Property ovsx.publishing.require-license
Type boolean
Default false
Compatibility Since 0.1.0

Whether published extensions are required to have a license. If active, unlicensed extensions are rejected.

Web UI

Property ovsx.webui.url
Type string
Default
Compatibility Since 0.1.0

Base URL of the web UI. This is required only if it's different from the server.

Property ovsx.webui.frontendRoutes
Type string[]
Default /extension/**,/namespace/**,/user-settings/**,/admin-dashboard/**
Compatibility Since 0.1.0

Routes to be forwarded to / because they are handled by the frontend.

Search Options

Property ovsx.search.relevance.rating
Type double
Default 1.0
Compatibility Since 0.2.0

Weight of user ratings for computing relevance. This has an impact on the order of search results when sortBy is set to relevance.

Property ovsx.search.relevance.downloads
Type double
Default 1.0
Compatibility Since 0.2.0

Weight of download counts for computing relevance. This has an impact on the order of search results when sortBy is set to relevance.

Property ovsx.search.relevance.timestamp
Type double
Default 1.0
Compatibility Since 0.2.0

Weight of publishing timestamps for computing relevance (newer extensions are ranked higher). This has an impact on the order of search results when sortBy is set to relevance.

Property ovsx.search.relevance.unverified
Type double
Default 0.5
Compatibility Since 0.2.0

Relevance factor for unverified extension versions. The combined relevance from the averageRating, downloadCount and timestamp criteria is multiplied with this value if the publisher of the extension is not a member of the extension's namespace or the namespace has no owner.

Property ovsx.search.relevance.deprecated
Type double
Default 0.5
Compatibility Since 0.17.0

Relevance factor for deprecated extension versions. The combined relevance from the averageRating, downloadCount and timestamp criteria is multiplied with this value if the extension is deprecated.

Elasticsearch

Property ovsx.elasticsearch.enabled
Type boolean
Default true
Compatibility Since 0.1.0

Whether to enable search functionality through Elasticsearch. By switching this off, it is not necessary to deploy Elasticsearch. Cannot be used together with ovsx.databasesearch.enabled.

Property ovsx.elasticsearch.clear-on-start
Type boolean
Default false
Compatibility Since 0.1.0

Whether to clear and rebuild the search index on startup. If disabled, the index is built only if it does not exist yet. Rebuilding the search index may take several minutes if there are many extensions.

Property ovsx.elasticsearch.host
Type string
Default localhost:9200
Compatibility Since 0.1.0

Host and port of the Elasticsearch instance.

Property ovsx.elasticsearch.ssl
Type boolean
Default false
Compatibility Since 0.1.0

Whether to connect with SSL.

Property ovsx.elasticsearch.username
Type string
Default
Compatibility Since 0.1.0

Username for basic authentication.

Property ovsx.elasticsearch.password
Type string
Default
Compatibility Since 0.1.0

Password for basic authentication.

Property ovsx.elasticsearch.truststore
Type string
Default
Compatibility Since 0.1.0

Path to a trust store file for SSL connection.

Property ovsx.elasticsearch.truststoreProtocol
Type string
Default TLSv1.2
Compatibility Since 0.1.0

Protocol for SSL connection.

Property ovsx.elasticsearch.truststorePassword
Type string
Default
Compatibility Since 0.1.0

Password for trust store file.

Property ovsx.elasticsearch.relevance.rating
Type double
Default 1.0
Compatibility Since 0.1.0, deprecated in 0.2.0, removed in 0.18.0 (use ovsx.search.relevance.rating)

Weight of user ratings for computing relevance. This has an impact on the order of search results when sortBy is set to relevance.

Property ovsx.elasticsearch.relevance.downloads
Type double
Default 1.0
Compatibility Since 0.1.0, deprecated in 0.2.0, removed in 0.18.0 (use ovsx.search.relevance.downloads)

Weight of download counts for computing relevance. This has an impact on the order of search results when sortBy is set to relevance.

Property ovsx.elasticsearch.relevance.timestamp
Type double
Default 1.0
Compatibility Since 0.1.0, deprecated in 0.2.0, removed in 0.18.0 (use ovsx.search.relevance.timestamp)

Weight of publishing timestamps for computing relevance (newer extensions are ranked higher). This has an impact on the order of search results when sortBy is set to relevance.

Property ovsx.elasticsearch.relevance.unverified
Type double
Default 0.5
Compatibility Since 0.1.0, deprecated in 0.2.0, removed in 0.18.0 (use ovsx.search.relevance.unverified)

Relevance factor for unverified extension versions. The combined relevance from the averageRating, downloadCount and timestamp criteria is multiplied with this value if the publisher of the extension is not a member of the extension's namespace or the namespace has no owner.

Database Search

Property ovsx.databasesearch.enabled
Type boolean
Default false
Compatibility Since 0.2.0

Whether to enable search functionality though DB queries. Cannot be used together with ovsx.elasticsearch.enabled.

File Storage

Property ovsx.storage.azure.service-endpoint
Type string
Default
Compatibility Since 0.1.0

Azure blob service endpoint URL (without parameters, must end with a slash). This is required in order to enable the Azure storage service. Example: https://openvsx.blob.core.windows.net/.

Property ovsx.storage.azure.sas-token
Type string
Default
Compatibility Since 0.1.0

The full query string containing the Azure SAS (Shared Access Signature) token.

Property ovsx.storage.azure.blob-container
Type string
Default openvsx-resources
Compatibility Since 0.1.0

Name of the Azure blob container.

Property ovsx.storage.gcp.project-id
Type string
Default
Compatibility Since 0.1.0

GCP project id. This can be omitted if the GCP client is able to detect the project from the environment.

Property ovsx.storage.gcp.bucket-id
Type string
Default
Compatibility Since 0.1.0

GCP bucket id. This is required in order to enable the Google Cloud storage service. Note that in order to upload files you need to authenticate with the storage service, e.g. by putting service account credentials into a file and pointing the environment variable GOOGLE_APPLICATION_CREDENTIALS to that file. Unauthenticated access is possible when migrating from GCP to another storage provider.

Property ovsx.storage.primary-service
Type string
Default
Compatibility Since 0.1.0

External storage service to use if multiple are active (azure-blob or google-cloud). All files that are not in the primary service are automatically migrated on application startup.

Property ovsx.storage.external-resource-types
Type string[]
Default *
Compatibility Since 0.1.0

Resource types to store in an external storage provider, or * for all types. Possible values are download (i.e. the vsix file), manifest, icon, readme, license, changelog, vsixmanifest, sha256, signature, publicKey, resource and namespace-logo. If only a subset of those types is chosen, the remaining types are stored locally. Up to version 0.17.0 files are stored as byte arrays in the database. From version 0.18.0 files are stored in the local file system. The ovsx.storage.local.directory property must also be configured for this feature to work.

Property ovsx.storage.migration-delay
Type long
Default 500
Compatibility Since 0.1.0

Delay in milliseconds between storage type migration of each file. This delay is important to avoid excessive load in the server application, since migration is performed by the server itself on startup. Longer delays decrease server load, but increase the total duration of file migration.

Property ovsx.storage.local.directory
Type string
Default
Compatibility Since 0.18.0

Base directory for local file storage. This is required in order to enable the local storage service.

Azure Download Logs

Property ovsx.logs.azure.sas-token
Type string
Default
Compatibility Since 0.6.0

The full query string containing the Azure SAS (Shared Access Signature) token to get Azure download logs.

Property ovsx.logs.azure.service-endpoint
Type string
Default
Compatibility Since 0.6.0

Azure download logs blob service endpoint URL. This is required in order to enable the Azure download count service.

Property ovsx.logs.azure.blob-container
Type string
Default insights-logs-storageread
Compatibility Since 0.6.0

Name of the Azure download logs blob container.

Upstream Registry

Property ovsx.upstream.url
Type string
Default
Compatibility Since 0.1.0

Base URL of the upstream registry instance.

VS Code

Property ovsx.vscode.upstream.gallery-url
Type string
Default
Compatibility Since 0.1.0

Gallery URL of a registry instance from which to fetch extension UUIDs. These UUIDs are required by VS Code to identify and auto-update installed extensions. If no upstream gallery is set, random UUIDs are generated for all published extensions.

Property ovsx.vscode.upstream.update-on-start
Type boolean
Default false
Compatibility Since 0.14.2

Whether to update public ids on startup from the upstream registry instance.

Eclipse

Property ovsx.eclipse.base-url
Type string
Default
Compatibility Since 0.1.0

Base URL of the Eclipse API.

Property ovsx.eclipse.publisher-agreement.version
Type string
Default
Compatibility Since 0.1.0

Current version of the Eclipse Publisher Agreement.

Property ovsx.eclipse.publisher-agreement.timezone
Type string
Default
Compatibility Since 0.1.0, removed in 0.15.2

java.time.ZoneId for timestamps returned by the Eclipse API.

Property ovsx.eclipse.check-compliance-on-start
Type boolean
Default false
Compatibility Since 0.1.0

Whether to check for publisher compliance on startup. A publisher is compliant when they have signed the Eclipse Publisher Agreement. The ovsx.eclipse.publisher-agreement.version property must also be configured for this check to run.

Migrations

Property ovsx.migrations.delay.seconds
Type boolean
Default false
Compatibility Since 0.9.1

Delay in seconds to run migrations. This delay is important to avoid distributing migration jobs to the old server instance, which prevents server shutdown.

Mirror Mode

Property ovsx.data.mirror.enabled
Type boolean
Default false
Compatibility Since 0.9.0

Whether to enable mirror mode.

Property ovsx.data.mirror.server-url
Type string
Default
Compatibility Since 0.9.0

Base URL of the Open VSX instance to mirror.

Property ovsx.data.mirror.schedule
Type string
Default
Compatibility Since 0.9.0

When to run the mirror job. A CRON expression is expected.

Property ovsx.data.mirror.user-name
Type string
Default
Compatibility Since 0.9.0

A username for the mirror mode user, so that its actions can be identified in the admin logs.

Property ovsx.data.mirror.requests-per-second
Type double
Default Double.MAX_VALUE
Compatibility Since 0.9.0

Limit the amount of requests per second to reduce strain on the mirrored Open VSX instance.

Property ovsx.data.mirror.read-only.disallowed-methods
Type string[]
Default
Compatibility Since 0.9.0

Disallowed HTTP methods (POST, GET, etc.) for server endpoints to limit access in mirror mode, e.g. disallow POST method to block publishing.

Property ovsx.data.mirror.read-only.allowed-endpoints
Type string[]
Default
Compatibility Since 0.9.0

Allowed server endpoints in mirror mode to override disallowed methods, e.g. disallow POST method but allow posting reviews.

Foreground HTTP Connection Pool

The foreground HTTP connection pool is used to make REST requests (upstream, Eclipse API) while serving requests.

Property ovsx.foregroundHttpConnPool.maxTotal
Type integer
Default 20
Compatibility Since 0.9.0

The maximum total HTTP connections.

Property ovsx.foregroundHttpConnPool.defaultMaxPerRoute
Type integer
Default 20
Compatibility Since 0.9.0

The default maximum HTTP connections per route.

Property ovsx.foregroundHttpConnPool.connectionRequestTimeout
Type integer
Default 10000
Compatibility Since 0.9.0

The time in milliseconds to wait for a connection from the HTTP connection pool.

Property ovsx.foregroundHttpConnPool.connectTimeout
Type integer
Default 10000
Compatibility Since 0.9.0

The time in milliseconds to establish the connection with the remote host.

Property ovsx.foregroundHttpConnPool.socketTimeout
Type integer
Default 10000
Compatibility Since 0.9.0

The maximum time of inactivity in milliseconds between two data packets.

Background HTTP Connection Pool

The background HTTP connection pool is used to download files in background processing (migrations, mirror mode).

Property ovsx.backgroundHttpConnPool.maxTotal
Type integer
Default 20
Compatibility Since 0.9.0

The maximum total HTTP connections.

Property ovsx.backgroundHttpConnPool.defaultMaxPerRoute
Type integer
Default 20
Compatibility Since 0.9.0

The default maximum HTTP connections per route.

Property ovsx.backgroundHttpConnPool.connectionRequestTimeout
Type integer
Default 30000
Compatibility Since 0.9.0

The time in milliseconds to wait for a connection from the HTTP connection pool.

Property ovsx.backgroundHttpConnPool.connectTimeout
Type integer
Default 30000
Compatibility Since 0.9.0

The time in milliseconds to establish the connection with the remote host.

Property ovsx.backgroundHttpConnPool.socketTimeout
Type integer
Default 60000
Compatibility Since 0.9.0

The maximum time of inactivity in milliseconds between two data packets.

Extension Control

Property ovsx.extension-control.update-on-start
Type boolean
Default false
Compatibility Since 0.17.0

Whether to run a job on startup to check for malicious and deprecated extensions. If disabled, the job still runs nightly.

Extension Integrity

Property ovsx.integrity.key-pair
Type string
Default
Compatibility Since 0.11.0

Whether to generate a signature archive (sigzip) for published extensions. By default it does nothing. There are 3 modes: create, renew and delete. When renew is specified, then on startup a new keypair is generated and a new signature is generated for each extension version in the background. When create is specified then on startup, if no keypair exists, a keypair is created and a signature is generated for each extension version in the background. A signature is generated for newly published extension versions in both renew and create modes. The delete mode deletes the keypair and all generated signatures.

Adding the Web UI

The Docker image of the server application does not include the web UI. The reason for this is that the UI can be customized. In case you don't need customization, you can use the default web UI image and deploy it next to the server.

Customization of the UI is done by creating an npm package with a dependency on openvsx-webui, which is a library of React components. A minimal frontend app is shown in the following TypeScript-React (tsx) code.

import * as ReactDOM from 'react-dom';
import * as React from 'react';
import { BrowserRouter } from 'react-router-dom';
import { ThemeProvider } from '@material-ui/styles';
import { Main, ExtensionRegistryService } from 'openvsx-webui';

const App: React.FunctionComponent = () => {
    const theme = ...        // Define a Material UI theme
    const pageSettings = ... // Define page settings (see below)
    const service = new ExtensionRegistryService();
    return <ThemeProvider theme={theme}>
        <Main pageSettings={pageSettings} service={service} />
    </ThemeProvider>;
};

const node = document.getElementById('main');
ReactDOM.render(<BrowserRouter><App /></BrowserRouter>, node);

See the Material UI documentation and the default theme to learn how to define a theme.

The page settings schema is defined in this interface. It includes general settings as well as React components to be rendered in specific parts of the UI.

Upstream Registry Instance

It is possible to configure another Open VSX instance as upstream. This means that API requests to extensions that are not found locally are forwarded to the upstream instance. With this mechanism, you can keep selected extensions in your own (private) instance and delagate to another (public) instance for all other extensions. For example, you can use https://open-vsx.org/ as upstream instance to enable access to all public extensions, but still point users to your private instance.

HTTPS Configuration

A reverse proxy can be used to make an Open VSX instance available over HTTPS. See the Open VSX HTTPS NGINX Configuration guide to learn how to configure NGINX as a reverse proxy.

Clone this wiki locally