Skip to content

Latest commit

 

History

History
482 lines (363 loc) · 18.7 KB

release-1.1.md

File metadata and controls

482 lines (363 loc) · 18.7 KB
Raw

EOEPCA System - Release 1.1

Release 1.1 is mostly a bugfix release, with some limited feature enhancements to 1.0.

Release 1.1 includes versions of the following building blocks:

  • Login Service
  • User Profile
  • Policy Enforcement Point (PEP)
  • Policy Decision Point (PDP)
  • Application Deployment & Execution Service (ADES)
  • Processor Development Environment (PDE)
  • Resource Catalogue
  • Data Access Services
  • Workspace

See the Release v1.1 Description for more details.

Changes Since v1.0

Summary of Fixes

Performance enhancements, refinements and fixes to the v1.0 features.

  • Resource Catalogue: database deployment switched to stateful set
  • Resource Catalogue: OpenSearch EO parameter detection fix
  • Resource Catalogue: OGC API Records: change items startindex to offset
  • Resource Catalogue: fixed compatibility with latest Werkzeug version
  • ADES: nodeselector fix for processing pods
  • ADES: k8s role creation fix for processing job namespaces
  • Policy Decision Point (PDP): fixed default authenticated validation for policies

New Features

  • Upgrade of supported Kubernetes version from 1.18 -> 1.22
  • Resource Catalogue implements OGC API Records virtual collections (pycsw RFC 10)
  • Resource Catalogue compatible with STAC API 1.0.0-rc1
  • Resource Catalogue supports custom database mappings
  • Resource Catalogue a Reference Implementation of OGC GeoRSS 1.0
  • Resource Catalogue: update based on latest CQL2 models
  • Data Access: support for STAC in the data access harvester
  • ADES: support for sub-workflows in application packages
  • ADES: support for application packages with multiple workflows
  • ADES: optimized method to parse the processing results
  • PDE: Fully containerised Processor Development Environment with authentication
  • Identity & Authorization support for external tools (e.g. QGIS)
  • Login Service: support for role attributes as requested by the Open Science Catalogue
  • Policy Decision Point (PDP): validation option based on Terms and Conditions
  • Policy Decision Point (PDP): endpoint for managing Terms and Conditions
  • Policy Enforcement Point (PEP): option for retrieving resource information by providing the protected URI

Known Issues

The following issues are known to affect the v1.1 release. The intention is to fix these issues as soon as possible and make a delta v1.1.1 release as soon as possible...

  • EOEPCA-621 - Data Access renderer performance issues with CREODIAS JPEG2000 files

Release 1.1 Scope

The release demonstrates the following capabilities:

  • User authentication:
    • Login with GitHub
    • Login with ESA Commercial Operator Identity Hub (COIH)
    • Login with username/password
  • Client Registration
    • Dynamic client registration via SCIM endpoint
  • Authorisation
    • Dynamic Resource Registration
      Resource servers dynamic registration of resources
    • Resource protection
      Enforcing a policy in which resources are owned and protected accordingly
    • Policy-based resource protection
      Enforcing policy based upon policy rules maintained in the PDP
  • Processing Capabilities (ADES resource server)
    • OGC WPS 2.0 and OGC API Processes interfaces
    • Secure protected resource server, with access policy enforcement via PEP
    • List available processes
    • Deploy process (docker container with CWL application package)
    • Execute process (create job)
    • Get job status
    • Data stage-in via STAC/OpenSearch catalogue reference
    • Data stage-out to S3 bucket
    • Undeploy process
    • Integration of Calrissian CWL Workflow engine
      Provides native Kubernetes integration and out-of-the-box support for a variety of execution patterns - such fan-in, fan-out, etc.
    • Dedicated user 'context' within ADES service
  • Processor Development Environment (PDE)
    • JupyterHub for multi-user sessions
    • JupyterHub integrated with Login Service for user authentication
    • JupyterLab for user PDE instance
    • Jupyter Notebooks for interactive analysis
    • Theia IDE to develop using an integrated development environment
    • Tools for application package testing
  • Resource Catalogue
    • Implements ISO Metadata Application Profile 1.0.0
    • Support for ISO-19115-1 and ISO-19115-2
    • OGC CSW 3.0.0 and 2.0.2 interfaces
      • Certified OGC Compliant and OGC Reference Implementation for both CSW 2.0.2 and CSW 3.0.0
      • Harvesting support for WMS, WFS, WCS, WPS, WAF, CSW, SOS
      • Federated catalogue distributed searching
    • OGC API Records
    • STAC (SpatioTemporal Asset Catalog)
    • OpenSearch
      • OGC OpenSearch Geo and Time Extensions
      • OGC OpenSearch EO Extensions
  • Data Access Service
    • OGC WMS 1.1 - 1.3 interfaces
    • OGC WMTS 1.0 interfaces with automatic caching
    • OGC WCS 2.0 interfaces with EO Application Profile
    • OGC OpenSearch with EO, Geo and Time Extensions
    • Workspace management API
    • Dataset registration API
    • Registration schemes for Sentinel-2 L1C/L2A data in Data Access Service and Ressource Catalogue
  • Data Harvesting
    • Population of resource catalogue and data-access services from external data offerings
    • Sources supported include: file-system, search service (e.g. OpenSearch), catalog file
    • Filters, e.g. time, bbox, collection
    • Post-processing to adjust harvested results, e.g. for completion of missing metadata
    • Harvested results are transformed to STAC items for registration
  • End-to-end Processing Execution
    • Authenticated user accessing protected ADES endpoints
    • Dynamic creation of ADES user context with dynamic resource protection
    • Processing inputs discovered in Resource Catalogue
    • Processing inputs accessed via S3 (e.g. CREODIAS eodata)
    • Processing stage-in using STAC file to describe inputs
    • Processing execution on ADES
    • Processing stage-in using STAC file to describe inputs
    • Processing stage-out to S3 bucket
    • Processing stage-out using STAC file to describe outputs
    • Secure interfacing between ADES and user's protected Workspace
      ADES client -> Get Workspace Details / (De-)Register Resources
  • User Workspace
    • Secure protection of user-owned resources, with access policy enforcement via PEP
      • User-specific S3 bucket for resource storage
      • User-specific Resource Catalogue
      • User-specific Data Access Services
    • Secure protected management interface (workspace-api), with access policy enforcement via PEP
    • Management functions via REST API...
      • Create workspace (admin)
      • Get workspace details (user)
      • Delete workspace (admin)
      • Patch workspace (admin)
      • Redeploy workspace (admin)
      • Register resources (user)
      • Deregister resources (user)
  • Sample application: s-expression for EO product band math
    Three application packages based-upon s-expression:
    • App s-expression
    • App Water Mask
    • App NVDI
  • Sample application: Normalized Hotspot Indices

Building Blocks

This section identifies the version of the building blocks components comprising this release, and provides links for further information. For each, we include an 'Example' deployment configuration using a flux HelmRelease resource - these must be adapted for individual deployments.

Alternatvely, for deployment advice, see the Deployment Guide which provides full system deployment descriptions, examples and supporting scripts.

User Management

Login Service

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • Gluu Server:
    • gluufederation/config-init:4.1.1_02
    • gluufederation/oxauth:4.1.1_03
    • gluufederation/oxtrust:4.1.1_02
    • gluufederation/wrends:4.1.1_01

Resource Guard

The Resource Guard acts as an umbrella for the protection of resource servers via the pep-engine (Policy Enforcement Point) and uma-user-agent (UMA User Agent) components.

Resource Guard:

Policy Enforcement Point (PEP)

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • mongo (latest)

UMA User Agent

Resources

Resources to support deployment and configuration...

Containers

Policy Decision Point (PDP)

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • mongo (latest)

User Profile

Resources

Resources to support deployment and configuration...

Containers

Processing and Chaining

ADES

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • Stage-in: terradue/stars:1.0.0-beta.11
  • Stage-out: terradue/stars:1.0.0-beta.11

Processor Development Environment (PDE)

Containers

Sample Applications

Sample application packages for deployment and execution on the ADES:

Resource Management

Resource Catalogue

Resources

Resources to support deployment and configuration...

Containers

  • pycsw (version eoepca-1.1.0)
    • Image: geopython/pycsw:eoepca-1.1.0

Additional container images:

  • Database: postgis/postgis:12-3.1

Data Access Services

Resources

Resources to support deployment and configuration...

Containers

Additional container images:

  • Cache: registry.gitlab.eox.at/vs/cache:release-2.0.9
  • Client: registry.gitlab.eox.at/vs/client:release-2.0.18
  • Database: bitnami/postgresql:11.13.0-debian-10-r40
  • Redis: bitnami/redis:6.0.8-debian-10-r0
  • Scheduler: registry.gitlab.eox.at/vs/scheduler:release-2.0.2

Workspace

Resources

Resources to support deployment and configuration...

Containers

Bucket Operator

Resources

Resources to support deployment and configuration...

Containers