CHANGELOG.yaml

meta:
  changes: a stack of externally observable changes. Newer changes appear at the top
  title: a short description of the change
  issues: a list of ticket URLs (GitHub, JIRA, ...) related to the change
  upgrade: a list of actions required to upgrade to a version that includes this change
  It may be any combination of the following values:
    - deploy: (requires `make deploy`)
    - reindex: (requires `make reindex`)
    - terraform: (requires `make terraform`)
    - requirements: (requires `pip -Ur requirements.dev.txt`)
    - clients: the service API changed in a way that requires clients to be updated
    - environment: An environment variable was renamed, added or deleted. The `environment` file in personal
                   deployments may need to be updated

changes:

  - title: Add non-fetch variant of DSS proxy; add server-side wait
    issues:
      - https://github.com/DataBiosphere/azul/issues/778
    upgrade:
      - clients
    notes: The added `/dss/files` endpoint has the same semantics as the corresponding `/fetch/dss/files` endpoint but
           it sends actual 301 and 302 status codes, as opposed to tunneling them through a 200 response with a JSON
           body. Both endpoints support a `&wait=1` parameter to enable server-side waiting. The endpoint will wait as
           long as it can, and then return a response with Retry-After set to the remainder. This degrades much more
           gracefully on clients that don't honor Retry-After, preventing them from quickly exhausting their maximum
           number of redirects to follow.

  - title: "Use ontology terms where applicable"
    issues:
      - https://github.com/humancellatlas/metadata-api/issues/31
    upgrade:
      - requirements
      - reindex

  - title: "Fix: SQS trigger initially disabled"
    issues:
      - https://github.com/DataBiosphere/azul/issues/335
    upgrade:
      - requirements

  - title: Prioritize `organ` and `organ_part` fields from organoids over specimens
    issues:
      - https://github.com/DataBiosphere/azul/issues/695
      - https://github.com/humancellatlas/data-browser/issues/302
    upgrade:
      - reindex

  - title: Migrate monitoring resources from HumanCellAtlas/dcp-monitoring to Azul
    issues:
      - https://github.com/DataBiosphere/azul/issues/622
      - https://github.com/DataBiosphere/azul/issues/623
    upgrade:
      - terraform
      - environment
    notes: Migrated monitoring resources defined in HumanCellAtlas/dcp-monitoring to Azul, enabling Azul to manage
           them. Additions include Route53 health check endpoints for Azul and Data Browser as well as the ability to
           generate and publish a dashboard definition to Grafana. It can only be published if the `azul_grafana_user`,
           `azul_grafana_password` and `azul_grafana_endpoint` variables are set. Management of all monitoring
           resources including the Grafana dashboard can be enabled globally by setting `AZUL_ENABLE_MONITORING` to 1.
           Typically this is only the case for main deployments.

           The `AZUL_ENABLE_CLOUDWATCH_ALARMS` variable was renamed to `AZUL_ENABLE_MONITORING`. If your personal
           deployment uses `AZUL_ENABLE_CLOUDWATCH_ALARMS`, you may need to update your personal deployment's
           `environment` file accordingly.

  - title: Require wheel 0.32.3 to prevent corrupted deployments
    issues:
      - https://github.com/DataBiosphere/azul/issues/340
    upgrade:
      - requirements
    notes: Before reinstalling your requirements, delete your virtual environment. Then, use the `python -m venv`
           command instead of `virtualenv` to create your virtual environment.

  - title: Add BDBag as an alternative manifest format
    issues:
      - https://github.com/DataBiosphere/azul/issues/604
    upgrade:
      - deploy
      - requirements
      - clients
      - reindex
    notes: The query parameter `format` has been added to the '/repository/files/export' endpoint. Possible values are
           `tsv` (default, retains previous functionality) and `bdbag`. The latter creates a manifest containing the
           information necessary for importing samples from a DSS instance into Broad's Terra analysis platform. The
           manifest data will be packaged in a BDBag and persisted as a compressed ZIP file in an S3 bucket.

  - title: Include content disposition header in manifest responses
    issues:
      - https://github.com/DataBiosphere/azul/issues/655
    upgrade:
      - deploy
    notes: All manifest responses include a content disposition header that specifies the file name as
           `hca-manifest-<UUID>.tsv`.

  - title: Improved health check endpoints include indexing progress information
    issues:
      - https://github.com/databiosphere/azul/issues/624
      - https://github.com/databiosphere/azul/issues/615
      - https://github.com/databiosphere/azul/issues/548
    upgrade:
      - deploy
      - clients
    notes: The health check endpoint (`/health`) of either lambda returns a 503 status code if the lambda (or any of the
           resources used by it) is unavailable (down). Both `/health` endpoints return the same response except that
           the service lambda's `/health` response includes the status of the indexer lambda and vice versa. A `/health`
           response includes the number of bundles and documents waiting to be indexed (`unindexed_bundles` and
           `unindexed_documents`). These fields can be used in the Data Browser for visualizing indexing progress.

  - title: Include content disposition header in the final /fetch/dss/files response
    issues:
      - https://github.com/DataBiosphere/azul/issues/660
    upgrade:
      - deploy
    notes: If `&fileName=...` is passed to `/fetch/dss/files` the response header will include that file name.
           Otherwise the header will specify the file's UUID as the file name.

  - title: Personal deployment should index a limited subset of bundles
    issues:
      - https://github.com/DataBiosphere/azul/issues/646
    upgrade:
      - environment
      - reindex
    notes: Please append `_set azul_dss_query_prefix '42'` to deployments/*.local/environment.

  - title: Add GA4GH Data Repository Service GET Method
    issues:
      - https://github.com/DataBiosphere/azul/issues/638
      - https://github.com/DataBiosphere/azul/pull/642
    upgrade:
      - deploy
      - clients
    notes: >
        Adds a Data Repository Service (DRS) endpoint at `ga4gh/dos/v1/dataobjects`.
        Implements the `GetDataObject` method, a subset of the DRS API.

  - title: Add endpoint to shorten URLs
    issues:
      - https://github.com/databiosphere/azul/issues/66
    upgrade:
      - terraform
      - deploy
      - clients
      - environment
    notes: >
           The `/url` endpoint was added to return a URL that redirects to a given long URL.
           Documentation: /lambdas/service/app.py#L701
           AZUL_URL_REDIRECT_BASE_DOMAIN_NAME and AZUL_URL_REDIRECT_FULL_DOMAIN_NAME should be copied from the example
           deployment environment to local deployments

  - title: Emulate HTTP redirects and retry-after in DSS file proxy
    issues:
    - https://github.com/DataBiosphere/azul/issues/551
    upgrade:
    - deploy
    - client
    notes: Refer to docstring for service API handler `/fetch/dss/files/{uuid}`.
           Note that the DSS file proxy endpoint changed from `/dss/files/{uuid}` to `/fetch/dss/files/{uuid}`

  - title: Make the manifest endpoint dual mode (standard HTTP and lifted 200)
    issues:
      - https://github.com/DataBiosphere/azul/issues/567
    upgrade:
      - deploy
    notes: The previous `/manifest/files` endpoint behaviour has been moved to `/fetch/manifest/files`.
           `/fetch/manifest/files` will return a 200 response with redirection information in the body.
           `/manifest/files` has been changed to return either a 301 or 302 response with a `Location`
           header to allow the client to automatically redirect.  This change allows different clients
           to be able to choose the type of response.  e.g. Data browser can handle redirects in Javascript
           and matrix service can automatically be redirected.

  - title: Use step functions to produce manifest
    issues:
      - https://github.com/DataBiosphere/azul/issues/546
      - https://github.com/humancellatlas/data-browser/issues/372
      - https://github.com/databiosphere/azul/issues/566
    upgrade:
      - terraform
      - deploy
      - clients
    notes: The manifest download endpoint should be changed to `/manifest/files` as described here
           https://github.com/DataBiosphere/azul/blob/8d8fbac6474868cc7ebaebc6a726c427fad5c14b/lambdas/service/app.py#L543
           This increases the time limmit for manifest generation but also requires clents to make changes to handle
           the download.

  - title: Replace `process` inner entity with `protocol`
    issues:
    - https://github.com/databiosphere/azul/issues/540
    upgrade:
    - deploy
    - reindex
    notes: Previously one process inner entity was created for every pair of protocol and process metadata entity.
           With this change only protocol metadata entities are indexed, the inner entity is renamed from process
           to protocol and unused fields have been removed. Only the fields instrumentManufacturerModel and
           libraryConstructionApproach are being indexed. This change also removes the grouping of the
           process/protocol inner entities by the libraryConstructionApproach field.

  - title: Add CloudWatch alarms on certain ES metrics to main deployments
    issues:
      - https://github.com/DataBiosphere/azul/issues/332
    upgrade:
      - terraform
    notes: Alarms will only be created for the main deployments. Personal deployments are not affected.

  - title: "Fix: file type summary is truncated to ten entries"
    issues:
      - https://github.com/DataBiosphere/azul/issues/535
      - https://github.com/HumanCellAtlas/data-browser/issues/318
    upgrade:
      - deploy

  - title: Remove protocol facet from service response
    issues:
      - https://github.com/DataBiosphere/azul/issues/516
      - https://github.com/databiosphere/azul/issues/540
      - https://github.com/HumanCellAtlas/data-browser/issues/320
    upgrade:
      - deploy

  - title: Removed `disease` extraction from `donor_organism`
    issues:
      - https://github.com/DataBiosphere/azul/issues/527
      - https://github.com/HumanCellAtlas/data-browser/issues/335
    upgrade:
      - deploy
      - reindex

  - title: Proxy requests to DSS /files, enforcing Retry-After on browser's behalf
    issues:
      - https://github.com/DataBiosphere/azul/issues/531
      - https://github.com/humancellatlas/data-browser/issues/331
      - https://github.com/humancellatlas/data-browser/issues/342
    upgrade:
      - requirements
      - deploy
    notes: This adds a `/dss/files/{uuid}` endpoint and changes `/repository/files` so that its `hits[].files[].url`
           response field points to that new endpoint (it pointed directly to the DSS before this change). The new
           endpoint forwards the request to the DSS. When the DSS responds with a 30x status code and a `Retry-After`
           header, the service sleeps for a certain amount of time before forwarding the DSS response back to the
           client. Clients don't need to be updated as long as they use `hits[].files[].url` for direct downloads.

  - title: Extract preservation_method instead of storage_method
    issues:
      - https://github.com/DataBiosphere/azul/issues/503
      - https://github.com/HumanCellAtlas/data-browser/issues/308
    upgrade:
      - requirements
      - deploy
      - reindex
      - clients
    notes: The `hits[*].specimens.storageMethod` response field was renamed to `preservationMethod`, the corresponding
           facet was already called `preservationMethod` so no change was required there.

  - title: Add CHANGELOG.yaml and expose the top ten entries via /version endpoint
    upgrade:
      - deploy
      - requirements