▶ [minor]
This version removes the unused deployment configuration variable queue.use_cloud_mirror and queue.public_artifact_ec2_proxies. Neither served any useful purpose, and it is unlikely that either value appears in any deployment configuration.
▶ [patch] #4125
Workerpools now correctly understand the reregistrationTimeout option.
▶ [MAJOR] #3773
Support for superseding has been removed. See the linked issue for the detailed reasoning. While workers still allow supersederUrl in payloads, it has no effect. Older workers running with newer services that try to supersede tasks will encounter errors. No known instances of superseding exist.
▶ [MAJOR] #4123
The taskcluster-client-web library no longer implements OIDCCredentialAgent. This agent interfaced with a login.taskclutser.net service that no longer exists.
▶ [MAJOR] #3604 The notify service no longer supports irc notifications. IRC is declining in popularity and no known deployments of Taskcluster support this functionality, but it is nonetheless considered a breaking API change.
▶ [minor] #4050
The queue has a new artifact type, link, allowing links between artifacts on the same task.
▶ [patch] #4057 All clients (JS, Python, Go, Web, Shell) now fail when an API method results in a redirect, rather than following that redirect. The API methods that return redirects are those related to Taskcluster artifacts, and these methods must be accessed by building and fetching a signed URL.
▶ [patch] #2721
Taskcluster-proxy now correctly proxies "non-canonical" URLs, such as those containing // or urlencoded values.
▶ [patch] #3878 The Taskcluster UI now handles artifacts better, avoiding huge URLs that expire quickly.
▶ [patch] #3983 The UI will no longer fail when viewing a task with dependencies that have expired.
▶ [patch] #4199
The sift dependency has been updated again, to a version that does not cause #4061.
▶ [patch] #1064
The taskcluster command now parses errors from the API, and does not show the command usage when an error occurs.
▶ [patch] #3758
The taskcluster command will now display a warning after a short delay if it is expecting a request payload on stdin.
▶ [minor] #3578
The queue service now uses taskQueueId internally instead of the pair provisionerId/workerType for tasks.
▶ [patch] #3894 Postgres errors now include a Sentry fingerprint to help distinguish them in error reports.
▶ Additional changes not described here: #2398, #2875, #3466, #3665, #3739, #3751, #3888, #4072, #4125, #4209, #3718.
57 Renovate updates
- Update dependency newrelic to v7.1.0 (2cb90683e)
- Update Node.js to v14.15.4 (bd0d9a57a)
- Update dependency @slack/web-api to v5.15.0 (fea65786f)
- Update dependency acorn-walk to v8.0.1 (1d857fc33)
- Update dependency utf-8-validate to v5.0.4 (cbbf60248)
- Update dependency koa to v2.13.1 (cd17fccb9)
- Update dependency googleapis to v66 (b6bd1a987)
- Update sentry monorepo to v5.29.2 (817a223d2)
- Update mui monorepo (c4f1f8c9c)
- Update dependency js-yaml to v4 (b8509c2a4)
- Update dependency bufferutil to v4.0.3 (f691ab48e)
- Update dependency eslint to v7.17.0 (7c79a038f)
- Update mdx monorepo to v1.6.22 (d236a70c6)
- Update module yaml to v2.4.0 (7e637237c)
- Update dependency prismjs to v1.23.0 (dc1886a2e)
- Update module Microsoft/go-winio to v0.4.16 (f562e47b3)
- Update dependency webpack-cli to v4.3.1 (d6b462978)
- Update dependency react-window to v1.8.6 (f3a183920)
- Update dependency react-virtualized to v9.22.3 (a481007f1)
- Update dependency react-ga to v3.3.0 (b3ddb6bd7)
- Update dependency codemirror to v5.59.1 (7bb0feb02)
- Update dependency @azure/arm-network to v23.1.0 (46a8dccc2)
- Update dependency c8 to v7.4.0 (466c3e6f7)
- Update dependency query-string to v6.13.8 (43170db8f)
- Update dependency tar-stream to v2.2.0 (a017d4d28)
- Update dependency ws to v7.4.2 (3dc4ab697)
- Update dependency sanitize-html to v2.3.0 (ff7cd00b0)
- Update dependency webpack-dev-server to v3.11.1 (423cf40ed)
- Update dependency tar-fs to v2.1.1 (28895e6cd)
- Update dependency webpack-cli to v4.3.0 (7211360c8)
- Update dependency utf-8-validate to v5.0.3 (d8d3803ec)
- Update dependency title-case to v3.0.3 (556ed50b6)
- Update dependency taskcluster-client to v39.2.0 (14a427606)
- Update dependency jwks-rsa to v1.12.1 (2d29ca170)
- Update dependency uuid to v8.3.2 (80911d9b4)
- Update neutrino monorepo to v9.5.0 (40b45edc8)
- Update dependency upper-case to v2.0.2 (7d5e79e58)
- Update dependency snake-case to v3.0.4 (c1de6493d)
- Update dependency query-string to v6.13.7 (19d86aa12)
- Update dependency pg to v8.5.1 (baa6a87fc)
- Update dependency highlight.js to v10.5.0 (9d1d25892)
- Update dependency apollo-server-express to v2.19.1 (56f4a7541)
- Update dependency @babel/plugin-proposal-decorators to v7.12.12 (a8b891eb3)
- Update dependency param-case to v3.0.4 (3dac59740)
- Update dependency nodemailer to v6.4.17 (501d01a1c)
- Update dependency nock to v13.0.5 (ad95fa052)
- Update dependency builtin-modules to v3.2.0 (e3c12a224)
- Update dependency mime to v2.4.7 (cd1f0615f)
- Update dependency matrix-js-sdk to v9.4.1 (2c5af7952)
- Update dependency markdown-it to v12.0.4 (078c57cd9)
- Update dependency codemirror to v5.59.0 (da7a9bed2)
- Update babel monorepo (7b34e84c0)
- Update dependency open-editor to v3 (146729258)
- Update dependency eslint to v7.16.0 (977f6bd49)
- Update dependency marked to v1.2.7 (605aae3d8)
- Update dependency hashids to v2.2.8 (0a9dc3b67)
- Update Node.js to v14.15.3 (edd186cab)
▶ [patch] This version fixes an error where a worker pool with an invalid providerId would cause all worker provisioning to cease.
▶ [minor] #3542 Docker-worker no longer supports VNC access to interactive tasks. This support has been broken for ages and unused.
▶ [patch]
The taskcluster-client-web library client classes now have a buildSignedUrlSync method.
▶ [patch] #4056 The taskcluster-proxy no longer follows redirects. In practice, this is only an issue when calling the artifact-related API methods that return a redirect to the artifact content. The proxy will now return the redirect response unchanged.
▶ [minor] #3578
The tasks table uses task_queue_id instead of separate provisioner_id/worker_type to identify task queues.
This change is applied through an online migration process.
▶ Additional change not described here: #3940.
5 Renovate updates
- Update Node.js to v14.15.2 (8689b010a)
- Update dependency hashids to v2.2.3 (7e4eec9db)
- Update dependency commander to v6.2.1 (beef8ecea)
- Update dependency newrelic to v7.0.2 (2068dbca1)
- Update dependency marked to v1.2.6 (7b44747e4)
▶ [patch] The octokit throttling plugin has been removed in this release. We did not appear to understand its assumptions. It will probably come back later once we understand it better.
▶ Additional changes not described here: #3892, #4012.
1 Renovate updates
- Update dependency sinon to v9.2.2 (0dc9ff6f3)
▶ [patch] #4034 The queue's artifact expiration crontask now uses a much more efficient query and should be able to keep up with the load.
▶ [patch] #3797 A race condition in github checks updates has been resolved
▶ [patch] #4064 Taskcluster services and docker-worker now use Node 14, the current LTS version.
▶ Additional changes not described here: #2981, #4100.
▶ [patch] #4059 Fixed an issue fetching GitHub metadata when using a Taskcluster instance without the anonymous role.
This presented as unexpected 'Failed to get your artifact.' errors.
▶ [minor] #4006
The takscluster-client-web library is no longer installable from a <script> tag.
Instead, it should be incorporated into the build process of the consuming application, like any other library.
▶ [patch] Improved error messages related to fetching artifacts for GitHub checks.
▶ [patch] #4061 This version fixes an issue with the "actions" button not appearing for task groups.
▶ [patch] #3939
The object service now supports uploadId in the upload process.
▶ [patch] #4074 We now use github's library for generating app jwt tokens instead of making our own tokens
▶ Additional changes not described here: #3951, #3999, #4036.
▶ [patch] #3901 Fixed a bug where signing public S3 artifacts would result in Forbidden errors on the task and task group views.
▶ [patch] #3867
Taskcluster-Github should now function correctly in a deployment with no scopes in the anonymous role.
If you have a locked-down deployment without allowing public artifacts fetching in your anonymous role, you must add
queue:get-artifact:public/github/customCheckRunText.md and queue:get-artifact:public/github/customCheckRunAnnotations.json
to the scopes of your task to avoid an error comment being added to your
commits. Note that this will change if you choose a custom artifact name (see custom artifact docs for more)
▶ [MAJOR] #3713
This version introduces a new, in-development object service. It is currently configured for a default replica count of 0, meaning that it will not run, and this is the recommended configuration. However, it will nonetheless require configuration of a new database user (<prefix>_object).
▶ [minor] #3669 The Azure worker-manager takes additional steps to verify the identity proof during worker registration. The identify proof is the output of the attested data API, which includes details about the worker and is signed by the Azure platform.
Previously, the worker-manager checked that the message signer was issued by one of four published intermediate certificates issued by a single root CA. Azure is planning to expand to five more root CAs (see Azure TLS certificate changes for details). The worker-manager now downloads an unknown intermediate certificate, verifies that it was issued by a known root CAs, and adds it to the list of trusted certificates. The 4 legacy intermediate certificates, still in use in Azure as of November 2020, are pre-loaded as trusted certificates.
The worker manager now verifies that the message signer is for
metadata.azure.com or a subdomain. This is true for any workers in the
Azure public cloud, but not the sovereign clouds like azure.us.
One of the new root CAs uses Elliptic Curve Cryptography (ECC) instead of RSA. The Azure worker-manager doesn't support this or other ECC certificates. This is tracked in issue #3923.
There is no performance change expected until Azure ships the TLS certificate
changes, planned by February 15, 2021. When new intermediate certificates are
used, there will be up to a 5 second delay on worker registration while the new
certificate is downloaded for the first time. A new manager log entry,
registration-new-intermediate-certificate, is emitted after a successful
download and verification, and includes the certificate details.
▶ [patch] #3899 Docker-worker now decompresses downloaded images when they have a compressed content-encoding, as artifacts produced by docker-worker now have.
▶ [patch] #3637 Taskcluster-Github should now avoid spamming an identical comment many times in certain situations.
▶ [patch] #3982 The quickstart now correctly shows whether the GitHub integration is enabled for a repository.
▶ [patch] #3578
There are two new API methods for the queue service: listTaskQueues and getTaskQueue
▶ [minor] #3578
The queue service now uses taskQueueId internally, instead of provisionerId/workerType, for worker info
purposes (provisioners, worker types and workers).
The queue_provisioners table is dropped and the queue_worker_types table is renamed to task_queues.
▶ [patch] #3832 Octokit now uses github's own retry/rate-limit plugins instead of our own.
▶ Additional changes not described here: #3712, #3715, #3717, #3719, #3808, #3881, #3898, #3917, #3935, #3937, #3954, #3986, #4009.
▶ [patch] #3906 Creating comments on github is fixed in this release
▶ [patch] #3903
Scopes are now expanded in between using a certificate's scopes and checking authorizedScopes
as well.
▶ [patch] #3908 E-mail and Slack notifications should now correctly link to the group when the group ID does not match the task ID.
▶ [patch] #3874 The notify service now has enough scopes to handle notifications on Taskcluster instances without the anonymous role.
▶ [patch] #3884
Clients created with third-party sign-in (e.g., taskcluster signin) will no longer be disabled if they contain assume:anonymous or scopes in that role.
▶ [patch] #3899
Docker-worker now skips gzipping artifacts with an .lz4 extension, in addition to the existing list of extensions.
▶ [patch] #3873
The /provisioners/<worker-type> view now works correctly, fixing the error about reading property replace of null.
▶ Additional change not described here: #3837.
▶ [patch]
Setting a node DEBUG env var via the debug field of service configs is supported again.
If left unset it will default to ''. Example:
auth:
debug: '*'▶ [patch] #3865 Livelog TLS support is now functional.
▶ [patch] #3851
The GitHub quickstart tool now generates correct .taskcluster.yml files, among other bugfixes.
▶ [patch] #3836 The web UI no longer fails with "ext.certificate.expiry < now".
▶ [patch] #3831 This version fixes an issue introduced in v38.0.0 which would cause the log viewer to display 401 errors.
▶ [patch]
Config types of env:list now generate the correct type in helm schemas.
▶ [patch] Fix one usage of Octokit in release machinery to fix releases
▶ [patch] #3843 Two bugs were fixed that together made it so that tasks could not use indexed images.
First is that docker-worker now correctly uses the task's credentials rather than
its own to query the index.
Second is that scopes are now expanded prior to limiting them with authorizedScopes
in addition to afterward.
▶ [patch] bug 3759 As of this version, the DB upgrade process correctly checks access rights and table structures of the Postgres database.
▶ [patch] #3839 This version fixes an error ("e.artifacts is undefined") in the UI when viewing a task without credentials. It also improves error reporting from the UI in general.
▶ [patch] This version includes an explicit scope to allow the github service to list task groups. Without this, GitHub projects using the older status API will appear "running" forever.
▶ [patch] #3733
The database abstraction layer now supports "online" migrations, iterating over large tables without blocking production use of those tables. These migrations are entirely managed by the existing db:upgrade and db:downgrade functions, so this presents no change for deployers.
▶ Additional changes not described here: bug 1609067, #3721, #3731, #3732, #3804, #3807, #3827, #3834.
▶ [patch] This version fixes an error in docker-worker's release script that caused the 38.0.0 release to fail.
▶ Additional change not described here: #3738.
▶ [MAJOR] #3615
RFC 165 has been implemented, allowing for greater administrator control over "public" endpoints. Previously these were guarded by no scopes and could be accessed by anyone with no way to limit this. In this release all unauthenticated API calls are now granted the scope assume:anonymous. Additionally, most previously unprotected endpoints are now guarded by at least one scope, to enable the following:
- To maintain current behavior, some scopes will need to be granted to the
anonymousrole. Refer to `the anonymous role section in the docs. - To entirely lock down the cluster from anonymous access, do not grant any scopes to role
anonymous - Pick and choose specific "public" endpoints to make available to anonymous requests
Performance testing results (refer to taskcluster#3698 for more details):
- Auth service CPU has seen an increase of 0%-15%
- Auth service memory has seen no increase
▶ [MAJOR] #3015
Generic-worker no longer supports the --configure-for-{aws,gcp,azure} options. Instead, the expectation is that generic-worker will be started by worker-runner. While it remains possible to run generic-worker without worker-runner in a "static" configuration, cloud-based deployments using worker-manager now require worker-runner.
▶ [patch] #3791
The shell client (the taskcluster command) now correctly handles the case where no credentials are provided. In previous versions, if used to call a method which required credentials, this would result in an error: Bad Request: Bad attribute value: id. With the inclusion of RFC#165 in this release, this error would occur when calling any method. The short story is, if you see such errors, upgrade the shell client.
▶ [patch] #3463 This release fixes a bug that may occur when a new task is quickly inserted twice into the index service. When the bug is triggered, one of the insert calls would fail with a server error. With this fix, the UNIQUE_VIOLATION error is caught, and the previously failed insert will update the task if the rank is higher. This bug was first seen in v37.3.0
▶ [patch] #3767
This version adjusts the Python client requirements to avoid aiohttp==3.7.0, which has a serious bug preventing use of HTTPS.
▶ [patch] #3502
A bug where authenticateHawk calls would occasionally return an invalid response has been fixed. This issue impacted
reliability but not security.
▶ [patch] #3748
The source for the gw-workers and occ-workers administrative tools has been removed. The gw-workers tool is now at https://github.com/taskcluster/community-tc-utils.
▶ Additional changes not described here: #3655, #3662, #3670, #3704, #3730, #3783, #3788, #3793.
▶ [minor] #3640
Notify routes can now include on-defined, on-pending and on-running.
on-any is now deprecated and there are two new alternatives:
on-transitionfor any state transition.on-resolvedfor terminal states (completed, failed and exception).
▶ [patch] taskcluster-web-server is now equipped with the anonymous role. This will allow it to assign the anonymous role to users who successfuly login.
▶ [minor] #3521
Taskcluster-proxy now adds a Content-Type header to proxied requests lacking one. While this behavior is not desirable, it matches the behavior of older versions and real tasks depend on it. A future version of Taskcluster will drop this behavior.
When this occurs, the worker will log a message containing the string "Adding missing Content-Type header". Use this logging to find tasks that fail to include the Content-Type header and adjust accordingly.
▶ Additional change not described here: #3679.
▶ [patch] #3659
Slack and Email notifications' Task Group URLs are now correct (containing /tasks).
▶ [patch] #3639
taskDefined messages will now always have an unscheduled status.
▶ [patch] #3631
Calling a JS Client constructor with no arguments works again -- assuming that any necessary configuration was passed to taskcluster.config(..).
▶ [minor] #3538
DB function get_workers is now deprecated.
▶ [patch] #3619
The tools/workerproto Go package is now available for external use, and its API is considered stable (in other words, breaking changes will result in a major version bump).
▶ Additional change not described here: #3591.
▶ [minor]
A new queue deployment configuration variable sign_public_artifact_urls has been added which enables AWS URL signing for all S3 artifacts when true.
▶ [minor] #3606
Slack support has been added to the notifications service. You can now
send notifications to Slack channels by using a
notify.slack.C123456.on-any style route, or by using the new /slack
API endpoint.
▶ [patch] #3588
Database URLs can now be specified in the configuration with ssl=authorized, in which case Taskcluster will validate the Postgres server's SSL/TLS certificate against trusted root CAs. It is unusual for databases to be deployed with such certificates. See the documentation for details.
▶ [patch]
The tutorial in the documentation has been updated and modified to offer better guidance for different deployments of Taskcluster. The ui.site_specific configuration has a new, optional tutorial_worker_pool_id property (documented here) defining a worker pool for use by readers of the tutorial.
▶ [patch] #3561 Bug fix: calls to workermanager.updateWorker for the static provider have been fixed.
▶ [patch] #3358 The "badge" SVGs provided by the GitHub service now render correctly instead of as black shapes.
▶ [patch] #3495 The web-based schema viewer now shows descriptions of each field.
▶ [minor] #3579 The purge-cache, built-in, and worker-manager services now use taskQueueId internally, instead of provisionerId/workerType.
▶ [patch] #3473 Docker-worker has been ugpraded to use a newer version of dockerode, and no longer directly uses dockerode-promise.
▶ Additional changes not described here: bug 1668111, #3035, #3210, #3287, #3543, #3544, #3599, #3525.
▶ [patch] #3513 Node has been upgraded to 12.18.4 to address CVE-2020-8201.
▶ [patch] #3501
The worker-manager expire-errors job now correctly runs the error expiration process.
▶ [minor] #3347
The Azure provider now accepts an ignoreFailedProvisioningStates property in its launch configs which will cause it to ignore ProvisioningState/failed/<code> states on VMs. This is specifically useful for ignoring OSProvisioningTimedOut when the Azure VM agent is not running.
▶ [patch] #3346 The Azure provider now looks only for well-understood failure-related states in the Azure API to determine when a worker has failed. In cases where these measures miss an event, (re)registrationTimeouts will terminate the worker.
▶ [patch] #3058
The worker-manager's Azure provider now more accurately tracks the state of workers, and will not mark a worker RUNNING until it has called registerWorker.
▶ Additional changes not described here: #3036, #3502, #3503.
▶ [patch] #3175 Taskcluster's Github integration has been updated to the new standard for webhooks detailed in this post
▶ [patch] The taskcluster-hooks-scheduler will no longer crash while trying to report errors firing hooks.
▶ [minor] #3189
The workerManager.removeWorker API method now works correctly for the static provisioner, and a new updateWorker API method supports modifying workers after they have been created.
▶ [patch] #3483
Faced with an error reclaiming a task, docker-worker will now correctly call reportException with reason internal-error.
▶ [patch] #3456
The workerManager.createWorker API method now correctly limits the workerGroup and workerId properties as described in the worker schema (38 characters, no dots).
▶ [minor] bug 1563191 generic-worker now logs the full task payload json schema if a task's payload fails json schema validation.
▶ [patch] #3355 The Taskcluster-GitHub service no longer throws errors on unknown pull-request actions in GitHub webhooks.
▶ [patch] #3464
Timestamps in the task status runs array are now formatted like all other timestamps in the Taskcluster API, without a trailing +00:00.
▶ [patch] #3354 This release handles error from malformed github check artifacts.
▶ Additional changes not described here: #3309, #3458.
▶ [MAJOR] #3216
The auth, github, hooks, index, and notify services no longer take Helm config <service>.azure_account_id, and auth no longer takes Helm config auth.azure_account_key, as these services no longer talk to Azure.
▶ [minor] #3216
The queue service no longer accepts the optional, and probably-unused, queue.azure_report_chance and queue.azure_report_threshold Helm configurations.
▶ [minor] #3168
The worker-manager now supports a scalingRatio that determines how much worker capacity to spawn per pending task.
The scalingRatio is a ratio of worker capacity to pending tasks - a ratio of 1.0 means that 1 capacity will be added for each pending task.
▶ [minor] #3033
The worker-manager updates the expires timestamp for AWS workers that are set to expire in less than a day.
Updating the expires timestamp is now handled in the worker-scanner scan() loop for all providers.
▶ [patch] bug 1637302 Docker-worker now allows configuring which artifacts it should compress on upload.
▶ [minor] bug 1623749 Docker-worker now allows features to be disabled in the worker config.
▶ [minor] bug 1623749 Docker-worker now allows scopes for devices and privileged containers to be per-pool, rather than global.
▶ [minor] #2973 Support docker images from tasks with only a docker v1.2 manifest.
▶ [minor] #1986
The maximum length of the hookGroupId and hookId identifiers is now 1000.
▶ [patch] #3366 A serious bug in dependency handling, introduced in v35.0.0, has been fixed. The issue occurred when a task on which more than 100 other tasks depend was resolved. In this case, some, but not all, of the dependent tasks would be marked pending.
▶ [patch] bug 1637302 Don't compress dmg files by default in docker worker.
▶ [patch] bug 1637302 Don't compress dmg or zst files by default in generic worker.
▶ [patch] #2992 Private artifacts are now accessable via the UI.
▶ [patch] #3398
This version upgrades JSON-e to 4.1.0, and in particular the $switch operator can now be used in hook task templates and in .taskcluster.yml files and everywhere else Taskcluster uses JSON-e.
▶ [patch] #3328
Database function compatbiility guarantees are now included in db/fns.md for reference by engineers writing database versions.
Takcluster-lib-entities has been removed from the codebase, as no entities-style tables remain.
▶ Additional changes not described here: #3178, #3334, #3337, #3342, #3344, #2910.
▶ [MAJOR] #2937
Github checks are now stored in a table called github_checks, and github integrations are now stored in a table called github_integrations. Both are accessed directly, rather than via taskcluster-lib-entities. This migration takes about 10 seconds for a million-row table.
▶ [MAJOR] #3216
The auth, github, hooks, index, and notify services no longer take Helm config <service>.azure_account_id, and auth no longer takes Helm config auth.azure_account_key, as these services no longer talk to Azure.
▶ [MAJOR] #3148 The tables in web-server are now all relational. The migration drops all data in these tables, which will have the effect of signing out all users and requiring them to sign in again. But it is a very quick upgrade.
Sign-ins will not work until the web-server service has been upgraded to this version (that is, sign-ins will not work during the time between the database upgrade and the services upgrade, nor if services are downgraded back to v35.0.0).
The web server service continues to honor web_server.azure_crypto_key, but now optionally takes an additional Helm variable web_server.db_crypto_keys as described in the deployment documentation
▶ [minor] #2933
The Queue service's workers, worker_types, and provisioners are now stored in a normal database table and access directly, rather than via taskcluster-lib-entities. If the queue_workers_entities table has many rows, this migration could take some time. Consider dropping all, or some, rows from the table before beginning the migration.
▶ [minor] #3083
The auth service's clients are now stored in the clients table and the service accesses that information directly, rather than via taskcluster-lib-entities. As the number of clients is small, this migration should be very fast.
▶ [minor] #2936 The hooks service now stores hooks and ancillary information about Pulse queues and hook history in normal database tables, without the use of taskcluster-lib-entities. This migration is quick.
The hooks service continues to honor hooks.azure_crypto_key, but now optionally takes an additional Helm variable hooks.db_crypto_keys as described in the deployment documentation
▶ [minor] #3216
The queue service no longer accepts the optional, and probably-unused, queue.azure_report_chance and queue.azure_report_threshold Helm configurations.
▶ [minor] #2931 The secrets service now stores its secrets in a normal table, without the use of taskcluster-lib-entities. The migration should be quick, as secrets are typically few in number (hundreds).
The secrets service continues to honor secrets.azure_crypto_key, but now optionally takes an additional Helm variable secrets.db_crypto_keys as described in the deployment documentation
▶ [patch] #3245
The taskcluster/websocktunnel and taskcluster/livelog docker images now include a leading v in their tags, e.g., taskcluster/websocktunnel:v36.0.0.
▶ [patch] A worker pool with no launch configs will no longer cause errors (although it will also not create any workers!)
▶ [patch] #3169
If workerTypeMetadata is given in a generic-worker worker pool definition, its contents will now be merged with the metadata from the provider and passed to generic-worker.
▶ [patch] bug 1654086 This version fixes a bug which would cause the hooks service to crash when sending error reports to denylisted addresses.
▶ [patch] bug 1645032 User IDs as received from Auth0 in the Mozilla-Auth0 login strategy are no longer suffixed with github usernames or firefox-accounts emails. In practice, such user IDs are unused.
▶ [patch] #3272
A mapping between DB and TC versions is now maintained automatically in db/versions/README.md.
▶ [patch] #3289
The DB schema is now documented in db/schema.md.
▶ [patch] #3276
The main branch of development on the Taskcluster repository is now named main.
▶ [patch] #2928 taskcluster-lib-postgres now allows calling stored functions with named arguments.
▶ Additional changes not described here: #3170, #3176, #3184, #3185, #3224, #3285, #3290, #3301.
▶ [patch] #2887 Generic-worker now supports reporting runtime errors to worker-manager via worker-runner.
▶ [MAJOR] #3148 The web-server service now stores Github access tokens in a dedicated table and accesses them directly, rather than via taskcluster-lib-entities. This upgrade drops existing tokens, meaning that users will need to sign in again after the upgrade is applied. This migration is very fast.
▶ [MAJOR] With this version, the auth, hooks, and secrets services no longer verify signatures on rows read from database tables. This is in preparation for a future version where these tables will no longer contain signatures.
▶ [minor] #2937
Github builds are now stored in a table called github_builds, and accessed directly rather than via taskcluster-lib-entities. This migration can process at least 40,000 rows in no more than a few seconds. For a table larger than that, deleting the table contents before running the migration is an option. This table backs the "status" and "badge" endpoints, so missing data is of minor consequence.
▶ [minor] #2938 The auth service's roles are now stored in a normal database table and accessed directly. This is a quick migration.
▶ [minor] #2935 The index service now uses its tables directly, rather than via taskcluster-lib-entities. This is step 2, a continuation of taskcluster#3141. Step 2 involved creating new DB functions and refactoring the service itself to use the new functions. The db upgrade should be very fast.
▶ [minor] #3112 The queue service now uses its artifact-related database tables directly, rather than via taskcluster-lib-entities.
▶ [minor] #2932 The queue service now uses its task- and task-group-related database tables directly, rather than via taskcluster-lib-entities.
▶ [minor] #3030 The worker manager's worker pool errors are now stored in a normal database table. This should be a small migration.
▶ [minor] #3240 Worker pool errors are now properly listable by workerPoolId.
▶ [patch] #3222
The persistent errors about missing function digest(text, unknown) logged by the database are now fixed.
▶ [patch] #3191
The task.extra.github.customCheckRun.annotationsArtifactName property is now correctly consulted for the name of the annotations artifact, as documented.
▶ [patch] The taskcluster-lib-postgres library now allows any Postgres collation that sorts ASCII characters correctly.
▶ Additional changes not described here: #3160, #3238.
▶ [MAJOR] #3112 Queue's artifacts table is upgraded to a normalized format. For deployments with many (millions) of artifacts, this migration will take too long to perform online, and should be performed in a scheduled downtime. Note that the "service migration" portion of the process is not included here, and the queue artifact code still uses entities-related functions to acces its data.
▶ [patch] bug 1637302 Docker-worker now correctly calculates artifacts hashes for chain-of-trust before compressing them.
▶ [MAJOR] #2935
The namespaces_entities and indexed_tasks_entities tables have now been
migrated to use relational tables. For deployments with many (millions) of
tasks, this migration will take too long to perform online, and should be performed in a scheduled downtime. Note that the "service migration" portion of the process is not included here, and the index code still uses entities-related functions to acces its data.
▶ [patch]
The db:upgrade and dev:db:upgrade commands can now take an optional database version to upgrade to, defaulting to the most recent version.
▶ Additional changes not described here: #3092, #3131.
▶ [MAJOR] #2934
Migrates Postgres Phase I table notify.denylisted_notification_entities to
Postgres Phase II table notify.denylisted_notifications.
▶ [patch] #3116
The db upgrade and downgrade scripts now verify that the default database collation is en_US.UTF8. No other collation is allowed.
Unfortunately, changing the default collation requires dumping and re-creating the database.
▶ Additional changes not described here: bug 1636193, #3093, #3147, bug 1635455.
▶ [patch] bug 1637302 Docker-worker now automatically gzips artifacts before uploading them. It sets content-encoding in the S3 headers so that most consumers should be able to transparently handle decompression.
▶ [MAJOR] #3012
An encrypted column "secret" has been added to the workers table. The
worker-manager service now requires an additional environment variable DB_CRYPTO_KEYS
to be set which is a JSON array where each element is an object of the form.
{
"id": "a unique identifier",
"algo": "aes-256",
"key": "32 bytes of base64 string"
}Note that for this upgrade it will only be an array of a single object.
▶ [patch] bug 1638921 Kubernetes cron tasks are now configured with concurrencyPolicy: Forbid, to prevent multiple pods of the same job from running concurrently.
▶ [patch] #3080
Docker-worker is now more careful to shut down only when it is idle and has not begun to claim a task, avoiding race conditions that could lead to claim-expired tasks.
▶ [patch] #3012 Worker runner can now re-register a worker with worker-manager, refreshing its credentials. This allows workers to run for an unlimited time, so long as they continue to check in with the worker manager periodically. Both docker-worker and generic-worker, as of this version, support this functionality. Older worker versions will simply terminate when their credentials expire.
▶ [patch] Docker-worker now includes an error message in the task log when uploading an artifact fails
▶ [patch] #2883
Endpoints that return worker pools now contain an existingCapacity field that contains the total
amount of capacity for the worker pool between all workers that are not stopped.
▶ [patch] #3004
Generic-worker now uses the task's credentials to fetch artifacts specified in the mounts property of the task's payload. This will allow use of private artifacts in mounts.
▶ [patch] #2882 Workerpools lists and views in the ui now show the amount of currently existing capacity is provided by the workers in the pool and the pending count of tasks.
▶ [minor] #3013
Github integration can now set annotations for check runs.
By default it will read public/github/customCheckRunAnnotations.json but it can be overridden by setting
task.extra.github.customCheckRun.annotationsArtifactName. The json will be passed along unmodified.
▶ Additional changes not described here: bug 1638921, #2887, #2890, #3021, #3067, #3079, #2962.
▶ [patch] Worker Manager now avoids scanning all the workers table in memory to avoid possible OOM issues.
▶ [patch] bug 1607605 Generic-worker now supports shutting down gracefully when instructed to do so by worker-runner, such as when a cloud VM is being terminated.
▶ [patch] bug 1639713
Tasks using the hostSharedMemory device capability will now properly mount /dev/shm from the host into the container.
▶ [minor] #2877
The wmworkers_entities table has now been migrated to use a relational table.
The new table is called workers. wmworkers_entities will get deleted.
▶ [patch] Release tasks now have access to taskcluster-proxy
▶ Additional change not described here: #2921.
▶ [patch] bug 1631824 The worker-manager azure provider now properly tracks and deletes all disks when a virtual machine has data disks created for it.
▶ [patch] A bug in the Azure provider which caused provisioning to fail when handling operations has been fixed.
▶ [patch]
Taskcluster services now include metadata at the top level of Fields for generic.* logging messages, rather than in meta or fields sub-properties.
▶ [patch] #2969
Docker-worker now only considers itself idle if its call to queue.claimWork returns no tasks. This prevents the situation where a very short afterIdleSeconds causes the worker to shut down while calling claimWork.
▶ [patch] #2925 Listing workers in the "stopping" state will no longer cause 500 errors.
▶ [patch] bug 1632929
Taskcluster-Github now uses a release event's target_commitish property instead of the tag property to determine the SHA of the released commit. This is important in cases where tags are created as part of the release-creation call, as GitHub sends the release event before the tag is created.
▶ [patch] bug 1636167 CI tasks are now generated in a decision task by https://hg.mozilla.org/ci/taskgraph
▶ Additional changes not described here: bug 1640267, #2827, #2890, #2912, #2913, #2951, #2952, bug 1634376.
▶ [patch] An incorrect use of a relative path caused sign-ins to fail in v30.0.1. This has been fixed.
▶ [patch]
Fix docker worker not working in the latest release of Taskcluster. It was
previously throwing taskVolumeBindings is not iterable.
▶ [patch] #2876 The purge cache UI view now allows filtering a search result by cache name.
▶ Additional change not described here: #2845.
▶ [patch] A typo causing index service not to start up in 30.0.0 is now fixed.
▶ [patch] bug 1638047
This release fixes a bug where the web UI opens the log viewer for any text/plain artifacts, which breaks for private artifacts. The web UI will now only use the log viewer for text/plain *.log files.
▶ [patch] bug 1587145 taskcluster-client-web now only builds a single umd asset. This asset is compatible with both cjs and esm.
▶ [minor]
Database version 11 removes the widgets table that was used to test Postgres deployment. It contains no useful data.
The hidden notify.updateWidgets API method, but this method was never meant to be used so this removal is not considered a breaking change.
▶ [patch] bug 1639913 Worker-manager now logs when a worker is removed, and includes debug logging of provisioning and scanning.
▶ [MAJOR] bug 1636321
The generic-worker configuration parameters livelogKey, livelogCertificate, livelogGETPort, livelogPUTPort, and livelogSecret are no longer needed and are prohibited in the worker's configuration.
▶ [minor] #2861 The unused and unmaintained docker-worker features balrogVPNProxy, balrogStagingVPNProxy, and relengAPIProxy have been removed.
▶ [patch] bug 1638370
Azure provider no longer has a race condition between registerWorker and checkWorker.
▶ [patch] Docker-worker will now fail early with a useful error message if the loopback audio or video devices are not available, but are configured.
▶ [patch]
The docker-worker version is now logged in the serviceContext.version property of its structured logs.
▶ [patch] bug 1627769 Worker lifecycle defaults are now being properly applied.
▶ [patch] #1061 In client-shell added flag --verbose/-v for getting log to stderr for all the commands.
▶ [patch] The docker-worker payload format is now available in Taskcluster's online documentation.
▶ [patch] #2844 All services are now invoked from the root of the monorepo directory.
▶ Additional changes not described here: bug 1636164, bug 1636174, #2822, #2838, #2844.
▶ [minor] bug 1638002
The Azure, AWS, and Google worker provisioners now use an instance's region or location as workerGroup, instead of the worker pool's providerId.
▶ [minor] #2811 The Queue schema now allows for ssh:// source urls.
▶ [patch] An issue with building external urls with traceId'd clients has been fixed
▶ Additional change not described here: bug 1637982.
No changes
▶ [patch] bug 1633582 Fixes an issue in the worker-manager google provider where improperly configured disk tagging caused worker creation to fail.
▶ [minor] bug 1619652
Taskcluster logs now include traceId and requestId fields on messages that have these in context.
A requestId is per http request and a traceId follows a request chain along as far as it goes so
for example a graphql request to web-server -> queue -> auth.authenticateHawk are all correlatable
as part of one trace.
As part of this change, by default in Kubernetes, requests between services are now routed directly using
Kubernetes dns service discovery. To disable this, you can set the top-level useKubernetesDnsServiceDiscovery
to false in your helm values.
▶ [patch] bug 1637104 The livelog, taskcluster-proxy, and websocktunnel Docker images now use statically-linked binaries, meaning they will not fail on startup.
▶ [patch] bug 1636189
The websocktunnel, livelog, and taskcluster-proxy images now have an /app/version.json as required by DockerFlow, and websocktunnel correctly services all three DockerFlow endpoints. In additional, all version.json files including that in the main taskcluster/taskcluster image now have a correct build URL.
▶ [patch] #2788 Docker-worker releases are now included in the assets on a Taskcluster release, with a well-documented format.
▶ [patch] #2739
Taskcluster-proxy assets, and a taskcluster/askcluster-proxy docker image, are now produced for every TC release.
▶ [patch] bug 1636163 docker-worker docs now show on docs website
▶ [patch] bug 1635897 Taskcluster-GitHub now correctly determines the sha for releases from signed tags.
▶ Additional changes not described here: bug 1561668, bug 1636165, #2783, #2808.
▶ [patch] bug 1636292 The bug in 29.4.0 which caused DB migration to fail given large WorkerPool table rows has been fixed with a patch to DB version 10.
▶ [patch] bug 1635985 Docker Worker code now lives in this repository instead of taskcluster/docker-worker
▶ [patch] bug 1631829 Fixes an issue where azure-provider wasn't properly tagging resources.
▶ [minor] bug 1630023 The worker manager's worker pools are now stored in a normal database table. This table is small, and the DB migration should complete in seconds.
▶ [patch] Fix missing db TypeError in purge-cache.
▶ [patch] bug 1633897 Remove outdated check for taskcluster.net when sending cookies. This was used back when the UI was hosted in heroku.
▶ Additional change not described here: bug 1633882.
▶ [minor] bug 1630019 The purge_cache service now uses normalized db tables
▶ [patch] bug 1633582 The worker-manager Google provider now labels worker disks with the same set of labels as VMs.
▶ [patch] #1536 taskcluster-client-web no longer shows the 'hawk is undefined' regression error.
▶ [patch] bug 1630023 DB version 8 introduces some utility functions that will be useful in migrating from (and downgrading to) tc-lib-entities-compatible tables.
▶ [patch] The morgan-debug logging for web services has been removed in favor of continued support of our api logging and iprepd logging in production
▶ Additional change not described here: bug 1633882.
▶ [patch] bug 1606006 Services that use ephemeral queues now use a different queue name on each connection. This avoids issues with RESOURCE-LOCKED from RabbitMQ.
▶ [minor] bug 1629807 Taskcluster login now includes a state token in the url search query during the login transaction to conform with the recommendations in rfc-261.
▶ [patch] bug 1631099 Taskcluster-GitHub now retries on 401 "Bad Credentials" errors from GitHub, as suggested by GitHub developers.
▶ [patch] bug 1633622
The taskcluster-client-web package now contains the build directory as expected.
▶ [patch] A dependency that was mistakenly thought to be unused has been added back
▶ [patch] bug 1627116 The worker manager AWS provider now tags EBS volumes created for EC2 instances with the same set of tags.
▶ [patch] bug 1631829 The worker-manager Azure provider now tags all worker related Azure resources with the set of standard tags.
▶ [patch] bug 1632325 release:publish tasks now save debug logs as artifacts
▶ [patch] bug 1631414 Worker-Runner is now properly documented in the Taskcluster documentation.
▶ Additional change not described here: #2681.
▶ [minor] bug 1551846
taskcluster-lib-app now includes endpoints /__version__, /__heartbeat__, and /__lbheartbeat__ to be compatible with Dockerflow requirements.
▶ [patch] bug 1631638 Overprovisioning alerts are now less spammy for small workerpool sizes
▶ [patch] #2562 The Websocktunnel repository has been moved into the monorepo, and websocktunnel is now released at the same time as the rest of the Taskcluster services, and with the same version number. Aside from a (large) change in version number, nothing else about websocktunnel has changed since v2.0.0.
▶ [patch] bug 1437952
The yarn backup:.. commands have been removed, as backups should now be done at the Postgres database level.
▶ [patch] bug 1628141
The default cpu and memory for each Kubernetes deployment are now set to better values based on experience at Mozilla.
▶ [patch] #2395 The deployment configuration now allows specification of some site-specific values. While these are optional, adding these values will help users to better navigate the documentation. See the deployment docs for information on the available values.
▶ [minor] bug 1540804
Config property publicIP of generic-worker workers is now optional. When not
provided, rdp into Windows workers will no longer be possible, Chain of Trust
environment reports will no longer include the public IP, and livelogs via
stateless dns server will no longer work (however this will not affect livelog
served over websocktunnel).
▶ [minor] #2647 The Taskcluster livelog tool has been merged into the Taskcluster monorepo, and will now be released in concert with the rest of Taskcluster. In the process of merging this tool, it was discovered that it handled HTTP Range requests incorrectly. On the assumption that this functionality was never used, it has been removed.
▶ [patch] bug 1591476 Worker-Runner now ignores any worker configuration in a cloud provider's user/meta/custom-data facility, instead using the configuration provided in response to the registerWorker REST API call. This functionality requires that the service deployment run at least Taskcluster v26.0.0.
▶ [patch] Worker-runner now gives better error messages when it does not have information such as the RootURL in its tagged data.
▶ [patch] bug 1516575 Worker-runner now protects itself and docker-worker from the Linux OOM killer
▶ [patch] bug 1629657 Workerpools are now a paginated list in the web ui.
▶ [minor] bug 1630113
Matrix integration now supports m.text, m.emote, and m.notice msgtypes. The default is
m.notice which was the only value supported previously.
▶ [patch] Make the error messages for custom checkrun text functionality clearer, so that the users don't have to read documentation.
▶ [patch] The Go implementation of the runner / worker protocol is now an internal library and not accessible from outside the Taskcluster repository.
▶ [patch]
The yarn dev:init command since 28.2.3 would create procs entries for write_docs and expireSentry that would cause yarn dev:apply to fail. That has been fixed, but such entries must be manually removed from dev-config.yml if they have already been added.
▶ [patch] #2465
The task for yarn test:meta was not failing properly in CI. This has been fixed, and failing meta checks have been resolved.
▶ Additional changes not described here: bug 1548036, bug 1619286, bug 1629168, bug 1630023, bug 1630124, #2268, #2631, #2637, #2534.
▶ [patch]
The db:upgrade and db:downgrade commands now correctly roll back on error.
▶ [patch] #2634
taskcluster-lib-entities .modify no longer reaches out to the db when the data
is not modified.
▶ [MAJOR] bug 1436478
The Taskcluster services now use a Postgres backend, instead of Azure Cables and Azure Containers. All data in Azure must be migrated to Postgres during a downtime using yarn importer:run, and this is planned for all known deployments. There should be no immediate user-visible impact from this change, aside from faster API responses, but it unlocks many planned improvements.
▶ [patch] #2615 Fix error showing when creating new client/role in the UI.
▶ [patch] bug 1525419
Generic worker tasks on Windows can now define environment variables that contain special characters ()%!^"<>&|. Previously they were not escaped.
▶ [patch]
Development environments now default to a lower per-pod CPU request, which should help reduce the compute cost of idle development environments. Run yarn dev:init to update these defaults for your dev environment.
▶ [patch] bug 1624602 Worker-runner is now more careful to read all output from the worker when the worker exits.
▶ [patch] bug 1552323 Fixes the bug: https://sentry.prod.mozaws.net/operations/taskcluster-community/issues/7766271
▶ [patch] Fix error in notify service (monitor is required)
▶ [patch] bug 1618333 Changelog entries now are categorized by the audience that they are useful for
▶ [patch] Now, if the worker process running in aws/gcp exits, it will be requested to worker-manager to terminate the instance.
▶ [patch] bug 1622943
The maximum value for a worker's lifecycle.reregistrationTimeout is now 30 days. Values greater than this cannot be represented in the worker's temporary credentials anyway.
▶ [minor] bug 1552323 Adds ability to customize checks output in taskcluster-github Checks feature. Apart from the bug mentioned, fixes the issue mozilla-mobile/fenix#6760
▶ [patch] #1389 Taskcluster UI nows offers a breadcrumbs view to easily jump back and forth when viewing indexes (/tasks/index/)
▶ [minor] bug 1616998 taskcluster-worker-runner has been renamed to worker-runner and its docs have been added to the reference section of the docs portal.
▶ [patch] #2522 Services that use a database now log information about that database, including connection pool counts and stored-function invocations.
▶ [patch] #2555 The azure-queue emulation library now omits expired messages from its counts. The visible effect is that pending counts for queues no longer include tasks past their deadline.
▶ [patch] #2553 The taskcluster-lib-azqueue library now returns "batches" of messages in the order they were inserted.
▶ Additional changes not described here: #1615, #2541.
▶ [minor] bug 1436478 Add a new library taskcluster-lib-entities that exposes the same API as azure-entities but uses postgres rather than azure for its database. Note that all of the services are still using azure-entities. Services will eventually switch to using this new library. Date to be decided.
▶ [minor] bug 1306494 Taskcluster UI now allow users to view the diff for scope changes (similar to the github write/preview functionality).
▶ [patch] #2292 Add a new library taskcluster-lib-azqueue that exposes the same API as the Azure Queue service but uses Postgres rather than Azure. Note that all of the services are still using Azure. Services will eventually switch to using this new library. Date to be decided.
▶ [patch] bug 1616931 Generic-worker now transmits its logs via taskcluster-worker-runner, in preparation for supporting arbitrary log destinations.
▶ [patch] bug 1621420 Prepare to update octokit dependency
▶ [patch] #2503 Some schemas in the Taskcluster documentation were not displayed with a "Cannot find .." error. This has been fixed.
▶ [patch] #2486 Taskcluster UI now allows users to add matrix rooms to the denylist addresses.
▶ [patch] Taskcluster deployments now support sending results to New Relic (optionally). See the deployment documentation for details.
▶ [patch] bug 1618991 The Go client now correctly returns an error when 500 responses are retried to exhaustion.
▶ [patch] #2498 The database upgrade command now checks roles and permissions attributes for database users.
▶ [patch] The linux-arm builds of generic-worker are now considered Tier-2, meaning that they are not tested in CI (but are still built). Testing is also disabled on Windows 10 / amd64 due to lack of capacity, but continues for Windows 2012 / amd64 so Windows / amd64 remains a tier-1 platform.
▶ [patch] #2536 The node-postgres library is now configured to correctly handle timezones. As no data was stored with timestamps until now, this is not a breaking change.
▶ [patch] bug 1622052 The protocol between workers and worker manager now correctly negotiates capabilities.
▶ Additional changes not described here: bug 1623183, #2527, #2539.
▶ [MAJOR] #2328 This version adds a temporary "widgets" API method to the notify service. This is intended to allow testing of the deployment process for Taskcluster services' backend database, and not for tracking of actual widgets.
This new API requires that Helm properties notify.read_db_url and notify.write_db_url be set correctly as documented in the deployment documentation.
▶ [minor] Add worker-runner binaries to the list of release artifacts
▶ [minor] bug 1621630
Support for short-circuiting of boolean logic in JSON-e templates such as .taskcluster.yml is restored.
▶ [patch] AWS, GCP and Azure providers support the "shutdown" message, which requests the worker-manager to terminate the instance
▶ [patch] bug 1621167 The Taskcluster-GitHub service now uses structured logging to describe its handling of events from GitHub. See its logging documentation for details.
▶ Additional change not described here: bug 1621270.
▶ [minor] bug 1621630 JSON-e has been reverted to v3.0.1, meaning that short-circuit evaluation of boolean operators is again unsupported. This support will return soon.
▶ [minor] bug 1621630 JSON-e has been reverted to v3.0.2, meaning that short-circuit evaluation of boolean operators is again unsupported. This support will return soon.
▶ [MAJOR] bug 1620109
The long-deprecated queue.defineTask API method has been removed.
▶ [minor] bug 1573192
A task's metadata.owner is no longer required to have the form of an email address, as discussed in RFC#153.
▶ [patch] Fixed worker-manager's azure-provider to properly report worker pool errors when provisioning workers fails.
▶ [patch] bug 1547731 The web-server service now includes structured logging for web-server requests.
▶ [patch] Changes version imports for internal go libraries that weren't properly updated by the release script to v26.
▶ [MAJOR] bug 1596177
Legacy create-task scopes without a priority, of the form queue:create-task:<provisionerId>/<workerType>, are no longer supported.
▶ [minor] Add support for a simple generic FreeBSD worker
▶ [minor] bug 1473155 Schemas are now displayed in a two-column viewer to provide a more comprehensive understanding of the schema structures. The left panel shows the overall data structure while the right panel shows additional properties to keep in mind for certain data within the schemas. Users can also expand or shrink $ref schemas when needed.
▶ [minor] bug 1618916
The Azure worker-manager provider now provides bootstrapping information to the worker in tags in addition to the customData instance metadata field, and worker-runner now expects to find data in tags. This avoids the use of the barely-functional customData. Reading this information from customData is now deprecated, but will continue to work at least until the next major Taskcluster release.
▶ [minor] The json-e library now supports short-circuiting in boolean logic, and so does Taskcluster for taskcluster.ymls now!
▶ [patch] bug 1619925 Bug fix: taskcluster-proxy credential updates from task reclaims no longer race with taskcluster proxy process termination. Previously if a task completed just as the task was being reclaimed, it was possible for generic-worker to terminate the taskcluster-proxy process while it was HTTP posting updated credentials to it, which caused generic-worker to crash.
▶ [patch] bug 1559434
Pulse passwords are now correctly encoded and can contain / characters.
▶ [patch] #2386 Taskcluster UI now no longer shows a cached view when a user deletes a role, client or hook.
▶ [patch] bug 1558240 The generic-worker logging change that appeared in v25.4.0 has been reverted.
▶ [patch] bug 1617685 The queue service will now start up even if the AWS IP-to-region mapping file is not accessible. In this case, it will use a local, cached copy of this information.
▶ [patch] bug 1618983
The worker-manager's static provider type now supports worker lifecycles, and in particular reregistrationTimeout.
▶ [patch]
Update registerWorker API to grant scopes for workers to terminate themselves
▶ [patch] bug 1591476
worker-manager's registerWorker() now returns worker config, and worker-runner (for Azure and static providers, others coming soon) merges that configuration with other configuration sources. This allows worker pools to include configuration for static workers, and allows Azure workers to fetch their config without referencing the non-functional customData instance metadata.
▶ Additional changes not described here: bug 1596171, #2441, bug 1455632.
▶ [minor] bug 1608185
Taskcluster-worker-runner now passes --with-worker-runner to generic-worker when running it directly. When running generic-worker as a Windows service, this argument should be included in the service definition.
Only generic-worker versions 25.0.0 and higher support this argument. In general, we recommend running matching versions of taskcluster-worker-runner and generic-worker.
▶ [minor] bug 1522154 Matrix notifications are now supported if a deployment is configured with credentials for a homeserver. The three fields needed are:
notify.matrix_base_url: foo # The homeserver where your client is registered
notify.matrix_user_id: bar # The user that will act on behalf of taskcluster
notify.matrix_access_token: baz # An access token for this userIf you are using riot, you can get the access token by following this guide.
▶ [patch] bug 1600071 Avoid overprovisioning for instances that take a long time to boot.
▶ [patch] #2404 Fix worker type page when the latest task has no runs. Previously, an error panel was being displayed with text "t.run is null".
▶ [patch] bug 1616922 Generic-Worker documentation is now included in the Taskcluster documentation site, and the generic-worker task payload has been slightly tightened.
task.payload.artifactsmust contain unique itemstask.payload.onExitStatus.retrymust contain unique items
▶ [patch] bug 1558240
Generic-worker now outputs a newline before === Task Finished ===, to ensure that line is separated from other output in the logs.
▶ [patch] bug 1433854 Task directories from previous task runs on Windows are now more aggressively purged.
This should reduce the amount of time spent trying to delete task directories between task runs, and also the amount of logging, in addition to freeing up more disk space.
This issue always existed on the Windows version of generic-worker. A similar issue existed on macOS and Linux but was fixed in bug 1615312 which was initially tagged for release in v25.0.0, but first appeared in release 25.3.0 due to some problems with the release process.
▶ [patch] #2004 The Task Details panel in the Task view now wraps the payload text in order to be able to see the complete payload without scrolling.
▶ [patch] bug 1618066 fix bug where workerInfo could have NaN values
▶ [patch] bug 1616649 reimplements azure-provider's use of the azure SDK to avoid blocking operations that can hold up worker-manager iterations resource creation operations that were previously waiting for completion in the provisioner now are tracked and checked on as part of the worker-scanner iteration
▶ Additional change not described here: bug 1616900.
▶ [minor] bug 1616214 Source code repositories taskcluster-worker-runner and jsonschema2go have been migrated to the taskcluster monorepo. This is an internal change that should not impact the release. However, it is a reasonably significant change to the build/release process.
▶ [patch] #2377 Editing a task that contains ISO-8601 dates embedded in larger strings no longer fails with "Invalid Date".
▶ [patch] bug 1616022 Fixes the version number reported by generic-worker. This was first attempted (unsuccessfully) in release 25.2.0.
▶ [patch] bug 1606874 The Taskcluster-GitHub service now checks that the person who filed a pull request is a collaborator and the repo from which the changes are being pulled belongs to a collaborator or is the usptream repository.
▶ [patch] This version removes the undocumented, deprecated WebListener class from taskcluster-client-web.
▶ Additional changes not described here: bug 1437193, #2371, #2375.
▶ [minor] bug 1616022 Generic worker now correctly reports its version number. The version number was incorrectly reported in release 25.1.1.
▶ Additional changes not described here: bug 1615762, #2367.
No changes
▶ [minor] bug 1587511
Worker pools that use cloud providers (aws, azure, google) now support a lifecycle.reregistrationTimeout config that
will make the credentials we hand to these workers expire within that amount of seconds. If the worker still exists
at that time, the instance will be terminated. This lays the groundwork for a subsequent release where you will
be able to have your workers reregister to continue working.
▶ [MAJOR] bug 1608828 Generic worker is now shipped as part of the taskcluster platform release. The generic-worker codebase has been integrated into the monorepo. The former generic-worker github repo is now archived. Consequently, the generic worker version number now matches the taskcluster platform release number. The generic-worker binaries are published to https://github.com/taskcluster/taskcluster/releases.
With this change, the import path for the Taskcluster Go client library changes from github.com/taskcluster/taskcluster/clients/client-go/vNN to github.com/taskcluster/taskcluster/vNN/clients/client-go. Functionality of the library remains unchanged.
▶ [patch] bug 1588099 InsufficientScopes errors now contain a simplfied scope expression describing the missing scopes. In most cases, this will be a single scope.
▶ [patch] bug 1615312
Old generic-worker task directories on POSIX systems (Linux/macOS) are now
deleted more aggressively, by first running chmod u+w -R <task dir> before
running rm -rf <task dir>.
This bug always existed, and could leave files on the filesystem from previous tasks. Those files were not readable to other task users under the generic-worker multiuser engine where they were owned by a different OS user, but they did consume disk space. The files were readable by other tasks under the generic-worker simple engine, where all tasks run as the same user, but simple engine is not used for tasks that contain sensitive/private information.
This bug was present in both the simple and multisuer engine, and has been fixed on both.
Cleanup of Windows task directories will be handled separately in bug 1433854.
▶ [patch] bug 1608185
The generic-worker binary now accepts a --with-worker-runner argument and expects to interact with worker-runner if that option is given. Otherwise, it will assume it is running alone and will not use any worker-runner features.
▶ Additional changes not described here: bug 1615631, #2312, #2321.
▶ [patch] bug 1611266 azure-provider now ensures generated adminPasswords meet all passwords requirements
▶ [minor] #2293 The Taskcluster Python client now has an helper function to easily upload artifacts.
▶ [minor] bug 1604175 The maximum "deadline" has been reverted to 5 days, after its change to 10 days in v24.1.3. Values over 7 days caused internal server errors anyway, because the Azure queue backend cannot handle delays greater than that value. Since this functionality never worked, the revert is considered minor.
▶ [patch] bug 1606874 Changes behavior of tc-github when checking the user permissions on PR: now tc-github always checks the permissions of the PR author (or the organization of the PR origin if the PR was made from a fork in an org)
▶ [patch] bug 1611266 Limit azure-provider name generation to alphanumeric to reduce invalid name errors (previously characters such as _ and - were included in some names and could be the ending character, resulting in errors)
▶ [patch] bug 1613150 Taskcluster services now run with Node version 12.15.0.
▶ [patch] bug 1584208 The client libraries' documentation has been throughly refactored and is now more helpful and contains better links to the documentation site.
▶ [patch] The deployment documentation now contains information on how Pulse users should be set up, as well as a complete schema for the Helm values file.
▶ [patch] bug 1604649 The queue now avoids calling GetEntity for a worker in claimWork when no work was claimed, providing a very minor reduction in Azure load.
▶ [patch] bug 1436478
This version includes the taskcluster-lib-postgres library, but does not use that library at runtime.
▶ Additional changes not described here: bug 1537922, bug 1588083, bug 1611694, bug 1611696, #1963, #2130.
▶ [minor] bug 1600966 Adds a provider for azure vm instances to worker-manager.
▶ [patch]
The Python client now normalizes the root URL in optionsFromEnvironment().
▶ [patch] #2269 Links to specific log lines now autoscroll to correct location.
▶ Additional changes not described here: #2266, #2232.
▶ [patch] #2031 Taskcluster UI revamped the date picker component to allow selecting the hour and the minute in addition to the date.
▶ [patch] bug 1608176
The go client's client.SignedURL(..) function can now accept and sign full URLs in its first argument. This allows signing arbitrary URLs, even if they are not on the same RootURL as the client.
▶ Additional changes not described here: bug 1606948, #2201.
▶ [patch] bug 1598649 Final bits of release debugging (sorry!)
▶ [patch] bug 1598649 Final bit of debugging of the release process. No other changes.
▶ [patch] bug 1598649 Further debugging of the release process.
▶ [patch] bug 1598649 Additional changes to the release process.
▶ [patch] Changes only to the release process.
No changes
▶ [patch] bug 1604175 A task's deadline can now be up to 10 days in the future (replacing the previous limit of 5 days).
▶ [patch] bug 1605933 Fix possible XSS vulnerability with the lazylog viewer
▶ [patch] #1660 Taskcluster UI now properly displays the error panel in the docs site.
▶ Additional changes not described here: bug 1588083, bug 1598643, bug 1598649, bug 1602985, #1684, #2130, #2187, bug 1598649.
▶ [patch] #2159 Taskcluster UI /auth/scopes view has been revamped to improve the experience and avoid confusions with the Clients and Roles views.
▶ [patch] #2166 Taskcluster UI Worker view now gracefully allows a user to quarantine a worker when a recent task has expired.
▶ [patch] bug 1537922
The auth.createClient API method is now properly idempotent, allowing the same call multiple times in short succession.
▶ [patch] bug 1603197 The notify service's irc process now logs a bit more contextual information about what it is up to, and ignores some common replies from IRC servers instead of considering them "unhandled".
▶ Additional changes not described here: #2125, #2179.
▶ [patch] Fix regression in Taskcluster UI custom actions in the task view not being triggered.
▶ [patch] Provisioning logic now counts workers correctly
▶ [patch] #2155 Taskcluster UI no longer takes the user to a different run when expanding the artifacts dropdown.
▶ Additional change not described here: #2152.
▶ [minor] bug 1599122 Worker Manager now does a better job at keeping provisioning in-sync with reality.
- Workers now have a top-level
capacityfield which is how many tasks it can run at once. - Workers now have
lastModifiedandlastCheckedfields which are useful for determining the state the worker is actually in vs what state Taskcluster thinks it is in. - When calling
createWorkermanually, you can now specify a capacity for the worker.
▶ [minor] bug 1587511
WorkerPools can now be configured to terminate workers that fail to register after some amount of time.
Both of the google and aws providers now support a lifecycle object that for now has a single key
of registrationTimeout. It is optional and if it is provided the value is an integer with the number
of seconds a worker has to register before it is terminated.
This helps catch misconfigured or broken workers before they become zombies or worse.
▶ [patch] #217 Taskcluster UI no longer displays duplicated app bars when connecting via SSH.
▶ [patch] bug 1595749 Taskcluster login now properly handles an edge case where a couple of users were hitting which prevented them to login.
▶ [patch] bug 1599550
The auth.sentry_* Helm parameters are no longer required. If they are omitted, then the service will start up but the auth.sentryDSN REST API method will return 404's.
▶ [patch] bug 1599893
Worker Manager now takes optional configuration to change the timings on the lib-iterate loops
that control provisioning. The values are worker_manager.provisioner_iterate_config and
worker_manager.worker_scanner_iterate_config. Each is a JSON object where you can set the following:
maxFailures- Which sets how many iterations in a row can fail before the task crashesmaxIterationTime- How long (in ms) an iteration is allowed to take before it is endedwaitTime- How long (in ms) to wait in between loops
▶ Additional changes not described here: #2114, #2130, bug 1577839.
▶ [patch] bug 1602642 The typo in configuration for aws s3 bucket credentialing is fixed.
It was set as allowdBuckets and is now allowedBuckets
▶ [patch] bug 1601149
The github.github_private_pem Helm configuration now correctly accepts a configuration containing raw (unescaped) newlines.
A change to how configuration values are escaped in the Helm templates caused this support to regress in 24.0.0.
▶ [patch] #2096 Workers in the UI are now displayed in a table instead of cards.
▶ [MAJOR] bug 1598758
Credentials for the auth.awsS3Credentials method are no longer specified in Helm properties auth.aws_access_key_id, auth.aws_secret_access_key, and auth.aws_region. Instead this information is now configured in auth.aws_credentials_allowed_buckets as described in the deployment docs. The region is no longer required, but the configuration must now include a list of supported buckets. For a quick update, set auth.aws_credentials_allowed_buckets to [{"accessKeyId": "<access_key_id>", "secretAccessKey": "<secret_access_key>", "buckets": ["<bucket_name>"]}].
▶ [MAJOR] Services that previously used hard-coded values despite advertising Helm parameters now honor those optional Helm parameters:
notify.irc_portgithub.provisioner_idgithub.worker_type
The last two parameters name a worker pool (<provisioner_id>/<worker_type>) that is used as a default for older (v0) .taskcluster.yml files.
Rather than set these parameters, users should be encouraged to set the values explicitly in .taskcluster.yml.
The notify service no longer accepts Helm configuration property notify.irc_pulse_queue_name. No known deployment has this value set.
▶ [MAJOR] bug 1577785
The Helm configuration properties queue.public_blob_artifact_bucket, queue.private_blob_artifact_bucket, and queue.blob_artifact_region are no longer allowed, as the artifact types these configured are no longer supported.
▶ [MAJOR] bug 1598329
The long-deprecated queue.pollTaskUrls API method has been removed.
▶ [minor] bug 1585157 All current worker-manager's API endpoints, queue's artifact-related endpoints, working and non-checks-related github's endpoints, and the listLastFires endpoint are being graduated from experimental status to stable.
▶ [minor] bug 1596615 Switch to Node 12.13.0
▶ [minor] #895 Taskcluster UI now uses the v4 version of material-ui. It was previously running on v3.
▶ [minor] #450 Taskcluster docs now supports quick search.
▶ [minor] bug 1518190 Taskcluster now supports backups, restores, and verification of Azure tables and containers. See the deployment docs for details.
▶ [minor] #2028 The Taskcluster Python client now has helper classes to ease integration into customers' projects.
▶ [patch] bug 1599291 Added logging around worker provisioning logic to keep better track of workers.
worker-requested,worker-running,worker-stoppedare all three new log messages that allow you to track the lifecycle of workersscan-seenreports on the state of the world that the worker-scanner has observed on each runsimple-estimatormessages now have an error status ifrunningCapacityis greater thanmaxCapacity. This state occurs due to a bug in worker-manager and should be reported to the taskcluster team if it occurs- This state will also report an error to a configured error reporter if you have one.
▶ [patch] Fix URL construction for signing in with multiple scopes.
▶ [patch] bug 1597331
Instances created by the AWS provider now have an explicit WorkerPoolId tag. The Google provider now supplies created-by and owner tags.
▶ [patch] #1398 Taskcluster UI "Compare Scopesets" and "Expand Scopesets" views now deeply linked. In other words, you can share the URL and still preserve state.
▶ [patch] bug 1600125 Taskcluster UI Secret view no longer requires the save button to be under the code editor to save a secret.
▶ [patch] bug 1600127 Taskcluster UI Secret view now allows making modifications to the secret multiple times without having to reload the page.
▶ [patch] #2073 Taskcluster UI Task view now properly links to the Worker view when clicking on the Worker ID.
▶ [patch] #2078 Taskcluster UI Workers view now include quarantined workers by default without having to toggle the filter dropdown.
▶ [patch] #1909 Taskcluster UI log viewer now displays the log name in the app bar.
▶ [patch] #1558 Taskcluster UI no longer requires two clicks to return back to the list of resources after editing a resource (e.g., a secret).
▶ [patch] #1913 Taskcluster UI no longer uses the same status color for pending and unscheduled labels.
▶ [patch] #2005 Taskcluster UI now adds more accuracy when displaying the distance between given dates in words.
▶ [patch] #1685 Taskcluster UI now allows editing a worker pool that is scheduled for deletion.
▶ [patch] bug 1597276 Taskcluster UI now doesn't open artifacts in the log viewer by default when the file is not plain text.
▶ [patch] #1874 Taskcluster UI now properly aligns menu items in action menu (speed dial).
▶ [patch] #2076 Taskcluster UI speed dial component no longer toggles on hover.
▶ [patch] Taskcluster login no longer throws a TypeError when a profile from the PersonAPI has no identities when logging in via auth0.
▶ [patch] bug 1597922 Taskcluster now has the necessary CSP headers to avoid clickjacking.
▶ [patch] bug 1596098 The Queue and Hooks services now return a 400 error when an entity is too large for the storage backend, instead of a 500.
▶ [patch] #1949 The Task view in Taskcluster UI now allows users to have the artifacts panel expanded on page load if the url has the artifacts hash (i.e., #artifacts)
▶ [patch] #1900 The Taskcluster UI Task view now shows "Reason Resolved" above the fold. You previously had to click "See More" to find this field.
▶ [patch] #1997 The log view in Taskcluster UI now properly scrolls horizontally. Some users were experiencing text truncation for long lines as well as scrolling issues on mobile.
▶ [patch] bug 1599564 The purge-cache service now recovers better from Azure errors, where previously a single Azure error would cause subsequent API calls to also fail until the service was restarted.
▶ [patch] #1455 The schema viewer in Taskcluster UI now properly shows a tooltip when pattern is cut off.
▶ [patch] bug 1491551
When an API request times out, the JS client now correctly retuns an error describing a timeout with err.code === 'ECONNABORTED', instead of err.code === 'ABORTED'.
▶ [patch] #1715 Worker Manager UI now provides a more recent version of workerPool configs for initial values.
▶ [patch] bug 1599122 Worker-manager's AWS provider now more precisely aligns its worker-spawning counts to the desired capacity. Due to rounding, it may previously have spawned up to one additional instance per launchConfig.
▶ [patch] bug 1586839 getInstallations endpoint was renamed to listInstallations in octokit. This patch fixes our call to the API
▶ Additional changes not described here: bug 1511676, bug 1579496, bug 1588096, bug 1596171, bug 1598643, bug 1598788, bug 1599299, #1244, #1412, #1421, #1658, #1747, #1751, #1774, #1822, #1908, #1953, #2019, #677, #1911, #1968, #1754, #1934, bug 1596417, #1773.
▶ [MAJOR] Support for several deprecated services has been removed.
- The login service has been removed from the codebase and from all client libraries. It was retired on November 9, 2019 when the external services that depended on it migrated to third-party login support. It was never part of the Helm deployment.
- Support for the deprecated ec2-manager and aws-provisioner services has been removed from all client libraries. These services are no longer running, so this should have minimal impact.
- Support for the long-removed events service and the never-released gce-provisioner service has been removed from the Go client.
▶ [MAJOR]
The Taskcluster Go client no longer uses the deprecated concept of BaseURL, instead requiring a RootURL. Users of the New and NewFromEnv functions do not need to change anything. However, any code that has manually constructed a client object, or set such an object's BaseURL property, must be updated to use RootURL instead.
▶ [MAJOR]
The auth.statsumToken method has been removed. The service for which this returns a token has not run for over a year, so the impact is minimal.
▶ [MAJOR] bug 1577785
The artifact types blob and azure are no longer supported. Neither of these types has seen real use, and both are broken in all known deployments of Taskcluster.
The Object Service will implement much of the same functionality, but likely with subtle differences. Removing these unused artifact types now will simplify migration to the Object Service once it is developed.
▶ [MAJOR]
The auth service no longer accepts Helm configuration properties auth.client_table_name or auth.role_container_name. These values are now assumed to be Clients and auth-production-roles, respectively. No known deployments of Taskcluster use any other value.
The auth service now honors sentry_organization, sentry_host, sentry_team, and sentry_key_prefix. Previously, the values of these properties were ignored.
▶ [minor] #1923 The web-server service now uses its own azure session table to keep track of sessions. This solves the following issues:
- Restarting the web-server service clears all user sessions
- Spinning up multiple werb-server services for load balancing is not possible since we stored sessions in memory and the latter belong to a single instance
▶ [patch] bug 1595221
Adds an LRU cache to getTask method, so that we don't have to make too many calls to Azure (tasks are immutable anyways)
The default value for the cache size is 10. The name of the optional prop in the dev-config.yml is queue.task_cache_max_size
▶ [patch] bug 1595838 Errors completing a blob artifact upload are no longer returned with statusCode 500.
▶ [patch] #1962 Taskcluster UI error panels are now scrollable.
▶ [patch] bug 1574854 Taskcluster UI now does not show a "404" text when a page could not be found in the UI so as not to pretend an HTTP response code that didn't occur.
▶ [patch] bug 1595734 Taskcluster UI now properly creates interactive tasks from the task creator.
▶ [patch] #1881 Taskcluster UI now properly renders the task title in the app bar.
▶ [patch] bug 1595418 Taskcluster UI now properly shows task dependencies of tasks that don't have a decision task. A task with no decision task is a common thing to have outside the firefox-ci cluster.
▶ [patch] #1951 Taskcluster UI now properly shows the Quarantine Until date.
▶ [patch] #1972 Taskcluster UI now shows up to 1000 workers and worker-types in the paginated table. We previously only showed ~15 rows per page.
▶ [patch] bug 1595667 Taskcluster third-party login UI now instructs users to sign in to provide credentials to a third party registered client instead of showing them the home page.
▶ [patch] bug 1596523 Taskcluster web-server process will stop crashing when something goes wrong when logging in.
▶ [patch] #1988
The built-in retrigger action no longer removes fields like taskId from within the task definition.
▶ [patch] bug 1593762 The google provider now accepts workerpools with underscores in the name
▶ [patch] bug 1595238 The queue service now polls Azure queues for deadline, dependency, and task claims less frequently when those queues are empty. This should reduce the rate of GetMessageRead and GetMessagesRead Azure API calls.
▶ [patch] bug 1579065
This release upgrades Hawk, the underlying authentication mechanism for REST API access, to @hapi/hawk since the older hawk dependency is depreciated.
▶ Additional changes not described here: bug 1596531, bug 1585141, #1946, #1995.
▶ [patch] Third-Party Logins now correctly intersect the requested scopes with the user's expanded scopes. Previous versions would result in a client with an empty set of scopes, when the required scopes were associated with a role given to the user.
▶ [minor] #1875 Taskcluster UI now adds the ability to cancel a task from the Task view
▶ [minor] #1919
Taskcluster UI now exposes an additional env var BANNER_MESSAGE to inform users with important messages (e.g., "Taskcluster will be down for maintenance on November 11") in the UI.
▶ [patch] bug 1588083
Deployment smoketests can now be run from a taskcluster/taskcluster-devel:v<version> Docker image.
See the deployment documentation for details.
▶ [patch] #1857
Errors regarding authorizedScopes are now formatted in Markdown, and thus more readable in error messages in the Taskcluster UI.
▶ [patch] #1895 Taskcluster UI CLI login now uses the intersection of scopes (?scope=...) with the user's scopes to generate the set of scopes added to the client.
▶ [patch] #1892 Taskcluster UI now adds the ability to retrigger a task from the Task view.
▶ [patch] #1879 Taskcluster UI now allows users to copy artifact links from index browser through the normal right-click-copy-link.
▶ [patch] bug 1593809
The taskcluster-github service now correctly uses the github.bot_username configuration to look up the latest status for a branch.
Deployments of Taskcluster should double-check that this value is set correctly; see the deployment docs for details.
▶ [patch] The taskcluster-index service now responds with a 404 and "Indexed task not found" when a task is not found, instead of the misleading "Indexed task has expired".
▶ [patch] bug 1593754 The web-server service now uses the correct Pulse namespace to listen for pulse messages. This fixes one more bug preventing task and task-group UI from dynamically updating.
▶ [MAJOR] bug 1591591
The deployment Helm variable ui.application_name has been renamed to a top-level applicationName. This value is now used as context in the GitHub status and check posts to PRs and commits.
▶ [MAJOR] bug 1590175
Worker pools now support instance capacity in configuration such that larger instances can handle more tasks if desired. The configuration option, instanceCapacity was already accepted but previously had no effect. As long as this value is set to 1 for all aws and google worker pools, this change will have no effect.
▶ [minor] #1758 Taskcluster shell client 'signin' command can now interact with the new UI.
▶ [patch] #1842 API documentation display is fixed.
▶ [patch] bug 1593142
AWS Providers in Worker Manager now handle RequestLimitExceeded errors from AWS gracefully with exponential backoff
▶ [patch] #1771 Taskcluster now properly allows a client to be saved when the "Delete on expiration" switch is changed when updating an existent client.
This release includes additional changes that were not considered important enough to mention here; see https://github.com/taskcluster/taskcluster/tree/v22.0.0%5E/changelog for details.
▶ [minor] bug 1588834
- AWS Provider worker pools now allow specifying additional userdata beyond that generated by the provider itself.
▶ [minor] #1529 When a third party site tries to login to the deployment, Taskcluster now attempts to auto login when there is only one login strategy configured. Previously, a user had to click on "Sign In" then click on the login strategy.
▶ [patch] #1839 Sign-In buttons now work properly with Firefox Nightly, instead of failing with a blank tab.
▶ [patch] #1835 Taskcluster now properly read the expires query parameter for whitelisted third-party login clients. It was previously creating third-party login clients using the maxExpires value. This issue was only seen with clients that are whitelisted.
▶ [patch] #1840 The Taskcluster UI can now fire actions with type 'task' without causing a schema validation error.
▶ [patch] #1838 The task-group and task views now update dynamically as tasks change status.
This release includes additional changes that were not considered important enough to mention here; see https://github.com/taskcluster/taskcluster/tree/v21.3.0%5E/changelog for details.
▶ [minor] bug 1589449
- Implements remove worker functionality in Worker Manager AWS provider.
- Corrects a typo in the route of remove worker api endpoint of Worker Manager
▶ [minor] #1713
Taskcluster now supports command-line logins via the UI. Query parameters
are client_id and callback_url.
▶ [minor] bug 1590848
The JSON-e context used to render .taskcluster.yml in GitHub repositories now contains taskcluster_root_url giving the root URL.
This can be used for conditionals in the file, or to generate URLs.
▶ [patch] bug 1545939 All long-runnning processes are now restarted once every 24 hours by kubernetes. This is partially to replicate how Heroku ran the services and partially just because it is a good idea.
This release includes additional changes that were not considered important enough to mention here; see https://github.com/taskcluster/taskcluster/tree/v21.2.0%5E/changelog for details.
No changes
▶ [minor] bug 1589449
- Implements remove worker functionality in Worker Manager AWS provider.
- Corrects a typo in the route of remove worker api endpoint of Worker Manager
▶ [minor] #1713
Taskcluster now supports command-line logins via the UI. Query parameters
are client_id and callback_url.
▶ [minor] bug 1590848
The JSON-e context used to render .taskcluster.yml in GitHub repositories now contains taskcluster_root_url giving the root URL.
This can be used for conditionals in the file, or to generate URLs.
▶ [patch] bug 1545939 All long-runnning processes are now restarted once every 24 hours by kubernetes. This is partially to replicate how Heroku ran the services and partially just because it is a good idea.
This release includes additional changes that were not considered important enough to mention here; see https://github.com/taskcluster/taskcluster/tree/v21.1.0%5E/changelog for details.
[MAJOR] (bug 1578900) * Worker Manager AWS Provider now requires the ec2:DescribeRegions permission in addition to the previous permissions.
The full permissions set is documented in the deploying workers section of the manual.
- Worker Manager AWS Provider now uses all the configs from the array of
launchConfigsworker pools use, rather than a single, randomly selected config. This allows per-region and per-zone resources to be specified. MinCapacity and MaxCapacity are now specified for the whole worker pool as opposed to for every individual config.
some/worker:
config:
minCapacity: 25
maxCapacity: 50
- regions: [us-central1, ...]
- capacityPerInstance: 1
- ...
+ launchConfigs:
+ - region: us-central1
+ capacityPerInstance: 1
+ ...[minor] (#1576) AWS Provisioner support has been removed from the UI and it is no longer a navigation menu item. This service has not been a part of the Taskcluster deployment for some time.
(bug 1589403) Fix a regression in Github logins. A header was not being set.
(#1573) The UI now properly listens to pulse messages.
It was previously hard-coded to a value that would only
work on https://taskcluster-ui.herokuapp.com/.
We now read the pulse namespace from PULSE_USERNAME.
(#1665) The web-server service now properly configures CORS for
its third party login endpoints /login/oauth/token and
/login/oauth/credentials.
(bug 1589368) Taskcluster-GitHub now correctly reports InsufficientScopes errors, instead of "Cannot read property 'unsatisfied' of undefined".
[MAJOR] The worker-manager service's google provider type now requires that worker pool definitions contain an array of possible variations of workers for the pool, in the launchConfig property.
See google provider type for more detail.
Note that this is a breaking change that will cause all google-based worker pools to stop provisioning until they have been updated to the new format.
To update, change the config field by moving all fields except minCapacity and maxCapacity into an array in launchConfigs:
some/worker:
config:
minCapacity: 25
maxCapacity: 50
- region: us-central1
- zone: us-central1-a
- capacityPerInstance: 1
- minCpuPlatform: "Intel Skylake"
- ...
+ launchConfigs:
+ - region: us-central1
+ zone: us-central1-a
+ capacityPerInstance: 1
+ minCpuPlatform: "Intel Skylake"
+ ...(bug 1585102) The GitHub service now posts a more useful comment to pull requests and commits when an InsufficientScopes error occurs.
The message now includes the scopes used to make the API call, including the assume:repo:.. role.
[MAJOR] (bug 1584321) Scopes for the Taskcluster services themselves are now handled internally to the platform, although access tokens must still be managed as part of the deployment process.
When deploying this version, remove all scopes and description properties from static/taskcluster/.. clients in the array in the Auth service's STATIC_CLIENTS configuration.
See the new docs on static clients for more background on this setting.
[minor] (bug 1586102) The github service now adds scopes for check/status scopes and its scheduler-id, where previously it had relied on specific configuration of the repo:github.com/* role.
There is no longer a need to add such scopes scopes to the role repo:github.com/*.
[minor] (#1486) The Worker-Manager google provider implementation now supports terminating instances in response to workerManager.removeWorker(..) API calls.
(#1495) In the previous version, indirect go dependency github.com/streadway/amqp had an invalid pseudo-version.
This has been fixed, and the tool that generated the incorrect dependency (renovate) has been disabled.
(bug 1585135) The fix in 18.0.2 is updated to replace all escaped newlines in the GITHUB_PRIVATE_PEM config, not just the first.
(bug 1585135) The github.private_pem configuration in GITHUB_PRIVATE_PEM can now be specified with "regular" newlines or with encoded newlines (\ \n).
This works around a bug in the generation of multiline secrets present in the Mozilla deployment pipeline.
No changes
[MAJOR] (bug 1583935) Administrative scopes for worker pools are now worker-manager:manage-worker-pool:<workerPoolId>.
Existing worker-manager:{create,update}-worker-type:<workerPoolId> scopes are no longer recognized.
[minor] (bug 1323871) Taskcluster now issues scopes based on repo access for Github logins. Static clients need to be updated in deployments.
(bug 1582376) Taskcluster now uses the AMQP server's value for frame_max, rather than enforcing its own limit of 4k.
The server level should be configured to 128k.
This is the default for RabbitMQ, so in most cases no change is required.
[MAJOR] (bug 1561905) 1. Static clients need to be updated in deployments.
2. The web-server service now requires azure credentials configured for login to work properly, namely
AZURE_ACCOUNT_ID, AZURE_SIGNING_KEY, and AZURE_CRYPTO_KEY.
3. For a third party to get TC credentials, it first needs to have a client registered in the deployment of the
web-server service. This is governed by the REGISTERED_CLIENTS configuration.
See https://docs.taskcluster.net/docs/manual/deploying/third-party for the shape of a client.
[MAJOR] (#1260) Google provider in worker-manager now requires you to manually set up
a service account for your workers to run under. If you are migrating
from a previously deployed worker-runner, you can just use the account
we created for you automatically before. It always had the name
taskcluster-workers.
Your config will changein the following way:
# Old
providers:
google-project:
providerType: google
project: ...
creds: ...
instancePermissions:
- ...
- ...
# New
providers:
google-project:
providerType: google
project: ...
creds: ...
workerServiceAccountId: ...(#778) User-created clients are regularly scanned, and disabled if the owning user no longer has the relevant scopes. Such users are now also disabled if the owning user has been removed from the identity provider.
(#1216) Users of taskcluster-ui are now logged out if they are not logged-in in the eyes of web-server. This would avoid having web-server be out-of-sync when restarted for example.
[minor] (bug 1561320) Taskcluster deployments now support sentry error reporting. You can configure this option by setting
an errorConfig at the top-level of your config:
rootUrl: ...
errorConfig:
reporter: SentryReporter
dsn: <your sentry dsn>
Errors will be reported to this project and tagged with service/process names in addition to taskcluster release version.
(bug 1574656) Worker-pool configurations for google-based providers now accept a workerConfig property, which is passed to new workers.
The existing userData property is deprecated.
[minor] (bug 1572775) * All lib-loader setup functions now get passed their own
name to allow logging more usefully.
- There is now a document in dev-docs explaining recommended monitoring practices.
[minor] (bug 1553953) The workerType identifier now has a more restrictive pattern:
- consisting of lower-case alphanumeric plus dash (
-) - from 1 to 38 characters long
- beginning with a lower-case alphabetic character
- ending with a lower-case alphanumeric character (not a dash) Any worker types not matching this pattern will no longer function as of this version.
This is considered a minor change because no known workerTypes (aside from some internal testing workerTypes) violate this pattern.
[minor] (bug 1572764) The go client doesn't log the full request in case of an error anymore.
It logs only the method, hostname, port and response body. It logs the
full request when the environment variable TASKCLUSTER_DEBUG is
defined.
[minor] (#1190) Updates a number of config variables including:
- Setting
pulse-namespaceper service is no longer supported - Services that no longer use aws directly no longer take credentials
- Setting table names for secrets, notify, and hooks services is no longer supported
The name of the hooks last fires table has changed so you must update your static
client scopes in your deployment from including auth:azure-table:read-write:${azureAccountId}/LastFire
to auth:azure-table:read-write:${azureAccountId}/LastFire3.
[MAJOR] (bug 1552970) The auth.gcpCredentials method no longer modifies the granting service account.
Instead, that service account must be configured with the "Service Account Token Creator" role prior to deployment of Taskcluster.
The format of configuration for these credentials has changed as well, now taking GCP_CREDENTIALS_ALLOWED_PROJECTS.
See the deployment documentation for more information.
[MAJOR] (bug 1570723) The deployment configuration value ui.ui_login_strategy_names is now required.
It should be a space-separated list of the names of the strategies in web_server.ui_login_strategies.
[minor] (#1140) Add Chain of Trust documentation for taskcluster worker implementations and maintenance.
[minor] (#1062) The taskcluster cli rerun action now takes a --force option. It will refuse to rerun non-exception, non-failed tasks without --force.
(#1108) The development process has been improved to use kubectl directly instead of helm. Helm is still used to render templates because we need to support it.
[MAJOR] The web-server application no longer generates a JWT when logging in. It uses sessions to keep track of users.
The JWT_KEY configuration variable in web-server should be replaced with SESSION_SECRET which is used to compute
the session hash.
[MAJOR] (#1005) There is now a checked-in helm chart in infrastructure/k8s. Using this anyone should
be able to deploy taskcluster by just setting up the configuration.
To facilitate this, some environment variables for configuring services have changed:
- All services now take
AZURE_ACCOUNT_IDinstead ofAZURE_ACCOUNTorAZURE_ACCOUNT_NAME - Hooks takes
AZURE_CRYPTO_KEYandAZURE_SIGNING_KEYinstead ofTABLE_CRYPTO_KEYandTABLE_SIGNING_KEY
[minor] (#1084) The Dockerfile for the Taskcluster services is now checked-in rather than generated at build time. It has been reordered so that changes to things other than package.json won't re-install packages.
Include generated APIs in python package.
[minor] Pulse messages now include a task's tags for better classification of the messages that are received.
[minor] (bug 1563545) The apiMethod log structure has been updated so that it now splits out query params into their own field and only logs the useful part of paths for resources.
[minor] (bug 1558345) The experimental workerManager.credentialsGoogle API method has been removed and replaced with a similar but more provider-agnostic workerManager.registerWorker method.
[minor] (bug 1523807) The taskcluster command-line interface (taskcluster-cli) has been incorporated into the main repository and will be relased with the same version numbers as the Taskcluster services.
[minor] The web-server application now uses CORS headers to limit access to the /graphql and /subscription endpoints to requests from the root URL origin.
An additional, optional configuration value, ADDITIONAL_ALLOWED_CORS_ORIGIN, provides a way to allow additional origins.
If it begins and ends with /, it is treated as a regular expression, allowing matching e.g., pull-request draft deployments.
[minor] What was previously the /worker-pools-errors/:workerPoolId API route is now spelled /worker-pool-errors/:workerPoolId.
This endpoint is still experimental so while this might someday be a breaking change, it is currently considered minor.
[minor] (bug 1563341) Worker-manager now allows getting workers by worker group and singly by worker ID, and creating and removing workers (for some providers). The static provider uses this capability to manage static workers, each authoritatively identified by a shared secret.
(bug 1547077) Emails now use the modern Taskcluster logo
The GRAPHQL_SUBSCRIPTION_ENDPOINT config for taskcluster-ui can now have scheme http or https instead of ws/wss.
This allows easier generation of this configuration as ${TASKCLUSTER_ROOT_URL}/subscription.
The existing schemas are still accepted so no configuration change is required.
With the proper scopes, github repositories can now override the default scheduler. Adding custom schedulerId to the task definition while using github's Statuses API might break the status reporting functionality of tc-github in the case of successful build. Therefore, this only works with experimental checks status reporting.
[minor] The AWS Provisioner and Provisioner views are no longer available, as the AWS provisioner itself will be removed in favor of the worker manager service.
[minor] (bug 1560649) The Go client is now hosted in the repository together with the services and other clients, and co-versioned with them. See the docs.
[minor] (bug 1559471) The web-server configuration for sign-in now requires a single JWT HS256 key (JWT_KEY) instead of a public/private key (JWT_PRIVATE_KEY/JWT_PUBLIC_KEY).
Changes are now recorded in the CHANGELOG.md file.
(bug 1547729) Hook fire attempts are now logged using structured logging, including when a hook "declines" to create a task.
(bug 1556526) The workerManager.updateWorkerType API method now allows extra fields such as lastModified, making read-modify-write usages easier to implement.
The search box in the log viewer now searches on enter.
The task group inspector now shows the full task name.
(bug 1558346) Workers can now report errors directly to the worker manager for display in the worker-manager UI.
Changes were not tracked for older releases of Taskcluster