Kubernetes/Kueue release candidate

[cortadocodes]

Contents (#723)

IMPORTANT: There are 2 breaking changes.

New features

• Add octue question ask CLI command
• Add ability to ask remote question from CLI
• Add octue question events raw command
• Add octue question events replay CLI command
• Allow requesting compute resource amounts per question
• Add cancel method to Service and Child
• Add octue question cancel CLI command

Enhancements

• Deserialise octue question ask inputs from JSON
• Add async option to octue question ask
• Deprecate octue run command
• Support parsing already-extracted questions
• Remove unnecessary log message
• Improve description of commands
• Suppress google crc32c warning
• Improve description of octue question ask local
• Move get-diagnostics command to octue question diagnostics
• Allow passing of attributes to octue question ask local
• 💥 BREAKING CHANGE: Allow setting of Octue services topic name via envvar
• Log when question event extraction is complete
• Log question acknowledgement to pub/sub
• Allow use of service registries with octue question ask remote
• Remove cloud run runtime warning
• Add GOOGLE_KUEUE to compute providers and remove old ones
• Remove option to parse events from Cloud Run
• Add log message explaining waiting for question acceptance
• Simplify question cancellation
• Avoid trying to cancel question if it's already finished
• Log warning instead of raising if question already finished
• 💥 BREAKING CHANGE: Remove outdated octue run CLI command
• Return result from Service.answer and octue question ask local
• Avoid getting non-result events from event store if unnecessary
• Warn if no result is returned by octue question events replay
• Make service/app config mandatory in answer_question
• Log warning if no events found in octue question events get
• Log warning if no events found in octue question events replay
• Remove redundant --only-handle-result option from replay command
• Return no result from replay command if no events/result
• Remove cloud run specific commands in dockerfiles
• Delete unsupported python3.9 dockerfile

Fixes

• Fix incorrect type
• Return question UUID for async octue question ask
• Fix event and attributes generation
• Add missing kwarg
• Fix argument name and backend format in CLI command
• Fix --asynchronous option in command
• Fix setting attributes for question event
• Return data from octue question ask remote properly
• Await publishing of result event
• Return early for async question in octue question ask remote
• Avoid assuming question is a dict
• Fix typo in OCTUE_SERVICES_TOPIC_NAME envvar
• Remove non-existent service revision check for now
• Add missing elif
• Convert attributes to correct types in make_attributes
• Extract attributes correctly from passed-in format
• Explicitly pass credentials to publisher/subscriber clients
• Refresh credentials before returning
• Make service config optional for octue question ask remote
• Only add resource request attributes if non-None
• Avoid passing invalid ID to child in octue question cancel
• Add missing key
• Remove restriction on attributes for PARENT sender type
• Fix log message
• Fix string interpolations in log messages
• Wait for event emission future by default
• Fix incorrect path change in dockerfiles
• Pass service/app config correctly in octue question ask local
• Pass CLI JSON output to stdout as valid JSON
• Use encoder when dumping to JSON in CLI
• Send output manifests to stdout in serialised form
• Avoid deserialising input manifest in octue question ask local
• Deserialise input manifest from JSON in octue question ask local
• Remove GOOGLE_KUEUE from log timestamp exclusion

Operations

• Increase minor version number
• Correct version number
• Remove terraform config
• Move pydocstyle config inside pyproject.toml
• Use pydocstyle within ruff in pre-commit checks

Dependencies

• Temporarily remove numpy and pandas dev dependencies
• Drop support for python 3.8 and 3.9
• Add numpy and pandas back to dev dependencies
• Update lock file

Refactoring

• Split octue question ask into ask and ask-local commands
• Rename answer_question to answer_pub_sub_question
• Move ask* subcommands under octue question ask group
• Rename octue question events raw to octue question events get
• Remove Cloud Run flask app
• Move dockerfiles into their own directory
• Move answer_pub_sub_question module and rename
• Remove push subscription creation logic
• Remove unused global variable
• Remove other mentions of cloud run
• Remove unused dockerfile and workflow
• Split pub/sub event/attributes extraction into two functions
• Move attributes extraction into events subpackage
• Rename function
• Factor out getting local octue version
• Move CRC32C warning catch below imports
• Rename function
• Rename answer_pub_sub_question to answer_question
• Allow create_sruid to create default SRUIDs
• Simplify octue question ask local
• Tidy up octue question events get
• Tidy up octue question events replay
• Simplify making an originator question
• Move no events/result warnings into underlying functions

Testing

• Update test for python3.13
• Update mock paths
• Fix test for different python versions
• Move cloud run deployment test
• Fix test
• Test octue question ask remote command
• Test octue question ask local command
• Test output manifests are passed to stdout properly from CLI
• Factor out repeated fixture
• Test octue question events get CLI command
• Test octue question events replay CLI command
• Test octue get-diagnostics is deprecated
• Replace cloud run deployment test with kueue equivalent

Reversions

• Revert "FIX: Refresh credentials before returning"
• Revert "FIX: Explicitly pass credentials to publisher/subscriber clients"

Chores

• Deprecate octue get-diagnostics CLI command
• Ignore GCP credentials files

Style

• Add missing ruff isort pre-commit check

Other

• Update event schema locally
• Document resource parameters in ask methods
• Document cancel methods
• Fix docstrings
• Update docstrings in answer_pub_sub_question
• Improve documentation of octue question ask remote
• Improve docstring
• Remove outdated docstring param
• Merge pull request Add Kubernetes/Kueue service backend #709 from octue/add-kueue-service-backend
• Temporarily hard-code WIF details into CI workflow
• Update docstring

---

Upgrade instructions

💥 Allow setting of Octue services topic name via envvar

Update the services topic in production to be named main.octue.services or set the environment variable OCTUE_SERVICES_TOPIC_NAME=octue.services

💥 Remove outdated `octue run` CLI command

Use the new octue question ask local CLI command instead.