-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Add a prototype Edge Worker client to the Go SDK #56290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds a prototype Edge Worker client to the Go SDK, allowing Go workers to connect to the Edge API server and execute Airflow workloads. The implementation includes JWT authentication to mimic specific Airflow and Edge provider versions, with fallback defaults that can be configured via environment variables.
Key changes:
- Adds Edge API client generation and JWT authentication middleware
- Implements edge worker with job fetching, heartbeat, and task execution capabilities
- Refactors configuration management for better reusability across worker types
- Consolidates logging configuration and level handling
Reviewed Changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| go-sdk/pkg/edgeapi/client.go | JWT authentication middleware for Edge API |
| go-sdk/pkg/edgeapi/client.gen.go | Generated Edge API client from OpenAPI spec |
| go-sdk/edge/worker.go | Core edge worker implementation with job management |
| go-sdk/pkg/config/config.go | Refactored shared configuration and logging setup |
| go-sdk/pkg/logging/level.go | Centralized trace level constant definition |
| go-sdk/pkg/logging/shclog/shclog.go | Updated to use centralized trace level |
| go-sdk/pkg/worker/runner.go | Removed verbose logging statement |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a specific Airflow and Edge provider version. These default to the currently released versions, but can be changed via env vars to work elsewhere. This works enough to run tasks, but there might need to be some changes to the Edge API to support non-python clients (for example, working out the versioning strategy to make it long-term supportible and not need the Airflow and Edge Provider version to match 100%). A chunk of the changes here are to make the config and global setup "more well structured" -- so that they are suitable to be easily called from multiple workers (Celery and Go, useful if we don't end up keeping the Celery worker)
As the Edeg API server is currently implemented we need to "pretend" to be a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.
This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).
A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.