Skip to content

Conversation

@ashb
Copy link
Member

@ashb ashb commented Oct 1, 2025

As the Edeg API server is currently implemented we need to "pretend" to be a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
@kaxil kaxil requested a review from Copilot October 1, 2025 16:37
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a prototype Edge Worker client to the Go SDK, allowing Go workers to connect to the Edge API server and execute Airflow workloads. The implementation includes JWT authentication to mimic specific Airflow and Edge provider versions, with fallback defaults that can be configured via environment variables.

Key changes:

  • Adds Edge API client generation and JWT authentication middleware
  • Implements edge worker with job fetching, heartbeat, and task execution capabilities
  • Refactors configuration management for better reusability across worker types
  • Consolidates logging configuration and level handling

Reviewed Changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
go-sdk/pkg/edgeapi/client.go JWT authentication middleware for Edge API
go-sdk/pkg/edgeapi/client.gen.go Generated Edge API client from OpenAPI spec
go-sdk/edge/worker.go Core edge worker implementation with job management
go-sdk/pkg/config/config.go Refactored shared configuration and logging setup
go-sdk/pkg/logging/level.go Centralized trace level constant definition
go-sdk/pkg/logging/shclog/shclog.go Updated to use centralized trace level
go-sdk/pkg/worker/runner.go Removed verbose logging statement

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@ashb ashb merged commit 2136f56 into apache:main Oct 1, 2025
51 checks passed
@ashb ashb deleted the go-edge-worker branch October 1, 2025 16:50
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 2, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 3, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 4, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 5, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 7, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 8, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 9, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 10, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 11, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 12, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
dabla pushed a commit to dabla/airflow that referenced this pull request Oct 12, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 14, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 15, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 17, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
abdulrahman305 bot pushed a commit to abdulrahman305/airflow that referenced this pull request Oct 19, 2025
As the Edeg API server is currently implemented we need to "pretend" to be  a
specific Airflow and Edge provider version. These default to the currently
released versions, but can be changed via env vars to work elsewhere.

This works enough to run tasks, but there might need to be some changes to the
Edge API to support non-python clients (for example, working out the
versioning strategy to make it long-term supportible and not need the Airflow
and Edge Provider version to match 100%).

A chunk of the changes here are to make the config and global setup "more well
structured" -- so that they are suitable to be easily called from multiple
workers (Celery and Go, useful if we don't end up keeping the Celery worker)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants