Updates Vendor Dependencies #1762

schnie · 2024-12-12T21:55:21Z

Description

I'm currently working on cleaning up the output of our astro dev commands and plan to hide most of it behind our --verbosity flag by default. Our standard output will be much cleaner and will aim to not confuse our users, especially new ones who don't need to see a ton of docker layer SHAs getting echod out every time.

In order to sufficiently control output, we need a much newer version of the docker compose libraries. The currently used version directly attaches to os.Stdout and we can't override. New versions allow this to be specified and allow more control over the output.

This PR is the result of running go get -u and updating all our packages. One off updates were not working well for me. This gets our packages up to date and fixes the subsequent fallout from those upgrades in our code. It's the bare minimum changes needed to simply update the packages.

Follow up PRs will utilize this change and fine-tune the output to our needs.

I wish this was smaller, but I feel like we need to get up to date for many reasons. Changes mostly boil down to:

Scattered docker/compose type updates. New packages changed paths, names, etc of types. Mostly same functionally.
Updating the linter made it much more aggressive, so fixed a lot of the things it was failing on.
Updated mocks for new docker/compose types.

🎟 Issue(s)

Related to https://github.com/astronomer/astro/issues/26133

🧪 Functional Testing

This is a non-functional update, so I've only updated the tests to support the changes in types, etc. To test for regressions, I've run the following suites against this branch:

We've triggered the Astro e2e-stage-cli-version-matrix workflow on the Astro repo and pointed at this branch. The workflow can be viewed here. Note the switch to this branch in this step.
We've also run the new (currently in development) pytest astro dev suite against this branch with the following output:

❯ poetry run pytest -x -s -v dev_test.py
Configuration file exists at /Users/schnie/Library/Preferences/pypoetry, reusing this directory.

Consider moving TOML configuration files to /Users/schnie/Library/Application Support/pypoetry, as support for the legacy directory will be removed in an upcoming release.
=============================================================================================================================================== test session starts ===============================================================================================================================================
platform darwin -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0 -- /Users/schnie/repos/astro-cli/test/integration-test/venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/schnie/repos/astro-cli/test/integration-test
configfile: pyproject.toml
collected 9 items                                                                                                                                                                                                                                                                                                 

dev_test.py::test_dev_init PASSED
dev_test.py::test_dev_start PASSED
dev_test.py::test_dev_ps PASSED
dev_test.py::test_dev_logs PASSED
dev_test.py::test_dev_parse PASSED
dev_test.py::test_dev_run PASSED
dev_test.py::test_dev_restart PASSED
dev_test.py::test_dev_export PASSED
dev_test.py::test_dev_kill PASSED

=============================================================================================================================================== 9 passed in 59.92s ================================================================================================================================================

Manual smoke testing locally

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

📋 Checklist

Rebased from the main (or release if patching) branch (before testing)
Ran make test before taking out of draft
Ran make lint before taking out of draft
Added/updated applicable tests
Tested against Astro-API (if necessary).
Tested against Houston-API and Astronomer (if necessary).
Communicated to/tagged owners of respective clients potentially impacted by these changes.
Updated any related documentation

schnie · 2024-12-13T16:38:39Z

.circleci/config.yml

@@ -5,6 +5,7 @@ jobs:
    working_directory: /go/src/github.com/astronomer/astro-cli
    docker:
      - image: quay.io/astronomer/ap-dind-golang:24.0.6
+    resource_class: large


Linting became a little heavier.

airflow-client/airflow-client.go

schnie · 2024-12-13T18:28:37Z

airflow/docker.go

+
+	// The latest versions of compose libs handle adding these labels at an outer layer,
+	// near the cobra entrypoint. Without these labels, compose will lose track of the containers its
+	// starting for a given project and downstream library calls will fail.
+	for name, s := range project.Services {
+		s.CustomLabels = map[string]string{
+			api.ProjectLabel:     project.Name,
+			api.ServiceLabel:     name,
+			api.VersionLabel:     api.ComposeVersion,
+			api.WorkingDirLabel:  project.WorkingDir,
+			api.ConfigFilesLabel: strings.Join(project.ComposeFiles, ","),
+			api.OneoffLabel:      "False",
+		}
+		project.Services[name] = s
+	}


Copied from https://github.com/docker/compose/blob/v2.32.0/cmd/compose/compose.go#L354-L367.

jeremybeard

This looks good. Approving ahead of a couple of small comments.

Also, I have a couple of PRs that overlap with this:

Upgrade golangci-lint to v1.62.2 #1756
Use new Logrus logger #1757
But more of an fyi that one of us will have merge conflicts to deal with :)

jeremybeard · 2024-12-17T04:10:05Z

airflow/docker_image_test.go

@@ -11,12 +11,13 @@ import (
 	"os"
 	"strings"

+	"github.com/docker/docker/api/types/image"


Can we add a lint rule that these imports need to be kept together?

Went ahead and cleaned up the 4 files that had the imports shuffled around nested under airflow directory. Will have to look at linters, not sure there.

jeremybeard · 2024-12-17T04:12:05Z

go.mod

+	k8s.io/klog/v2 v2.130.1 // indirect
+	k8s.io/utils v0.0.0-20241210054802-24370beab758 // indirect
+	sigs.k8s.io/structured-merge-diff/v4 v4.4.3 // indirect
+	sigs.k8s.io/yaml v1.4.0 // indirect
 )

 replace github.com/jaguilar/vt100 => github.com/tonistiigi/vt100 v0.0.0-20190402012908-ad4c4a574305


What are these replace statements for?

Looks like we can remove this distribution one. It was needed as I was working through updates last week, but seems to build fine without this replace now.

The ldez/tagliatelle one unfortunately seems to still be required. It's pulled in through the golangci-lint package and it feels like something off about the way they package things. Without this pin, I get the following error linting:

❯ make lint go run github.com/golangci/golangci-lint/cmd/golangci-lint version # github.com/golangci/golangci-lint/pkg/golinters/tagliatelle ../../go/pkg/mod/github.com/golangci/golangci-lint@v1.62.2/pkg/golinters/tagliatelle/tagliatelle.go:13:3: unknown field Rules in struct literal of type "github.com/ldez/tagliatelle".Config make: *** [lint] Error 1

jeremybeard · 2024-12-17T04:16:32Z

Update 1 minute later: I merged my linter upgrade PR. Sorry, hopefully it's not too hard to resolve.

neel-astro

Left a minor nit, and apart from merge conflicts the changes LGTM. Approving ahead of time.

airflow/docker.go

schnie force-pushed the update-compose-libs branch 6 times, most recently from 0347ee6 to 60051b8 Compare December 13, 2024 04:01

Updates vendor dependencies

11fc3ac

schnie force-pushed the update-compose-libs branch from 60051b8 to 11fc3ac Compare December 13, 2024 04:08

schnie marked this pull request as ready for review December 13, 2024 16:37

schnie requested review from neel-astro, jeremybeard, kushalmalani and lzdanski as code owners December 13, 2024 16:37

schnie commented Dec 13, 2024

View reviewed changes

airflow-client/airflow-client.go Outdated Show resolved Hide resolved

schnie commented Dec 13, 2024

View reviewed changes

schnie changed the title ~~Updates vendor dependencies~~ Updates Vendor Dependencies Dec 16, 2024

jeremybeard approved these changes Dec 17, 2024

View reviewed changes

neel-astro approved these changes Dec 17, 2024

View reviewed changes

airflow/docker.go Outdated Show resolved Hide resolved

schnie added 6 commits December 17, 2024 16:28

Merge branch main

e8bec99

Defer resetting logging configuration

22fd915

Clean up imports

cf4a57b

Remove bodyclose linter and revert in-line ignores

85cf1df

Merge branch 'main' into update-compose-libs

f484cb1

Remove docker distribution replace in go.mod

577bce3

schnie merged commit 986bdcc into main Dec 17, 2024
3 checks passed

schnie deleted the update-compose-libs branch December 17, 2024 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates Vendor Dependencies #1762

Updates Vendor Dependencies #1762

schnie commented Dec 12, 2024 •

edited

Loading

schnie Dec 13, 2024

schnie Dec 13, 2024

jeremybeard left a comment

jeremybeard Dec 17, 2024

schnie Dec 17, 2024

jeremybeard Dec 17, 2024

schnie Dec 17, 2024

jeremybeard commented Dec 17, 2024

neel-astro left a comment

Updates Vendor Dependencies #1762

Updates Vendor Dependencies #1762

Conversation

schnie commented Dec 12, 2024 • edited Loading

Description

🎟 Issue(s)

🧪 Functional Testing

📸 Screenshots

📋 Checklist

schnie Dec 13, 2024

Choose a reason for hiding this comment

schnie Dec 13, 2024

Choose a reason for hiding this comment

jeremybeard left a comment

Choose a reason for hiding this comment

jeremybeard Dec 17, 2024

Choose a reason for hiding this comment

schnie Dec 17, 2024

Choose a reason for hiding this comment

jeremybeard Dec 17, 2024

Choose a reason for hiding this comment

schnie Dec 17, 2024

Choose a reason for hiding this comment

jeremybeard commented Dec 17, 2024

neel-astro left a comment

Choose a reason for hiding this comment

schnie commented Dec 12, 2024 •

edited

Loading