Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Airbyte CDK: move from gradle to poetry #36250

Merged
merged 57 commits into from
Mar 25, 2024
Merged

Airbyte CDK: move from gradle to poetry #36250

merged 57 commits into from
Mar 25, 2024

Conversation

erohmensing
Copy link
Contributor

@erohmensing erohmensing commented Mar 18, 2024 •
edited
Loading

What

  • Migrate from Gradle to Poetry for airbyte-cdk build
  • Remove all gradle-related files
  • Remove java installation from CI pipelines and replace with installing poetry
  • Move setup.py implementation to poetry
  • Move high-usage scrips to poetry poe tasks:
    • Testing, linting, building (generator scripts)
    • There is a separate PR to handle the mypy implementation since it has its own CI workflow and is more complicated with the changed files
  • Remove unused scripts:
    • low-code-unit-test.sh
    • validate-yaml-schema.sh
  • Update docs to stop referencing gradle: ⚠️ I did my best here. A lot of these docs are severely out of date and deserve a separate thorough lookover. TODO: create an issue enumerating docs that should be re looked over

@erohmensing
move check tasks out of gradle into a shell script
d61c8c2
@erohmensing
move extra tasks i'm not sure what we'll do with yet
565e9d9
@erohmensing
update check python cdk ci task and remove clean task. expect this to…
d30ab5f 
… fail
@erohmensing
trigger tests?
427fe61
@erohmensing
fail scripts on errors
8b0ed42
@erohmensing
delete prep java task
0396930
@erohmensing
remove extra dependencies from gradle configuration
92186ef
@erohmensing
comment out actual image pushing and stuff that posts to slack for te…
38ab886 
…sting
@erohmensing
remove anything about ec2 runners
b3a1d61
@erohmensing
remove one more potential push
fb338b4
@erohmensing
comment out autocommit
691596b
@erohmensing
comment out pull request creation
530e0c3
@erohmensing
remove dependency
9463599
@erohmensing
comment out publish python package
8397962
@erohmensing
Revert "remove anything about ec2 runners"
e055280 
This reverts commit fd9ba60.
@erohmensing
remove java and gradle from publish command definition
aa211f7
@erohmensing
bring back dependency
d88f786
@erohmensing
use --load instead of --push in order to not push to remote
5a064ee
@erohmensing
comment out cdk tests while working on publish
031e97e
@erohmensing
add path to command
b683693
@erohmensing
move assemble tasks out of gradle into a shell script
892e377
@erohmensing
remove gradle stuff
bb6b015
@erohmensing
first pass - setup to poetry
2d9595e 
fix issues

remove setup.py

move scripts into poetry

update workflows

remove assemble.sh in favor of composite build command

remove check.sh script in favor of composite commands (add flake8 dependency, others not necessary)

try new version of pflake8

downgrade pytest to 6.2.5

add lockfile
@vercel Vercel
Copy link

vercel bot commented Mar 18, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Mar 25, 2024 11:34pm

@erohmensing Graphite App
Copy link
Contributor Author

erohmensing commented Mar 18, 2024 •
edited
Loading

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @erohmensing and the rest of your teammates on Graphite Graphite

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Mar 18, 2024
@erohmensing erohmensing changed the title move assemble tasks out of gradle into a shell script Airbyte CDK: move from gradle to poetry Mar 18, 2024
@erohmensing erohmensing mentioned this pull request Mar 18, 2024
Closed
@erohmensing
Copy link
Contributor Author

Not quite ready, marking RFR for tests to run

@CLAassistant
Copy link

CLAassistant commented Mar 25, 2024 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@erohmensing erohmensing force-pushed the ella/cdk-gradle-to-poetry branch from dceee45 to 5d1ce5f Compare March 25, 2024 20:42
@erohmensing erohmensing changed the base branch from ella/aws-runners to master March 25, 2024 21:06
@erohmensing erohmensing force-pushed the ella/cdk-gradle-to-poetry branch from 0ff3dd1 to 1f7b851 Compare March 25, 2024 21:08
@erohmensing
Copy link
Contributor Author

@alafanechere

I'm not sure I get how you replaced low-code-unit-tests and validate-yaml-schema scripts.

I 🪓 them with Brian Lai's permission! Better Pr description should help here.

Totally get rid of bash scripts
Make the build poe task produce all the artifacts required for a pypi publication.
Document the build poe task

Agreed. I think our current publish via github action doesn't need the artifacts (well, i'm about to find out!) but as we move to airbyte ci, this makes sense. In the meantime before fleshed out docs, I added help info to all of the poe tasks.

@erohmensing erohmensing force-pushed the ella/cdk-gradle-to-poetry branch from d233044 to f2d7e97 Compare March 25, 2024 22:59
@erohmensing erohmensing force-pushed the ella/cdk-gradle-to-poetry branch from f2d7e97 to 18e4d75 Compare March 25, 2024 23:31
@erohmensing
Copy link
Contributor Author

/approve-and-merge reason="Formatting check can't find the changed files, probably due to the weird stacking-unstacking history of this branch. Formatting succeeds locally on the latest airbyte CI version."

@octavia-approvington
Copy link
Contributor

A job well done
nice job

@octavia-approvington octavia-approvington merged commit 195408e into master Mar 25, 2024
26 of 27 checks passed
@octavia-approvington octavia-approvington deleted the ella/cdk-gradle-to-poetry branch March 25, 2024 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@girarda girarda girarda left review comments

@octavia-approvington octavia-approvington octavia-approvington approved these changes

@alafanechere alafanechere alafanechere approved these changes

Assignees
No one assigned
Labels
area/documentation Improvements or additions to documentation CDK Connector Development Kit
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@erohmensing @CLAassistant @octavia-approvington @girarda @alafanechere @octavia-squidington-iii