Migrate from micromamba to pixi #230

kklein · 2024-06-21T10:15:14Z

No description provided.

kklein · 2024-06-21T12:26:12Z

.readthedocs.yml

@@ -2,17 +2,14 @@ version: 2
 build:
  os: ubuntu-20.04
  tools:
-    python: mambaforge-4.10
+    python: mambaforge-latest
+  commands:


@pavelzw This example seems to be structured a little differently:
https://github.com/prefix-dev/pixi/blob/main/examples/readthedocs-extend/docs/.readthedocs.yaml

i took https://github.com/prefix-dev/pixi/tree/main/examples/readthedocs-override as inspiration. but maybe the readthedocs-extend is a better fit for sphinx ...

kklein · 2024-06-21T13:07:40Z

Hi @pavelzw ! Any idea about the origin of this failure?

https://github.com/Quantco/datajudge/actions/runs/9613959735/job/26517778652?pr=230

pavelzw · 2024-06-21T13:24:40Z

    × unknown environment 'snowflake-sa2'

we commented this env out because it took too long to solve on windows... i think this is an issue with pixi. for now, let's deactivate it. i'll file an issue

pavelzw · 2024-06-21T13:38:04Z

prefix-dev/resolvo#49

.github/workflows/ci.yaml

pixi.toml

ivergara · 2024-06-21T14:45:52Z

pixi.toml

+db2-py38 = ["db2", "py38", "test"]
+db2-py39 = ["db2", "py39", "test"]
+db2-py310 = ["db2", "py310", "test"]
+# db2-py311 = ["db2", "py311", "test"]


Not sure why you have some environments commented out.

these are not solvable but might be in the future. no strong opinions on whether we should keep them commented out or remove the lines

I don't care if they're commented. I just wondered because this is the reason a PR merging into this branch is failing and wanted to understand why. I'd say leave them commented out for now.

I removed the comment for now

.github/workflows/ci.yaml

Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com>

.github/workflows/ci.yaml

ivergara · 2024-06-21T20:05:02Z

.github/workflows/ci.yaml

+          - latest
+          - 11
+        env:
+          - postgres-py38


This doesn't look complete. We should have the cross product of python, postgres and sqlalchemy versions. Here it looks like python and sqlalchemy are decoupled now.

This observation applies to the others too.

i think testing against the border of the cross product matrix should be fine as well, otherwise this can easily explode. i also don't think we would catch significantly more errors that way 🤔

We're aware it can explode. That's why in the obscure databases we covered a few combinations.

We can discuss how we effectively do integration tests. Doing the borders of the testing phase space is fine by me.

ivergara · 2024-06-21T20:05:27Z

.github/workflows/ci.yaml

+          - db2-py311
+          - db2-py312
+          - db2-sa1
+          - db2-sa2


We werent testing db2 with sqlalchemy 2 before.

ivergara · 2024-06-21T20:06:57Z

.github/workflows/ci.yaml

+          - snowflake-py311
+          - snowflake-py312
+          - snowflake-sa1
+          - snowflake-sa2


We werent testing sqlalchemy 2. Although It might be interesting to give it a second try.

Furthermore, we weren't testing all python versions for these more "esoteric" databases to reduce time, and costs.

ivergara · 2024-06-21T20:12:01Z

.pre-commit-config.yaml

+        always_run: true
+        require_serial: true
+        pass_filenames: false
+      - id: black


The target versions for black and pyupgrade are gone from here. Are they implicitly using the value for requires-python in pyproject.toml?

probably. but we want to move to ruff anyway in a separate PR (which definitely uses requires-python)

Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com>

codecov · 2024-06-23T18:48:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.14%. Comparing base (c52ad81) to head (2d4714f).
Report is 39 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #230      +/-   ##
==========================================
- Coverage   92.49%   92.14%   -0.36%     
==========================================
  Files          18       18              
  Lines        1973     1973              
==========================================
- Hits         1825     1818       -7     
- Misses        148      155       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kklein · 2024-06-23T18:53:41Z

@ivergara any idea why the BigQuery authentication is failing?

ivergara · 2024-06-24T07:25:42Z

@ivergara any idea why the BigQuery authentication is failing?

Not really. Nothing seems to have changed in this PR. My suggestion for now is to skip this integration test and deal with it separately.

kklein · 2024-06-24T08:03:46Z

@pavelzw I compared the readthedocs pages manually; the version number/PR number is the only difference I came across

kklein · 2024-06-24T12:35:22Z

Thanks for your help @ivergara and @pavelzw :)

* Draft usage of pixi. * Adapt more integration tests. * Adapt bigquery integration test task. * Migrate docs deployment to pixi. * Update build and publish. * Run postinstall for readthedocs. * Adapt liniting to rely on pixi. * Add pytest as dependency. * Adapt readthedocs configuration. * Update development instructions. * Switch to pixi in helper script. * Adapt development.rst. * Comment out impala run for debugging. * Move flit to host-dependencies. * Remove redundant dependency. * Update .github/workflows/ci.yaml Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com> * Add color to ci tests. * Update .github/workflows/ci.yaml Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com> * Update .github/workflows/ci.yaml Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com> * Remove configurations which were erroneously added. * Remove configurations which were erroneously added. * Consistently use ubuntu-latest. * Update .github/workflows/ci.yaml Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com> * Also run unit tests on macos and windows. * Fix impala test syntax. * Downgrade sqlalchemy version for impala tests. * Disable BigQuery tests for now. --------- Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com>

Draft usage of pixi.

50622dd

kklein added the sqlserver label Jun 21, 2024

kklein added 2 commits June 21, 2024 12:30

Adapt more integration tests.

e1b5693

Adapt bigquery integration test task.

8c7b453

kklein added the ready label Jun 21, 2024

kklein added 8 commits June 21, 2024 12:38

Migrate docs deployment to pixi.

54630fd

Update build and publish.

28b3c30

Run postinstall for readthedocs.

48ea9e9

Adapt liniting to rely on pixi.

e5f63a9

Add pytest as dependency.

cbce273

Adapt readthedocs configuration.

9246bc8

Update development instructions.

bc66fe1

Switch to pixi in helper script.

3c06b09

kklein changed the title ~~Draft usage of pixi.~~ Migrate from micromamba to pixi Jun 21, 2024

Adapt development.rst.

7e4ce72

kklein commented Jun 21, 2024

View reviewed changes

Comment out impala run for debugging.

951c28e

pavelzw reviewed Jun 21, 2024

View reviewed changes

.github/workflows/ci.yaml Show resolved Hide resolved

ivergara reviewed Jun 21, 2024

View reviewed changes

pavelzw reviewed Jun 21, 2024

View reviewed changes

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

kklein and others added 4 commits June 21, 2024 17:14

Move flit to host-dependencies.

35ab3ab

Remove redundant dependency.

aaa5cbf

Update .github/workflows/ci.yaml

03ba29c

Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com>

Add color to ci tests.

b182853

pavelzw reviewed Jun 21, 2024

View reviewed changes

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

.github/workflows/ci.yaml Outdated Show resolved Hide resolved

ivergara reviewed Jun 21, 2024

View reviewed changes

kklein and others added 6 commits June 23, 2024 18:12

Update .github/workflows/ci.yaml

6087281

Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com>

Update .github/workflows/ci.yaml

fe7786e

Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com>

Remove configurations which were erroneously added.

c85ad7d

Remove configurations which were erroneously added.

d7720e6

Consistently use ubuntu-latest.

a22ef2a

Update .github/workflows/ci.yaml

2b7aeb0

Co-authored-by: Pavel Zwerschke <pavel.zwerschke@quantco.com>

Also run unit tests on macos and windows.

1cfafb5

kklein added 2 commits June 23, 2024 21:02

Fix impala test syntax.

66973f5

Downgrade sqlalchemy version for impala tests.

0f0cb2a

kklein force-pushed the migrate_to_pixi branch from 5b54a0c to 0f0cb2a Compare June 23, 2024 19:54

Disable BigQuery tests for now.

2d4714f

kklein marked this pull request as ready for review June 24, 2024 08:02

kklein requested review from ivergara and pavelzw June 24, 2024 08:02

ivergara approved these changes Jun 24, 2024

View reviewed changes

pavelzw approved these changes Jun 24, 2024

View reviewed changes

kklein merged commit 73c1792 into main Jun 24, 2024
57 of 58 checks passed

kklein deleted the migrate_to_pixi branch June 24, 2024 12:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from micromamba to pixi #230

Migrate from micromamba to pixi #230

kklein commented Jun 21, 2024

kklein Jun 21, 2024

pavelzw Jun 21, 2024 •

edited

Loading

kklein commented Jun 21, 2024

pavelzw commented Jun 21, 2024

pavelzw commented Jun 21, 2024

ivergara Jun 21, 2024

pavelzw Jun 21, 2024

ivergara Jun 21, 2024

kklein Jun 23, 2024

ivergara Jun 21, 2024

pavelzw Jun 22, 2024

ivergara Jun 22, 2024

ivergara Jun 21, 2024

ivergara Jun 21, 2024

ivergara Jun 21, 2024 •

edited

Loading

pavelzw Jun 22, 2024

codecov bot commented Jun 23, 2024 •

edited

Loading

kklein commented Jun 23, 2024

ivergara commented Jun 24, 2024

kklein commented Jun 24, 2024

kklein commented Jun 24, 2024

Migrate from micromamba to pixi #230

Migrate from micromamba to pixi #230

Conversation

kklein commented Jun 21, 2024

Choose a reason for hiding this comment

pavelzw Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

kklein commented Jun 21, 2024

pavelzw commented Jun 21, 2024

pavelzw commented Jun 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivergara Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 23, 2024 • edited Loading

Codecov Report

kklein commented Jun 23, 2024

ivergara commented Jun 24, 2024

kklein commented Jun 24, 2024

kklein commented Jun 24, 2024

pavelzw Jun 21, 2024 •

edited

Loading

ivergara Jun 21, 2024 •

edited

Loading

codecov bot commented Jun 23, 2024 •

edited

Loading