refactor: use nix to manage development dependencies #2924
Conversation
cpcloud
force-pushed
the
nix-env
branch
5 times, most recently
from
575dd6f
to
bf9d3b1
Compare
docs/web/contribute.md
Outdated
|
|
||
| This may seem overly strict, but it takes a large amount of the release burden off |
There was a problem hiding this comment.
can you show more what is needed on commit messages?
There was a problem hiding this comment.
Yup, here's a screenshot of running cz (which replaces git commit) that shows a subset of what's possible:
Let's say I'm committing a feature, so I select feat. I'd get this prompt now:
Here you can put whatever component or filename you want, let's say I'm editing sql generation, I'd then get prompt to add a commit title:
You then get asked for a longer description which is optional unless there's a breaking change:
... breaking changes are the next question:
Finally you're asked about open issues the commit affects:
The commit conventions are taken from the Angular community: https://github.com/angular/angular.js/blob/master/DEVELOPERS.md#-git-commit-guidelines
There was a problem hiding this comment.
can you add this to the docs :-> (screen shots and everything). as this is a new workflow
There was a problem hiding this comment.
To be clear, it isn't necessary to use cz, but it automates making the commit messages conform (which is now verified in CI).
There was a problem hiding this comment.
i c.
ok then prob a good idea to show this workflow AND show that link above for the allowed messages.
There was a problem hiding this comment.
Yup, will add to the docs for sure.
There was a problem hiding this comment.
If this is verified in CI then I would say it's necessary to use cz (which is fine), otherwise I can see the commit message doesn't pass CI easily.
Yup. In short it's used to generate an environment file in CI.
The minimum versions for dependencies are set in
Light documentation of it exists in |
|
@cpcloud left some comments, in general +1 to standardize/automatic things so thanks for this! My main concern is that I haven't used many of the tools here (I think some other folks as well), so would be really helpful to write dev docs describing how the new CI/dev/release works. |
Heck yeah! @jreback has requested more docs as well. I think I'll also add a screencast to really help clarify how these things are used. |
cpcloud
force-pushed
the
nix-env
branch
5 times, most recently
from
331383c
to
49a9d55
Compare
| @@ -0,0 +1,3 @@ | |||
| # Release Notes | |||
|
|
|||
| <!--next-version-placeholder--> | |||
There was a problem hiding this comment.
FYI this is where new release notes based on commit logs will start to be automatically populated upon release.
|
@jreback @icexelloss I've added a bunch of docs on the dev and release process. Please review at your earliest convenience! |
There was a problem hiding this comment.
umm why all the references to 'breaking changes' here? why is this in any way controlled by the commits that someone pushes? how could they possibly know if something is a breaking change or not?
sure the maintainers can know / understand this, but these are general instructions.
docs/web/contribute.md
Outdated
| If you don't have a database for the backend you want to work on, you can check the configuration of the | ||
| continuos integration, where docker images are used for different backend. This is defined in | ||
| `.github/workflows/main.yml`. | ||
| Once you're inside of a nix-shell, the first thing you want to do is make a branch. Let's call it `useful-bugfix`. |
There was a problem hiding this comment.
since this is not specific to nix, can you be more generic here
There was a problem hiding this comment.
Can you be more descriptive in what you're asking for?
docs/web/contribute.md
Outdated
|
|
||
| ## Commit philosophy | ||
| **NOTE:** Ensure that you have run `pre-commit run` **before** running `cz`, otherwise |
There was a problem hiding this comment.
is the pre-commit still available? i think you removed the config?
There was a problem hiding this comment.
Yes. I've already explained this elsewhere in the PR.
docs/web/contribute.md
Outdated
| We use [numpydoc](https://numpydoc.readthedocs.io/en/latest/format.html) as | ||
| our standard format for docstrings. | ||
| - required: `git` | ||
| - required: `cz` (available in the `nix` environment) |
There was a problem hiding this comment.
is there a doc link on cz?
There was a problem hiding this comment.
I don't know what you're asking. Do you want me to put a link to the cz repository/documentation?
It seems like you might not have had a chance to take a look at
The answer to this question is also in the above links, but I'll give a brief summary. The point of semantic release is to entirely automate the drudgery of release. This is done by giving semantics to commit messages, e.g., I think you may be missing the fact that these version determinations are batched at release time: there won't be a new release per commit, that would be excessive. Instead, the commits between the last release and the current commit are examined for All of this is covered in the semantic-release repo, please take a look at that. Also note that this PR doesn't implement the release automation. I explicitly state that it'll be done in a follow up in the PR description. Did you read that? Please do so if you haven't.
Not sure who If by |
|
by they i mean - a person pushing commits they cannot possibly know what is. breaking change or not the maintainers could know but o am not sure how this helps |
Not sure what the objection is here. We don't have random people making commits. Every change goes through review. If a change needs a commit to indicate a breaking change, then it's up to maintainer to make sure that the commit has it. Typically this would be done by referring someone to the docs. |
this seems like quite a lot of work - meaning the commits need to be exactly right having release notes as part of the change is way easier |
It's not a lot of work, and cz handles it. I've used this procedure at both past companies and in my own projects and whatever small overhead there is in making your commits more structured is easily made up by the ability to release with confidence, whenever we want. cz makes this process even easier because it'll prompt you, and construct the commit message correctly. Check out the prompts I show above, which you explicitly requested. It might be useful to try it out, so that if you have specific objections I can address those. We don't have that many breaking changes. We do however need to release faster, and it needs to be a super boring and routine procedure that anyone can do with a single click. I would like to move forward with this unless there specific actionable objections, in which case I will address them. |
|
ok will look again tomorrow in any event would like to make the docs super clear about this |
|
btw likely we are plaguaged by pypa/setuptools#2941 (to mitigate can pin < 60) |
cpcloud
force-pushed
the
nix-env
branch
2 times, most recently
from
a56923b
to
78d47e9
Compare
Thanks. For now it looks like this PR isn't affected. @jreback On a different note, I've reworked the docs and made it even easier to use |
|
kk will look shortly |
|
/condalock |
| # if using nix | ||
| $ ./dev/poetry2setup -o setup.py | ||
|
|
||
| # it not using nix, requires installation of tomli and poetry-core |
There was a problem hiding this comment.
these are in the conda files?
There was a problem hiding this comment.
Not explicitly, but they are both a pip install/conda install away.
|
/condalock |
|
Thanks. I'm going to turn off the |
|
thanks @cpcloud great |
|
🎉 This PR is included in version 2.1.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This is a PR to change the way local development is done in Ibis (while preserving as much backwards compatibility with conda and non-conda/non-nix workflows as possible!), accomplishing the following things:
The main change here is the addition of a development environment that is self-contained and derived from a single source of truth (
pyproject.toml), usingnix.See
docs/web/contribute.mdfor more details.Note that both conda and setuptools-based development workflows are still supported as first class citizens.
DONE:
A move todone in refactor(ci): move to poetry #2937poetryfor dependency management.It is still possible and supported to use setup.py and setuptools if you like, this functionality is checked in CI.
More extensive CI testing, with minimal additional CI time used. It looksdone in refactor(ci): move to poetry #2937like roughly on average 3 minutes of additional CI time occur, with more of
the library being tested on Windows in particular
Automatic generation of a conda recipe for every release, uploaded to GitHubI plan to do this in conda-forge directly, using their integrated bot automationreleases (this is the remaining 1% that isn't automated but this can
be automated by pushing a feedstock pull request directly from CI)
Automated Dependabot updates for github-actions and poetry dependenciesthis is done using Renovate in Configure Renovate #3053Generating a setup.py file from the project'spyproject.tomlfileThis is now just checked that it doesn't need to be regenerated, since we want to ensure this is tested in CI where relevant.done in ci: add workflow to automatically update setup.py #3054, necessary for Renovate dependency update PRsAutomatic generation of a conda environment file, uploaded to GitHub releases, useful for developers using condaWe have automatically generated lockfiles for linux, macos, and windows, for all three versions of Python currently supported (3.7, 3.8, 3.9). These live underconda-lockand can be used to set up a development environment very quickly since there's no solve step required. Done ci: generate conda lock files in CI #3077, ci: use git push instead of github action for lock file PR update commits #3080, ci: pull first when updating conda lock files #3083, ci: fix committing conda lock files from CI #3087Follow ups:
workflow_dispatch(click-to-run) GitHub Actions workflow that cuts a release to PyPI and GitHub Releases.Automatic nix dependency updates via PR submission every six hours or on-demand. This has to be done in a follow up.Done in ci: add automatic dependency update PRs #3182Happy to address any concerns here, let's automate everything!