Add tasks

[michaeldel]

Task

I find it pretty useful for Poetry to be able to run tasks in development, similarly to NPM's scripts (that can be launched with npm run).These tasks will not be available in end-user package installation.

They are defined in a tool.poetry.tasks section, pretty similar to tool.poetry.scripts:

[tool.poetry.tasks]
foo = "echo 'bar'"

And may be run like so

poetry task foo

They are automatically run within the project's virtualenv (the same way poetry run runs scripts).

This new feature should also address some concerns expressed here in #241. make is already suited to take care of tasks running but:

• it is not available on Windows by default (and might sometimes be painful to install and work with, especially in CI environments)
• it requires an additional file whereas this tool.poetry.tasks feature will be consistent with the new pyproject.toml project files.
• make still has to be called through poetry run in order to be integrated within the current virtualenv, hence missing the "shorter command" initial benefit

Current tool.poetry.scripts are currently not that well suited for such operations as well, indeed:

• they can only run python scripts
• they are meant to be installed and available to end-users (which may not be desirable for development tasks)

Use cases

Here are some common use cases where such feature may come in handy.

Lint and test code

[tool.poetry.tasks]
lint = "black --check ."
test = "pytest tests/"

Run development or production server

[tool.poetry.tasks]
dev = "FLASK_APP=app.py flask run"
start = "gunicorn app.wsgi --log-file -"

Clean environment

[tool.poetry.tasks]
clean = "poetry run manage.py flush && rm -rf media"

Improvements

I have first chosen tasks as name in order not to mismatch tasks with scripts (which can be run from an end-user perspective). It may feel a bit of a duplication among some so these might be renamed to "development scripts" (tool.poetry.dev-scripts) in order to be consistent (also poetry run probably remains a better command name than poetry task for that sort of thing).

It does also currently not support passing arguments to tasks, maybe adding an option for this might be an interesting idea (similarly to npm run task -- args).

Some other options might also be considered such as --silent or --pwd.

Current implementation is of course very rudimentary and will probably be further developed later if merged.

Checklist

• Added tests for changed code.
• Updated documentation for changed code.

[jacebrowning]

Placing these in [tool.poetry.dev-scripts] to be run as $ poetry run <name> seems preferable to me because:

• It mirrors the syntax of the existing [tool.poetry.scripts]
• One does not have to know which section the script is in to run it

[Dr-ZeeD]

@jacebrowning Personally, I'd rather have $ poetry run <shell command> reserved for shell commands... Someone like me might have the bright idea to name a "task" test and then I can not access the builtin shell command anymore 😁

[Dr-ZeeD]

On another note, if I understood it correctly this is also some direction the @tox-dev team wants to go, i.e., general automation. Is it really necessary for poetry to also include this functionality, then?

[michaeldel]

@ZeeD26 even though @tox-dev is working on similar features does not mean we should not implement it for Poetry, indeed not everyone uses Tox and adding it ot a project for the sake of tasks sounds overkill to me.
I also think that it makes much more sense to integrate a task system to a tool like Poetry that I find more generalist when it comes to "project management" (kindof).

@jacebrowning I would also prefer such syntax but I am very worried about the aliasing it could involve, maybe it would not be that bad though (@ZeeD26 you may still run the test command without poetry run since there is no dependency from the virtualenv 😄, but I understand your concern)

One other "issue" with [tool.poetry.dev-scripts] would be the need to remain consistent with [tool.poetry.scripts], that is setting a new specific syntax for "shell tasks" in order not to conflict with the current syntax, something along the lines of:

[tool.poetry.dev-scripts]
foo = 'foo:main'
test = { task = "pytest tests/" }

This might get pretty cumbersome 🤔

[woile]

I'm really looking forward to see this feature merged. Most of the time adding a Makefile to a small project feels like too much, but I still want to have the commands documented, and it's much easier to keep it up to date if I use them constantly instead of writing them once in the README and forgetting.

It would be awesome if I could just remember poetry run test and forget about the underlying technology (pytest, pytest + codecov, unittest, nose, etc) and the args (if they are included in the future).

I really like the way yarn behaves. Where any script added to scripts can be executed directly using yarn.

E.g:

{
  "scripts": {
    "build": "node build.js"
  }
}

and then yarn build.

This could be really useful. If I redefine publish instead of executing the default behavior from poetry, I could do something completely different, this would help adoption of the tool.

[purificant]

I would really like this feature. Having this built in makes it easier to share knowledge amongst developers when adopting poetry as a tool of choice for dependency management on a project.
All typical activities such as test / lint / build / ci / deploy / publish can be automated through clearly defined scripts without further tooling.

Do not mind if it's poetry run or poetry task, although the first one is one less character. dev-scripts sounds good too, it's clear that they are intended for package developers and that they are scripts, which reduces cognitive load for users already familiar with the concept of npm scripts for example.

Thank you @michaeldel

[erksch]

What's the status of this? Do you need any help on getting this done?

[purificant]

There are conflicts in the PR that need to be resolved

[erksch]

I was able to merge the current master in this branch without any conflicts (the particular part in poetry/utils/env.py did not change much).

@michaeldel I opened a pull request on your tasks branch with the updates.

[michaeldel]

Codebase has been updated, also resolved conflicts.

[Kamforka]

Is there any documentation available on this implementation yet?

So far it's not clear if we ended up using tools or dev-scripts?

[woile]

It would be nice to have this instead:

[tool.poetry.dev-scripts]
foo = 'foo:main'
test = { task = "pytest tests/" }

Makes it easy to remember being able to reuse poetry run instead of having 2, one for python scripts and one for shell scripts.

I think tasks and dev-scripts will generate confussion.

[michaeldel]

@Kamforka there is minimal documentation in cli.md. At the moment the section used is still [tool.poetry.tasks].

The subcommand name (currently task) has still not be settled, run feels terser but may cause issues because of name conflicts (e.g. a task and a script both called start). Considering we change it so, how do we handle those name conflicts?

• Should one take priority on the other? (e.g. scripts considered higher priority than tasks) Also raising a warning telling about the conflict?
• Should an error be raised instead?

Implementing tasks that way (using the run subcommand) will also probably involve more work when it comes to modifying the scripts operation, this sounds to be considered.

@woile of course dev-scripts would fully replace tasks to avoid confusion.

[erksch]

@michaeldel do you mind update tasks to dev-scripts then?

[brycedrennan]

I'd rather direct people to use Makefiles, which are cross-language compatible, than to build yet another language-specific task runner. Unless poetry is managing python versions as well (with pyenv) then I'll still have to have a Makefile anyway.

But yeah I don't know about windows support.

[purificant]

What's with the holy crusade of trying to teach everyone about Makefiles? We get it, they've been around since early 1970s. They solve some problems, most of those problems have been solved again by many other tools many more times since then. If you want to use Makefiles they will give you a gun and enough rope to shoot / hang yourself with, whichever you prefer.

[michaeldel]

@erksch I am taking care of it as soon as I got time, in some days probably.

@brycedrennan make is pretty cumbersome to use within a Python project using poetry (cf first post), finding a way to "activate" the Poetry environment might simplify this, but I have not studied all the work implementing this would involve. Windows support also sounds like an issue to me, it is still very widely used as a development environment despite its flaws.
I personally find getting development scripts into pyproject.toml the thing to do since it is striving to become the standard Python projects' main management file.

[brycedrennan]

@michaeldel yeah good points. I’ve got no answer for windows support so I must admit this will be helpful for a lot of projects.

[Kamforka]

Unfortunately I'm also doomed to develop under Windows and deploy to Linux and I know a lot others who have to deal with this.

Having poetry as a single source of truth might ease the pain a little bit though.

[erksch]

@hangtwenty
I think the alternative approach you are proposing would not add so much value to poetry. If you set up invoke tasks it's not that hard to just call them with invoke. I see no benefit in moving the call to peotry.

Important is the difference between something like a make, invoke or shovel task and the proposed tasks we (or I) would like to have in poetry. I would write a shovel task for something like seeding my database or setting up constraints or whatever but I would not want to write a shovel task for executing my tests or linting my code. The difference is that the first ones are tasks and the second ones are convenience aliases for executing a tiny bit of code.

And why would adding scripts to the toml result in a mess? Making task runner tasks for everything would be the greater mess.

[Askir]

I'm not sure if everyone is actually scared of a result that would look like this. And even if that is the case you'd just move that mess from the pyproject.toml to another file depending on what task runner you use.

---

I do agree with @erksch on the fact that these scripts/tasks are mostly aliases. As soon as the scripts/tasks become complicated they are moved to different files (see the npm example).
They abstract from the modules that are actually used in the project. Because every project might use a different test runner but poetry run tests would always stay the same.

I think the dream of yarn and npm is that you can start developing simply after running git clone and yarn install. And I personally would love if that was the same for poetry. And installing dependencies is just one of the hassles that i have to go through as a developer.

If I want to test I need to know the testing library that the repository uses, if it is pytest I run poetry run pytest if it is pyunit I run poetry run pyunit if it is mamba I run poetry run mamba then I still need to know the folder where the testing files are saved.

Now the same steps happen when i want to lint. Does the project use prospector? or simply pylint?
All these questions would not have to be asked if I could simply run poetry run lint or poetry run tests.

---

Now the alternative solutions that are suggested are:

• Use Make!
  Well now I have to run poetry install and install Make for this project.
  Just to again call poetry commands from the Makefile?

• Use invoke!
  This makes a bit more sense as it will load the environment already, but again I need to know that behind the scenes the project uses invoke and not a different task runner. It doesn't streamline the process as well as tasks/scripts would do. You could still use invoke to test you'd just have to add invoke tests behind your custom poetry run tests command.

[floer32]

@erksch You mention aliases - I understand what you mean but then let’s compare to aliases in shell scripting. In bash for example aliases can still have further arguments added to them. Would pyproject.toml aliases support that? Or do people have to create redundant aliases with slight variations? Or start using environment variables a bunch? Would pyproject.toml tasks be able to call each other? Do we want to deal with pyproject.toml tasks calling multiple layers of “poetry run” subshells (and does that have any weirdness to consider?)

These questions are answerable but the point I’m trying to make is that aliases are not as simple as you are implying. They only seem simple to you because they are your personal preference.

To others of us, this kind of hardcoding of shell invocations into text is not simple. Instead it proliferates custom work. It does not have less cognitive overhead than the alternative(s), it’s just less consistent.

Contrast with the case where the tasks are executed by a well-defined task runner or sure you can do your own custom approach visible right there. Either way people don't start digging into poetry when they need to debug. The dig into the thing they chose. It's more explicit, and explicit is better than implicit in Pythonic code.

Also I’m not trying to get tit for tat, my higher-level point here is that any approach to this should probably be considered slowly and carefully.

[floer32]

The issue #940 could be widely useful regardless of whether there is something like pyproject.toml in-line classes.

So then why not do that first, and see how it goes, and let this incubate - instead of immediately adding more semantics to the pyproject.toml?

[floer32]

@Askir It looks like something needs clarifying. You said

This makes a bit more sense as it will load the environment already, but again I need to know that behind the scenes the project uses invoke and not a different task runner.

What do you mean? I mentioned the example of poetry inv my_task. inv in that case is invoke, and that has nothing to do with poetry; when you install invoke it creates both entrypoints (see invoke docs: http://docs.pyinvoke.org/en/1.2/#the-invoke-cli-tool )

Per #940 this would be a general concept for running entrypoints. It could be implemented almost trivially, i.e. if poetry <subcommand> is not known, it poetry would check if <subcommand> is an existing executable (like which but portable), if yes then it proceeds.

So you may have thought I was suggesting a way to glue a specific task runner to poetry.

There is no magic and there is no 'behind-the-scenes' to understand. Literally the only difference from current behavior is that you could save typing three characters. poetry run inv my_task would be the same thing today. poetry inv my_task would be the convenience.

To further illustrate the point,

# equivalent to 'poetry run inv my_task'
poetry inv my_task        

# equivalent to 'poetry run invoke my_task'
poetry invoke my_task  

# because it's just 'poetry run' the semantics for adding extra arguments are obvious
poetry inv my_task --flag --blah

# equivalent to 'poetry run flake8 .'
poetry flake8 .

# equivalent to 'poetry run pytest tests/'
poetry pytest tests/

Those examples so far are Python entrypoints. it'd probably make sense if it just worked for anything run worked for though. which means it's anything in your shell.

poetry echo "hello world"              # equivalent to: poetry run echo "hello world"

and so on.

[floer32]

I only used invoke as an example so far because it is an extremely mature option (when you consider its previous life as Fabric), and it is a go-to on many projects.

http://docs.pyinvoke.org/en/1.2/concepts/configuration.html#setup
http://docs.pyinvoke.org/en/1.2/getting-started.html#declaring-pre-tasks
http://docs.pyinvoke.org/en/1.2/concepts/invoking-tasks.html#how-tasks-run

---

It's just poetry run though. You could use whatever you want.

Using doit would the least difference from 'hardcoding a buncha strings to invoke as commands', yet like invoke, it has a clear path forward once your project is not "just" simple anymore. And we don't overload poetry with it forcing choices about how "tasks" should look. It's a modular approach.

http://pydoit.org/usecases.html#simplify-cumbersome-command-line-calls
https://github.com/pydoit/doit#sample-code
http://pydoit.org/tutorial_1.html#doit-task
http://pydoit.org/tasks.html#task-selection
http://pydoit.org/task_creation.html#custom-task-definition
http://pydoit.org/usecases.html#speed-up-by-parallel-task-execution

lint = "black --check ."

👇

# dodo.py
def task_lint():
    return {'actions': ['black --check .']}

poetry run doit lint would do this currently; then after #940 it'd be poetry doit lint - poetry doit lint is no less clean than poetry task lint so it's quite compatible with the proposed usages in this thread

---

Here is another example, implementing all of the OP's tasks with doit in a way similar to the "flat hardcoded aliases" idiom that you'd have if you inlined all this stuff to the TOML.

aliases = dict(
    lint    = ["black --check **/*.py"],
    test    = ["pytest tests/"],
    dev     = ["FLASK_APP=app.py flask run"],
    start   = ["gunicorn app.wsgi --log-file -"],
    clean   = ["poetry run manage.py flush", "rm -rf media"]
)

# small bit of boilerplate, this is a way that doit lets us generate task functions.
# http://pydoit.org/task_creation.html#custom-task-definition
class Aliases(object):
    @staticmethod
    def create_doit_tasks():
        return (({'actions': actions, 'basename': name, 'targets': [name]}) for name, actions in aliases.items())

Even though there is that bit of boilerplate, don't lose sight of the important part: once any one of your tasks outgrows hardcoded-string-aliases, you can use the full features of your chosen task-runner.

Personally I don't care about 'one-liners' so I'd make them all "real" tasks from the start. But that's the thing ... I can make the choices for my project, and you can make your choices.

---

In both examples given, invoke and doit, you have the tools to have tasks depend on each other, to run 'as needed', to watch your files and re-run (inotify), and so on and so on. doit has features where it can parallelize task execution, but keeps sequential dependencies running in parallel within that.

Yes Poetry can, and should be, opinionated about some things. This is an opinionated stance too, and it encourages users of poetry to keep their dev tasks cohesive. That means you don't have a split between your stringy tasks and your "real" tasks when your needs get complicated.

---

TLDR ... modular / separation of concerns / tools doing their primary things well. Poetry doesn't have to reinvent any wheels for this -- or at least we can address #940 now (simple) and consider doing other things later.

[actionless]

it seems the people who proposing alternative solutions (make, pyinvoke, pydoit, etc) not following the main motivation behind this idea -- to avoid having extra dependencies or extra files in project for this more than trivial purpose

[funkyfuture]

i don't see any value in this proposal for poetry, but only maintenance cost.

[maxsu]

Can we merge this code please? This feature is a big part of why I fell in love with NPM/Yarn, and I really need it to help onboard my team with a minimum of cognitive burden.

No, I will not be teaching them to use make. Ever.

Anyone saying to use other tools please ask yourself whether you've actually made a PR on this project, and if the answer is no, go ahead and defer to and support the folks who have contributed code or have a genuine vision for what this project can do if it defies the pythonic status quo.

The old pythonic ways lead to stagnant, schizophrenic designs, high cognitive loads, and incredibly slow progress. To wit, we have been flogging this issue for over half a year using meaningless and counterproductive rationale such as pythonicity and "do one thing and do it well", when in fact this feature and a complementary cluster of others were already solved and integrated into a coherent whole with resounding success in another community over 5 years ago. If you'd like a more genuine opportunity to contribute, stop pulling the handbrake on this project and go triage issues on the tools you're recommending. I assure you they exist. And are miserable, if you dare face them.

To conclude my screed, @sdispater are there any gaps preventing the integration of the current PR? If so, what are they? Can we move forward here? I'd really like to show this feature to the team :)

[pmav99]

old
stagnant,
schizophrenic designs
high cognitive loads
incredibly slow progress
flogging
meaningless and counterproductive rationale

@maxsu Regardless of the issue at hand, reserving this sort of toxicity for your self, or, if that is not possible, to communities that may welcome such behaviour would be a great first step towards contributing.

[brycedrennan]

@maxsu

It would be nice to have a single tool that could “do it all”, I’m not opposed to that in principle but I am opposed to adding features when core functionality isn’t complete. Last I checked (admittedly a few months ago) there were several blockers to rolling poetry out in my corporate environment. (#524 #697 #800).

NPM is a great example. It tried to do a lot and was so bad at its core functionality that it had to be replaced by a Yarn. Not the kind of “success” I’d hope for poetry.

I also think some people here are underestimating the complexity of task runners. It’s easy to get the simple cases working but we will inevitably also want the more complex cases working. That’s why there are whole libraries to deal with solving that problem.

As with any open source software of decent size, there are lots of competing visions for what the software “should” be. It’s generally valuable to have respectful debates about the merits of new features. It’s up to the maintainers, as informed by the input of everyone, to decide what fits the vision. I don’t think it’s productive to tell people with differing opinions they aren’t “genuinely” contributing.

I’m hesitant to post my contact info here but it’s pretty easy to find. Id be happy to learn more about your python pain points (outside of this thread) and see if we can find ways to make things easier for your team (for free). I do have a bit of experience with this sort of thing. Let me know!

[sdispater]

Poetry is a package and dependency manager, not a task manager. This feature is beyond the scope and the original purpose of Poetry and will likely never be integrated into it. I want to keep Poetry simple and intuitive so I carefully think before adding new features to the core codebase since the weight of maintaining it will fall on the shoulders of the maintainers.

That being said, when the planned plugin system is fully implemented (which will be after the release of the 1.0 version) anyone will be free to add features via plugins. This is the best tradeoff I can find and will hopefully help build an ecosystem around the core project and keep its simplicity.

@maxsu I am grateful of anyone willing to take some of their free time to make a PR for Poetry. I really do. However, that does not mean I will blindly accept any PR, especially when they add new features, if they do not align with the purpose of the project. I know this can be sometime disappointing to see your PR rejected after spending time on it but as a project maintainer it's my responsibility to determine what's best for the project. And that means, sometimes, to turn down good and thorough suggestions like this one.

@michaeldel Thanks a lot for taking the time to make this PR! I really appreciate it. However, for the reasons mentioned above, I don't think this align with the purpose of Poetry. Nonetheless, like I said this would be a perfect fit for a plugin once the plugin system is ready. So keep an eye on the project for when this will be announced.

[nat-n]

Since this PR probably still get some google traffic I'd like to share a solution I'm working on that solves more or less that same problem as this PR.

Poe the Poet a task runner that works well with poetry.

Basic usage is quite similar to proposal in this PR, but it also supports defining tasks using shell syntax, as references to python functions similar to scripts, as well as composing tasks into sequences.

Besides lightweight configuration contained in the pyproject.toml, one key advantage over alternatives like make is that you can easily pass extra arguments to tasks without extra configuration.

Example configuration with different task types:

[tool.poe.tasks]
test       = "pytest --cov=poethepoet"                                # simple command based task
mksandwich = { script = "my_package.sandwich:build" }                 # python script based task
tunnel     = { shell = "ssh -N -L 0.0.0.0:8080:$PROD:8080 $PROD &" }  # shell script based task

Example invocation:

# extra argument is passed to the underlying pytest command
poe test -v   

# If poethepoet is added as a dev-dependency
poetry run poe tunnel

The intention is to eventually have comparable functionality to make wrt task composition, conditional execution, and command line completion.

EDIT: also I intend to support poe as a poetry plugin in addition to a standalone tool, once that API is available.

[JosXa]

@nat-n I assume you will turn this into a plugin as soon as that system is ready, correct? 🚀

[acalvino4]

I know a lot have others have already expressed similar sentiments, but I'll just throw in my two cents:

1. It is beyond obvious that this is an immensely desired feature. Every comment expressing support for the idea in this thread has many, many thumbs-up and hearts, while comments suggesting alternatives have been met with the opposite.
2. Bringing up the fact that there are alternatives is a moot point. Yes, we can figure out an alternative way, but the whole point of developer tooling is to make our lives easier. If the alternative ways are not as easy as the proposed way, then they are not a counterpoint to the proposal. The fact is that using other tools to implement this functionality introduces unneeded complexity into a project. Therefore, in the interest of making developer lives easier, integrating a simple version of this into the ecosystem makes perfect sense.
3. On the point of restricting features for the sake of maintenance, I get you - but since it is clear that there are a lot of people invested in this functionality to the point where they are willing to submit PR's, I don't think that it should be out of the question. The community is obviously invested. (Granted, I've never been in charge of maintaining an open source project, so I don't know how much it involves and this point isn't a hill I'm going to die on)
4. Having an implementation doesn't mean it has to be full-featured. In other words, accepting an initial proposal doesn't mean that you have to accept every improvement and use case out there. For example, 81 people explicitly expressed approval for the initial proposal on this thread, with only 1 dissenter. If merging this makes all those people happy (and all the others who just didn't hit the thumbs up or who never came across this thread), then let's do it, and if some fringe extended use case comes up in an issue later that only 2 people are interested in, well there's no obligation to implement that....
5. In my mind the simplest version of this is one where the command that is aliased in the tasks section gets passed as is to a shell running in the virtual environment. Making those scripts cross-platform or what-not is up to the developer as needed. I don't think anyone is asking for poetry to be a task runner in the sense that gulp is...

[paulcruse-syn]

This missing feature is honestly the only thing holding me back from moving my team over from pipenv.

The fact that this tool doesn't have a simple npm run <command> or pipenv run <command> equivalent, is little bit weird at this point, given ubiquity of the use-case and the demand for it.

I honestly don't care if its run, task,command or whatever, but I don't believe the answer is to require another dependency and plugin. Though I will admit, other tool use the command scripts and it does increase the learning curve that this tool has taken that term and made it mean something different.

I know this is already closed, but please add this feature... 😄

[nat-n]

@nat-n I assume you will turn this into a plugin as soon as that system is ready, correct? 🚀

Poe the poet can now be used as a plugin, including support for enhancing poetry's own commands with hooks!

@paulcruse-syn I can understanding the preference for an "all in one" solution, but the plugin workflow is pretty smooth. Though it's still only supported in the beta release of poetry.

[neersighted]

This is an issue with Homebrew, and not with Poetry itself. We do not provide support for downstream repackaging, and indeed, they appear to not install Poetry in a way where poetry self can be used. poetry self is only expected/tested to work when using the recommended install methods.

This closed PR is not a good place to discuss this or request support -- I'd suggest getting support through Discord/Discussions, and #2496 as the correct place to express support for this proposed feature. Please refrain from any 👍/me too spam -- reacts suffice for that, but please do chime in if you have something novel to add that has not been said already.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.