Some improvements to pythonlib

[jodersky]

This is in view of #3928, specifically some essential things required for the first task, example/pythonlib/basic/.

You can look at commits individually if you like. The gist of this PR is to improve the scaffolding we currently have to unblock the previously mentioned tasks. It can be summarized by the following points (roughly each corresponding to one commit):

• make run task the default, following the same convention used in ScalaModule and JavaModule
• add a command to run an interactive REPL. This is named console, again following the conventions of scalalib
• rework the way source files are handled:
  • support multiple source directories, again following similar conventions of scalalib
  • instead of aggreting python scripts in one syntetic directory, keep them where they are and instead manipulate PYTHONPATH or pass in parameters to various directories where appropriate
• define a module for unit tests, similar to the TestModule of scalalib

[himanshumahajan138]

@jodersky TBH this is what i was looking for, Great !!!

Also i want to add something like the way i used Repl i think that is also good See #3986

And the way of testing is now professional and matching with other examples (i didn't liked my way of testing)

So i think i should wait till this PR gets Merged and after wards i will work further on #3986 till then let me copy your changes and work around with the same pattern

Thanks...

[jodersky]

@himanshumahajan138, that sounds good. I'll finish up test support, then we can hopefully merge, and you take it from there to finish the example/pythonlib/basic/ support of #3928.

I can continue on example/pythonlib/dependencies/ of the same issue then, if that's ok with you

[himanshumahajan138]

@jodersky ya that's great we will work together to bring the professional python first class support to mill

Team Work is the Dream Work !!!

[himanshumahajan138]

./mill mill.scalalib.scalafmt.ScalafmtModule/
./mill mill.javalib.palantirformat.PalantirFormatModule/
./mill mill.kotlinlib.ktlint.KtlintModule/

@jodersky these commands will fix the lint error automatically from the root

[jodersky]

Alright, I think this is ready for review now.

[lihaoyi]

Looks pretty good overall. Left some comments. @jodersky you mentioned you looked at how Bazel handles Python, if in that research you found anything relevant to this PR can summarize whatever you learned that's relevant in the PR description

[himanshumahajan138]

@jodersky i think we should work on linting before deps coz deps don't require much logical and backend code but linting is far harder and more complicated and in that case, experience is more required 😉

Rest is You's Choice...

[jodersky]

@jodersky i think we should work on linting before deps coz deps don't require much logical and backend code but linting is far harder and more complicated and in that case, experience is more required 😉

I'm not sure. I think dependencies are a very important feature, and they aren't all that easy when you consider that we'll need more than just building a pip command line.

For example, we'll need to deal with things like installing wheels. Also, there are a couple of "quality-of-life" features that I'd like to include:

• ability to specifcy dependencies via an existing requirements.txt file (this is inspired by pants' python support)
• on dependency conflicts, mention which modules requested the conflicting version (right now, since we're only building a pip command line, it's really hard to find out what module is the culprit)

[himanshumahajan138]

@jodersky i think we should work on linting before deps coz deps don't require much logical and backend code but linting is far harder and more complicated and in that case, experience is more required 😉

I'm not sure. I think dependencies are a very important feature, and they aren't all that easy when you consider that we'll need more than just building a pip command line.

For example, we'll need to deal with things like installing wheels. Also, there are a couple of "quality-of-life" features that I'd like to include:

• ability to specifcy dependencies via an existing requirements.txt file (this is inspired by pants' python support)
• on dependency conflicts, mention which modules requested the conflicting version (right now, since we're only building a pip command line, it's really hard to find out what module is the culprit)

Ok that's great afterall this all wil be available for mill 😊

[jodersky]

@lihaoyi, I think I addressed all your comments. Would you like to go over it once more? Regarding the python example, it will all be reworked by @himanshumahajan138.

[lihaoyi]

@jodersky looks great, I'll merge it and we can continue iterating on it after