Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes: #3862; Added an example Python Support case study #3882

Merged
merged 19 commits into from
Nov 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion docs/modules/ROOT/pages/extending/new-language.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,28 @@ programming language toolchain. It would take significantly more work to flesh o
the featureset and performance of `TypeScriptModule` to be usable in a real world
build. But this should be enough to get you started working with Mill to add support
to any language you need: whether it's TypeScript or some other language, most programming
language toolchains have similar concepts of `compile`, `run`, `bundle`, etc.
language toolchains have similar concepts of `compile`, `run`, `bundle`, etc.


== Python Support

This section demonstrates how to integrate `Python` support into `Mill`.
We will define a simple `PythonModule` trait that can resolve dependencies,
perform type checking on local code, and optimize the final bundle.

`Note`: This integration is for `educational purposes only`, showcasing common technique
used in building language toolchains, and is not intended for production use.

include::partial$example/extending/newlang/5-hello-python.adoc[]

include::partial$example/extending/newlang/6-python-modules.adoc[]

include::partial$example/extending/newlang/7-python-module-deps.adoc[]

include::partial$example/extending/newlang/8-python-libs-bundle.adoc[]



As mentioned, The `PythonModule` examples here demonstrate
how to add support for a new language toolchain in Mill.
A production-ready version would require more work to enhance features and performance.
114 changes: 114 additions & 0 deletions example/extending/newlang/5-hello-python/build.mill
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
// == Python Integration in Mill

// This Example shows how to integrate https://www.python.org[Python] into a Mill build, enabling Python script compilation.
// Mill does not come bundled with Python support, so we will set up integration using `python3`, which is generally pre-installed.

// === Python initialization Setup
//
// First, The `setup` task creates a Python virtual environment and installs `mypy` for type-checking.

// `mypy` verifies type correctness using Python's `type hints`, helping to catch errors in development.

// Guides:
// https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments[Python virtual environment]
// & https://mypy.readthedocs.io/en/stable[mypy]

package build
import mill._

def setup: T[PathRef] = Task {
val pythonVenv = Task.dest / "venv" / "bin" / "python3"

os.call(("python3", "-m", "venv", Task.dest / "venv"))
os.call((pythonVenv, "-m", "pip", "install", "mypy"))

PathRef(pythonVenv)
}

// === Defining our Sources

// The `sources` task specifies the directory for Python source files (`src` folder).
// This setup facilitates organizing and accessing Python files needed for building, testing, or analysis.

def sources: T[PathRef] = Task.Source(millSourcePath / "src")

// === Type Checking

// The `typeCheck` task verifies that the code in the main Python file passes type checks.
// It checks for errors before execution to ensure a reliable setup for running Python scripts.

def typeCheck: T[PathRef] = Task {
val pythonVenv = setup().path

os.call(
(pythonVenv, "-m", "mypy", "--strict", sources().path / mainFileName()),
stdout = os.Inherit
)

PathRef(pythonVenv)
}

// At this point, we have a minimal working build, with a build graph that looks like this:
//
// ```graphviz
// digraph G {
// rankdir=LR
// node [shape=box width=0 height=0 style=filled fillcolor=white]
// setup -> typeCheck
// sources -> typeCheck
// mainFileName -> typeCheck
// }
// ```

// Here is the `main.py` file
/** See Also: src/main.py */
himanshumahajan138 marked this conversation as resolved.
Show resolved Hide resolved

// === Running

// The `mainFileName` task defines the name of the main Python script (`main.py`).

// The `run` function runs the main file with user-provided command-line arguments.
// It uses the virtual environment's Python interpreter to execute the script, with output displayed in the console.

def mainFileName: T[String] = Task { "main.py" }
def run(args: mill.define.Args) = Task.Command {
val pythonVenv = typeCheck().path

os.call(
(pythonVenv, sources().path / mainFileName(), args.value),
stdout = os.Inherit
)
}

// Note that we use `stdout = os.Inherit` since we want to display any output to the user,
// rather than capturing it for use in our command.

// ```graphviz
himanshumahajan138 marked this conversation as resolved.
Show resolved Hide resolved
// digraph G {
// rankdir=LR
// node [shape=box width=0 height=0 style=filled fillcolor=white]
// setup -> typeCheck -> run
// sources -> typeCheck
// mainFileName -> typeCheck
// sources -> run
// mainFileName -> run
// mainFileName [color=green, penwidth=3]
// run [color=green, penwidth=3]
// }
// ```

// Running run command will return the result to the console.

/** Usage

> ./mill typeCheck
Success: no issues found in 1 source file

> ./mill run Mill Python
Hello, Mill Python!
15

*/

// This completes a basic `Python integration in Mill`.
// Next steps could involve transforming this into a `reusable PythonModule`
6 changes: 6 additions & 0 deletions example/extending/newlang/5-hello-python/src/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import sys
def add(a: int, b: int) -> int: return a + b
def main() -> None: print("Hello, " + " ".join(sys.argv[1:]) + "!")
if __name__ == "__main__":
main()
print(add(5, 10)) # Error Example: add("5", 10) will cause a TypeError
123 changes: 123 additions & 0 deletions example/extending/newlang/6-python-modules/build.mill
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
// == Re-usable PythonModule
// This example illustrates the `PythonModule` trait for managing Python scripts within multiple Mill objects.

// ==== Using Same PythonModule for Multiple Objects

// `PythonModule` automates essential tasks in Python project management within Mill,
// such as setting up virtual environments, managing source files, performing type checks, and running scripts.

package build
import mill._

/** `PythonModule`: Trait for basic Python support in Mill */
trait PythonModule extends Module {

def sources: T[PathRef] = Task.Source(millSourcePath / "src")
def mainFileName: T[String] = Task { "main.py" }

def setup: T[PathRef] = Task {
val pythonVenv = Task.dest / "venv" / "bin" / "python3"

os.call(("python3", "-m", "venv", Task.dest / "venv"))
os.call((pythonVenv, "-m", "pip", "install", "mypy"))

PathRef(pythonVenv)
}

def typeCheck: T[PathRef] = Task {
val pythonVenv = setup().path

os.call(
(pythonVenv, "-m", "mypy", "--strict", sources().path / mainFileName()),
stdout = os.Inherit
)

PathRef(pythonVenv)
}

def run(args: mill.define.Args) = Task.Command {
val pythonVenv = typeCheck().path

os.call(
(pythonVenv, sources().path / mainFileName(), args.value),
stdout = os.Inherit
)
}

}

// ==== Example

// Below are three objects extending `PythonModule`, each representing a unique configuration.

object foo extends PythonModule {
override def mainFileName: T[String] = Task { "foo.py" }
object bar extends PythonModule {
override def mainFileName: T[String] = Task { "bar.py" }
// Inherits PythonModule features
}
}

object qux extends PythonModule {
override def mainFileName: T[String] = Task { "qux.py" }
// Independent, but uses PythonModule methods
}

// We have used three different Python Scripts `foo/src/foo.py`, `foo/bar/src/bar.py`, `qux/src/qux.py`

/** See Also: foo/src/foo.py */
/** See Also: foo/bar/src/bar.py */
/** See Also: qux/src/qux.py */
himanshumahajan138 marked this conversation as resolved.
Show resolved Hide resolved

// Use the following commands to run each module, displaying unique outputs per configuration:

/** Usage

> ./mill foo.run Mill
Hello, Mill Foo!

> ./mill foo.bar.run Mill
Hello, Mill Foo Bar!

> ./mill qux.run Mill
Hello, Mill Qux!

*/

// The Final working build, with a build graph that looks like this:

// ```graphviz
// digraph G {
// rankdir=LR
// node [shape=box width=0 height=0 style=filled fillcolor=white]
// subgraph cluster_3 {
// style=dashed
// label=qux
// "qux.setup" -> "qux.typeCheck" -> "qux.run"
// "qux.sources" -> "qux.typeCheck"
// "qux.mainFileName" -> "qux.typeCheck"
// "qux.sources" -> "qux.run"
// "qux.mainFileName" -> "qux.run"
// }
// subgraph cluster_1 {
// subgraph cluster_2 {
// style=dashed
// label=bar
// "bar.setup" -> "bar.typeCheck" -> "bar.run"
// "bar.sources" -> "bar.typeCheck"
// "bar.mainFileName" -> "bar.typeCheck"
// "bar.sources" -> "bar.run"
// "bar.mainFileName" -> "bar.run"
// }
// style=dashed
// label=foo
// "foo.setup" -> "foo.typeCheck" -> "foo.run"
// "foo.sources" -> "foo.typeCheck"
// "foo.mainFileName" -> "foo.typeCheck"
// "foo.sources" -> "foo.run"
// "foo.mainFileName" -> "foo.run"
// }
// }
// ```

// Next, we will look at how to Manage `Module Dependencies` for `Python` in `Mill`.
4 changes: 4 additions & 0 deletions example/extending/newlang/6-python-modules/foo/bar/src/bar.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
import sys
def main() -> None: print("Hello, " + " ".join(sys.argv[1:]) + " Foo Bar!")
if __name__ == "__main__":
main()
4 changes: 4 additions & 0 deletions example/extending/newlang/6-python-modules/foo/src/foo.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
import sys
def main() -> None: print("Hello, " + " ".join(sys.argv[1:]) + " Foo!")
if __name__ == "__main__":
main()
4 changes: 4 additions & 0 deletions example/extending/newlang/6-python-modules/qux/src/qux.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
import sys
def main() -> None: print("Hello, " + " ".join(sys.argv[1:]) + " Qux!")
if __name__ == "__main__":
main()
Loading
Loading