-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Standardize the entry-point for Julia execution #50974
Conversation
Not completely true. You can bundle multiple executables with an app and each executable is given a mapping to the Julia function it invokes when started. But that could be changed to give arguments to the new main function or something. |
Anything you'd want us to change in StaticCompiler? (not that we have a ton of flexibility but..) |
In AVRCompiler.jl, I've established the convention of having a module-centric |
Yeah, at the low level we're pretty much tied to |
Addendum to the above: The reasoning for requiring a module in AVRCompiler.jl and not directly an entry function is because that way I can get a reference to global variables that may need interning as well (doesn't help a lot, but feels nicer to be able to check from there instead of always chasing through pointers..). Additionally, because there is no compilation of additional code on microcontrollers, having a "This is the main module of the compiled binary" makes sense semantically. |
Not immediately. There's longer term thoughts on unifying all the modes of app creation behind a new user experience. This is part of the thinking in that direction, but anything new would start out as a new package first (that may depend on PackageCompiler/StaticCompiler/etc).
I think making
I think it'd be fine to wrap that into an array abstraction. I deliberately left |
base/client.jl
Outdated
elseif isassigned(REPL_MODULE_REF) | ||
ret = REPL_MODULE_REF[].main(ARGS) | ||
else | ||
error("No entry point defined and REPL not loaded.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like an API break, since we expect Julia to work correctly to run scripts via exec_options
particularly when the REPL is not loaded.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm fine with falling back to exec_options for the time being, though we may want to re-arrange later once we figure out what the compiler interface looks like.
src/jlapi.c
Outdated
JL_TRY { | ||
size_t last_age = ct->world_age; | ||
ct->world_age = jl_get_world_counter(); | ||
jl_apply(&start_client, 1); | ||
ret = jl_unbox_int32(jl_apply(&start_client, 1)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should probably have a type-assert here first
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_start
is internal, so I though it was fine, but might as well be defensive about it.
Capturing some discussion from this morning:
|
That said, overall, I didn't hear any strong objections to this, so I'm gonna go ahead and clean this up and get it ready for merging. |
While that's true in general, requiring it makes things harder because it requires array & String support in my runtime. Having ARGS be optional means I could conceivably migrate to the standardized approach with relatively little effort. I don't know whether a special type for ARGS could accomplish the same thing though - it depends on the specifics. |
No it doesn't the custom empty array type can be trivial without runtime requirements. |
If that's the case, then I'd be happy - I just know that dealing with arrays and strings in regular Julia code is a bit of a pain, due to their interactions with the runtime due to their dynamic lengths. The idea for making it optional is half inspired by C, which doesn't mandate the argc/argv structure either (see here). A specific singleton type for "empty ARGS" should be workable, though I suspect that would be just the same as having no arguments passed at all. |
One thing I'd like to bring up related to this topic is that we don't currently have an ergonomic way to write a script that takes advantage of the pkgimages stuff. If we're going to discuss a common entrypoint, we should at least try and keep our eyes open for designs that would make it easier to streamline this kinda clunky process for having good precompilation in scripts. Currently what you'd need to do is to create a local package project, dev it, then write all your script functions into that package project, and then write precompile directives into the package, and then finally write a script which basically just does Additionally, the binary artifact (pkgimage) associated with |
It also doesn't make really sense for (standalone) applications that you'd like to distribute without a dedicated/existing Julia installation. |
I think that is mostly orthogonal to this proposal and more a question of more granular caching, which has been talked about but I don't think there are any concrete plans. |
This is a bit of a straw-man proposal (though I think mergable if people agree) to standardize the execution entrypoint for Julia scripts. I think there's at least four different ways that people might run a script: - As `julia main.jl` - As a PkgCompiler sysimage, then calling the main entry point - As a PkgCompiler "app", with the magic `julia_main` function - As a StaticCompiler product with an explicit entrypoint specified on the API. The main problem I have with all of these variants is that they're all different and it's kind of a pain to move between them. Here I propose that we standardize on `Main.main(ARGS)` as the entrypoint for all scripts. Downstream from that proposal, this PR then makes the following changes: 1. If a system image has an existing `Main.main`, that is the entry point for `julia -Jsysimage.so`. 2. If not, and the sysimage has a REPL, we call REPL.main (we could handle this by defaulting `Main.main` to a weak import of `REPL.main`, but for the purpose of this PR, it's an explicit fallback. That said, I do want to emhpasize the direction of moving the REPL to be "just another app". 3. If the REPL code is called and passed a script file, the REPL executes any newly defined Main.main after loading the script file. As a result, `julia` behaves the same as if we had generated a new system image after loading `main.jl` and then running julia with that system image. The further downstream implication of this is that I'd like to get rid of the distinction between PkgCompiler apps and system images. An app is simply a system image with a `Main.main` function defined (note that currently PkgCompiler uses `julia_main` instead).
Rebased, dropped RFC, addressed review comments, added docs, news, a couple simple tests. I think this is essentially ready. |
NEWS.md
Outdated
* The entry point for Julia has been standardized to `Main.main(ARGS)`. When julia is invoked to run a script or expression | ||
(i.e. using `julia script.jl` or `julia -e expr`), julia will subsequently run the `Main.main` function automatically if | ||
such a function has been defined. This is identended to unify script and compilation workflows, where code loading may happen | ||
in the compiler and execution of `Main.main` may happen in the resulting executable. For interactive use, there is no semantic | ||
difference between defining a main function and executing the code directly at the end of the script. ([50974]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* The entry point for Julia has been standardized to `Main.main(ARGS)`. When julia is invoked to run a script or expression | |
(i.e. using `julia script.jl` or `julia -e expr`), julia will subsequently run the `Main.main` function automatically if | |
such a function has been defined. This is identended to unify script and compilation workflows, where code loading may happen | |
in the compiler and execution of `Main.main` may happen in the resulting executable. For interactive use, there is no semantic | |
difference between defining a main function and executing the code directly at the end of the script. ([50974]) | |
* The entry point for Julia has been standardized to `Main.main(ARGS)`. When Julia is invoked to run a script or expression | |
(i.e. using `julia script.jl` or `julia -e expr`), Julia will subsequently run the `Main.main` function automatically if | |
such a function has been defined. This is intended to unify script and compilation workflows, where code loading may happen | |
in the compiler and execution of `Main.main` may happen in the resulting executable. For interactive use, there is no semantic | |
difference between defining a `main` function and executing the code directly at the end of the script. ([50974]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is talking about the behavior of the binary, not the semantics of the language, so lower case is the appropriate style.
Can/should this catch when two packages define & export a |
I think that can be a tweak to the existing error message, but I also think that can be a separate PR. |
just to make sure I understand --- when loading the symbol via that is
prints only |
correct |
What we were proposing is a global called |
Does that mean if someone has a script
that |
Yes. But I think actually both functions would run, Magically running user-code is the wrong thing to do in a 1.X release, and it should not even be up for debate, opt-outs, or hacky work-arounds. I think the two options need to be:
|
Could the opt-in be as simple as declaring compat |
Another option that was brought up: Change the name of the driver executable. The compiler driver was gonna be |
I compiled this commit from source. I can confirm what @odow mentioned, it runs I know this thread is about what to do instead, but I wanted to say that I find it surprising that it runs And given that it does run ╭─ ~/g/j/HelloWorld - 8e14322
╰ cat src/HelloWorld.jl
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────
│ File: src/HelloWorld.jl
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────
1 │ module HelloWorld
2 │
3 │ export main
4 │ main(ARGS) = println("Hello World!")
5 │
6 │ end # module HelloWorld
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────
╭─ ~/g/j/HelloWorld - 8e14322
╰ ../julia --startup-file=no --project -e "using HelloWorld"
Hello World!
╭─ ~/g/j/HelloWorld - 8e14322
╰ ../julia --startup-file=no --project -e "import HelloWorld" It also only runs I get that this works by checking if Also, doesn't this mean you can't have a script that loads multiple packages with This is what happens when I test this: ╭─ ~/g/j/HelloWorld - 8e14322
╰ cat test.jl
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────
│ File: test.jl
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────
1 │ module testing
2 │
3 │ export main
4 │ main(ARGS) = println("testing")
5 │ end
6 │
7 │ using .testing
8 │
9 │ using HelloWorld
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────
╭─ ~/g/j/HelloWorld - 8e14322
╰ ../julia --startup-file=no --project test.jl
WARNING: both HelloWorld and testing export "main"; uses of it in module Main must be qualified Thinking about this more, this does feel like it really would benefit from a standardized entrypoint treatment via For beginners writing a script, it doesn't get simpler than it is already, i.e.: println("hello world") For anyone generating a package using |
I'm starting to warm up to the From the POV of static compilation, using |
Another idea could be a different file extension for scripts that have At some level I find the idea of an entry-point function almost a bit at odds with the current idea of what a script in Julia is. Isn't the entry point just the global code in a Julia file, essentially? The whole concept that one might have a script file with lots of global code (that isn't type/function/global var definitions) that runs first, and then another function with a specific name that runs in a second phase seems a bit weird to me in general... Maybe something like a |
I think having to specify the entrypoint in the Project.toml sounds like way too much friction and jumping around. You then cant just write a script, you have to create a project for every script which is pretty crappy IMO. |
Yes, I would say Project.toml or any other file specification of an entry point is a non-starter for discussion. It would make it way too annoying to use.
PackageCompiler already has support for this, it wouldn't really make things easier. |
Do we really need/want script style code to be shared as a fully compiled executable though? I think it's ok to seperate the two use cases - script style is just "run IMO scripts should be kept small & self-contained, and once your script grows to a size that sharing it as a naively-runnable thing becomes a burden, that's exactly the point where you'd want to have dependency management, versioning & a well-defined entry point anyway. Placing that in |
Another advantage of placing it in a For example, python has entrypoints that can be defined in [project.scripts]
my-cli = "my_package.my_module:main_cli" If someone import sys
from my_package.my_module import main_cli
sys.exit(main_cli()) Rust has something similar where you can define a [[bin]]
name = "my-cli"
path = "src/main.rs" and this builds a target called In both these cases it is a few lines of toml code and little to no friction in my opinion. I agree with @Seelengrab; from my experience a lot of Julia users tend to work in If this kind of "entrypoint" feature were part of a $ julia -e 'using Pkg.add("Genie")'
$ genie server imo, one of the reasons Python is so popular for writing command line tools is that it makes it extremely easy to be able to
In the example of |
I really like this. As a user this would certainly give me the impression that the execution of julia code may be different depending on how I invoke it. Separating the drivers implies to users (new and old) that executing a Julia file with I think introducing entry point changes with |
A new driver sounds reasonable, whether it's named |
Following the discussion in #50974, it became clear that there is significant objection to changes to the behavior of the julia CLI driver. Some commenters found changes to the behavior acceptable if they were unlikely to impact existing code (e.g. in the case of #50974 using `__main__` instead of `main`), while other were worried about the reputational risks of even changing behavior in the corner case. In subsequent discussion it became relatively clear that the only way forward that did not raise significant objection was to introduce a new CLI driver for the new behavior. This may seem a bit drastic just for the change in #50974, but I actually think there's a number of other quality-of-life improvements that this would enable us to do over time, for example: - Autoloading/caching of all packages in the manifest - auto-selection of julia versions (integrate juliaup?) In addition, it doesn't seem so bad to have some CLI design flexibility to make sure that the `juliac` driver is aligned with what we need. This PR is the minimal infrastructure to add the new drivers. In particular, it does the following: 1. Adds two new cli drivers, `juliax` and `juliac`. At the moment, `juliac` is a placeholder and just errors, while `juliax` behaves the same as `julia` (except to error on the deprecated `--math-mode=fast`, just as an example of a behavior difference). 2. Documents that the behavior of `julia` (but not `juliax` or `juliac`) is pat of the julia public API. 3. Switches the cli mode based on the argv[0] of the binary. I.e. all three binaries are identical, except for their name, the same way that, e.g. `clang` and `clang++` are the same binary just with different names. On Unix systems, these are symlinks. On windows, separate copies of the same (small) binary. There is a fallback cli option `--cli-mode` that can be used in case the argv[0] detection is not available (e.g. for some fringe embedded use cases). 4. There is currently no separate `-debug` version of the new drivres. My intention is to make this dependent on the ordinary debug flags, rather than having a separate driver. Once this is merged, I intend to resubmit #50974 (chaning `juliax` only), and then finish and hook up `juliac` shortly thereafter.
#51417 for the new cli drivers, which emerged as the preferred option in subsequent discussion. |
Was this breaking? I see first a "minor change" label, then label changed, and it's a long discussion, I think things changed here (why the new juliax PR), so it's best to change the label back if it wasn't actually breaking. In case people are searching through, it seems odd to see breaking (merged) PR, not on the 2.0 milestone. [Which has been renamed recently to "potential 2.0", I'm not sure, is it now more (or less) potential?] |
This was first declared a minor change and then merged. Afterwards, it was decided that this change is breaking; thus it was reverted in #51196. |
As they say, if at first you don't succeed, try again, then try again, add an extra layer of indirection and take a little bit of spice from every other idea and you've got yourself a wedding cake. Or something like that, I don't know - at times it felt like this cake was getting a bit burnt. Where was I? Ah yes. This is the third edition of the main saga (#50974, #51417). In this version, the spelling that we'd expect for the main use case is: ``` function (@main)(ARGS) println("Hello World") end ``` This syntax was originally proposed by `@vtjnash`. However, the semantics here are slightly different. `@main` simply expands to `main`, so the above is equivalent to: ``` function main(ARGS) println("Hello World") end @main ``` So `@main` is simply a marker that the `main` binding has special behavior. This way, all the niceceties of import/export, etc. can still be used as in the original `Main.main` proposal, but there is an explicit opt-in and feature detect macro to avoid executing this when people do not expect. Additionally, there is a smooth upgrade path if we decide to automatically enable `Main.main` in Julia 2.0.
As they say, if at first you don't succeed, try again, then try again, add an extra layer of indirection and take a little bit of spice from every other idea and you've got yourself a wedding cake. Or something like that, I don't know - at times it felt like this cake was getting a bit burnt. Where was I? Ah yes. This is the third edition of the main saga (#50974, #51417). In this version, the spelling that we'd expect for the main use case is: ``` function (@main)(ARGS) println("Hello World") end ``` This syntax was originally proposed by `@vtjnash`. However, the semantics here are slightly different. `@main` simply expands to `main`, so the above is equivalent to: ``` function main(ARGS) println("Hello World") end @main ``` So `@main` is simply a marker that the `main` binding has special behavior. This way, all the niceceties of import/export, etc. can still be used as in the original `Main.main` proposal, but there is an explicit opt-in and feature detect macro to avoid executing this when people do not expect. Additionally, there is a smooth upgrade path if we decide to automatically enable `Main.main` in Julia 2.0.
As they say, if at first you don't succeed, try again, then try again, add an extra layer of indirection and take a little bit of spice from every other idea and you've got yourself a wedding cake. Or something like that, I don't know - at times it felt like this cake was getting a bit burnt. Where was I? Ah yes. This is the third edition of the main saga (#50974, #51417). In this version, the spelling that we'd expect for the main use case is: ``` function (@main)(ARGS) println("Hello World") end ``` This syntax was originally proposed by `@vtjnash`. However, the semantics here are slightly different. `@main` simply expands to `main`, so the above is equivalent to: ``` function main(ARGS) println("Hello World") end @main ``` So `@main` is simply a marker that the `main` binding has special behavior. This way, all the niceceties of import/export, etc. can still be used as in the original `Main.main` proposal, but there is an explicit opt-in and feature detect macro to avoid executing this when people do not expect. Additionally, there is a smooth upgrade path if we decide to automatically enable `Main.main` in Julia 2.0.
As they say, if at first you don't succeed, try again, then try again, add an extra layer of indirection and take a little bit of spice from every other idea and you've got yourself a wedding cake. Or something like that, I don't know - at times it felt like this cake was getting a bit burnt. Where was I? Ah yes. This is the third edition of the main saga (#50974, #51417). In this version, the spelling that we'd expect for the main use case is: ``` function (@main)(ARGS) println("Hello World") end ``` This syntax was originally proposed by `@vtjnash`. However, the semantics here are slightly different. `@main` simply expands to `main`, so the above is equivalent to: ``` function main(ARGS) println("Hello World") end @main ``` So `@main` is simply a marker that the `main` binding has special behavior. This way, all the niceceties of import/export, etc. can still be used as in the original `Main.main` proposal, but there is an explicit opt-in and feature detect macro to avoid executing this when people do not expect. Additionally, there is a smooth upgrade path if we decide to automatically enable `Main.main` in Julia 2.0.
I would like to suggest a change to the doc to recommend users define |
``` | ||
|
||
However, note that the current best practice recommendation is to not mix application and reusable library | ||
code in the same package. Helper applications may be distributed as separate pacakges or as scripts with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pacakges
-> packages
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the catch, but commenting on old (merged and subsequently reverted PRs doesn't really help). If the issue is present in current documentation, please submit a new PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry about that. I checked master
and it has been fixed already!
This is a bit of a straw-man proposal (though I think mergable if people agree) to standardize the execution entrypoint for Julia scripts. I think there's at least four different ways that people might run a script:
julia main.jl
julia_main
functionThe main problem I have with all of these variants is that they're all different and it's kind of a pain to move between them. Here I propose that we standardize on
Main.main(ARGS)
as the entrypoint for all scripts. Downstream from that proposal, this PR then makes the following changes:Main.main
, that is the entry point forjulia -Jsysimage.so
.Main.main
to a weak import ofREPL.main
, but for the purpose of this PR, it's an explicit fallback. That said, I do want to emhpasize the direction of moving the REPL to be "just another app".julia
behaves the same as if we had generated a new system image after loadingmain.jl
and then running julia with that system image.The further downstream implication of this is that I'd like to get rid of the distinction between PkgCompiler apps and system images. An app is simply a system image with a
Main.main
function defined (note that currently PkgCompiler usesjulia_main
instead).