Re-integrate jpm into janet #1444
Comments
|
I think having something like jpm be bundled with janet has merits. I'm not so sure about trying to bring jpm itself back in though. Possibly a new thing might be better? Haven't thought too deeply about it though. |
|
Once mentioned idea of moving jpm into spork and making spork part of the janet build and release sounds the best to me. |
I agree with that. jpm has usablility problems that make that not ideal. So current jpm issues as I see it:
I guess I don't think the entirety of This kind of flow is pretty common in repl-first languages, and avoids some of the annoying and complicated setup of jpm, where it needs to be installed system wide in a compatible way to Janet. Configuration would be done solely by where I also don't think "installation" into a tree should be that complicated by default - a git project that was installable this way could just be a project that had a Lastly, the make-style build system built into jpm can be nice but I think is something that could just be left in spork if people needed it - most projects I've seen really don't need it. Curious on peoples thought |
|
The hardest part is of course an opinionated C build system (which is actually what |
|
I like the ideas of having a few simple primitives for module management etc. in janet itself and keeping more complicated parts in spork to import as needed. |
...
Interested in this kind of decoupling along with having some key feature(s) make it into janet. Having build system bits be in spork or elsewhere might be nice for extensibility and experimentation by 3rd parties. |
|
An observation about the specific example repl session above... IIUC, it demos the idea of being able to install spork with just an installation of janet. ATM, I think a C compiler is required in the case of spork because it has native modules. Possibly the janet-built-in stuff can use a C compiler if one is detected and bail otherwise? |
So spork already actually contains some of the needed functionality for building C modules (see spork/cc) - my idea is that we would eventually have sport be able to self bootstrap with it's own utilities. EDIT: as far as installing jpm without a C compiler, also a possibility but a number of modules depend on other native modules like |
|
So I was able to a working version of the An example of what the spork hooks look like can be found here: https://github.com/janet-lang/spork/tree/bundle-hooks/hooks Usage would be something like the following: Curious on feedback here - I know this does not include all of the features that This could also be put into spork rather than Janet as there really isn't anything here that needs integration with |
|
I just tried it, and it seems to be working well for me. |
|
bakpakin wrote:
This sounds good to me. tionis wrote:
You mean this sort of thing? On a side note, |
|
Did some more work on the branch, and added All that said, I can see functionality here growing and growing and growing, and at some point a line needs to be drawn. Not only should the functionality be simple but genuinely useful, the implementation needs to be fairly simple. |
Tried out briefly -- seemed to work here.
I agree about these endpoint characteristics -- wonder how to get there though (^^; |
|
Tried it briefly on Windows. Encountered at least one issue with |
|
I haven't tested any of this on windows but I wouldn't be surprised if there are path issues. |
|
@bakpakin Is the idea that module authors will include hooks files in the module to run on build and install? Or is this just an aspect of the proof of concept that will have improved UX later on? I will say that I like the way modules all currently have a |
|
This is all proof of concept, having each "hook" be a function inside a single "project" file in some ways makes more sense. I will likely change over to this way, and was playing with this locally. However, one of the issues we ran into with JPM was how to evaluate such files before dependencies were installed - separate files avoids that problem. I'm hesitant to commit even to this whole idea since JPM does already work, but I do want to explore it. |
The idea was that authors would include their own hooks, but UX would certainly be improved. However, not sure the same level of "declarativeness" as is in JPM is likely. |
|
@bakpakin I realise it's annoying when people compare Janet to Clojure and so at the risk of doing that in a way that turns you off completely, you could do something similar to what Clojure did when they introduced Unlike Clojure, Janet's data-only file ( Something like: {
:name "Foo"
:description "The best module"
:author "Janet Doe"
:license "MIT"
:url "https://git.example.org/foo"
:repo "git+https://git.example.org/foo"
:deps [“https://git.example.org/bar”]
:src [“foo/a.janet”
“foo/b.janet”]
:build-file "./build.janet" # potentially this is only required if you want to use a different file from the default
:install-file "./install.janet" # same as above
}JPM could then be updated to look for this file if there is no |
|
I like the idea of a If we went that route, we might want to settle on some clear conventions (maybe enforced by the Janet binary to one extent or other?) about how to include "third party" project metadata consistently. |
Not at all, that is a good comparison and a valid point. The data only file would allow more universal processing of bundles, and is relatively low effort to implement. I don't expect we would do much processing of this data in the bundle/ module besides move it around uncorrupted - not even sure we want to have the capacity to fetch remote dependencies over a network. Currently there really isn't even the concept of dependencies - the user would be responsible for providing them. There are a lot of ways to package and distribute software versions and dependencies that are not quite orthogonal, (npm/cargo/go style language package managers, system package managers, version control (git submodules, monorepos), containers, VMs, tarballs, One thing though that might be feasible and useful though is a way to check all installed bundles and make sure that they are working with all other installed bundles, basically a "check-installation" hook where authors of packages could enforce any constraints they wanted on the environment, and raise a (nice) error if there is a problem. For example, if a software author chooses to use a dependency that releases using semver, incremental version numbers, date ranges, etc., then they could do something like: # check if sporky installed with compatible version
(check-semver-for-installed-bundle "sporky" ">1.2.3,<1.2.8")
# check if sparky is installed with the correct git hash
(require-exact-git-hash "sparky" "abc1234")
# check if spot is installed and built within the specified dates
(require-build-date-range "spot" "2023-05-06" "2024-01-28")
# or use feature checking
(require-import-and-binding "spork/zig" zig-compile-and-link-v2)
# or only warn on mismatched bundle
(warn-exact-git-hash "spark" "abc1234" "Only git hash abc1234 has been tested and verified as dependency for this version")
# And finally run a simple smoke test
(import spot)
(spot/smoke-test)inside a hook. However, trying to automatically "resolve" versioning conflicts and search for the best version of packages for compatibility is something that should be up to users. Or at least, outside the scope of |
Even if you're using vendoring as a way to distribute code together with its dependencies, you still need some way to get the dependencies into the project tree in the first place and if you want to make that something you can do from the REPL, downloading remote dependencies is still an important piece of that, isn't it? |
|
Well in the case of vendoring I would think that the dependencies would be managed using git-subtree, git-subrepo, git submodules or maybe just by extracting an archive and commiting it. |
|
Vendoring increases repository size. Even though rust statically links a binary, it doesn't force repositories to carry dependencies. Haskell allows a program to depend on functional packages installed onto the system or ~/.cabal. I prefer functional packages that you can see in haskell, guix, and nix. I think haskell dependency model is a role model for janet. |
|
Git submodules increase size of archives. |
Yes, but doing so is actually fairly simple. E.g. with git: (defn bundle/install-git
"Install from git"
[repo-url name]
(def tempdir (string (dyn *syspath*) "/" (os/cryptorand 16) ".git"))
(os/mkdir tempdir)
(defer (os/execute ["rm" "-rf" "--" tempdir] :px)
(os/execute ["git" "clone" "--depth=1" repo-url tempdir] :px)
(bundle/install tempdir name))
name)You could do similar with mercurial, curl + tar, svn, cvs, perforce, etc. However, the only lightweight way to embed into janet is to shell out, so I'm hesitant to include it in Janet itself. |
So Haskell + Cabal seems to use something called PVP, which seems like a slightly different flavor of semantic versioning. As for vendoring, it will not be the only way to do things, but I would encourage it and it needs to be something that could be officially well supported.
The idea being that dependency resolution and fetching sources/binaries over a network sits at a layer above this. |
|
I'm not particularly fond of PvP. I prefer semantic versioning for its simplicity. |
|
@bakpakin wrote:
I agree regarding those should be at a separate layer. I tend to think of package installation as a flow of steps:
Fetch should likely be something along the lines of a dispatcher that sees an URI and finds a URI handler that exists for it and calls it to execute a fetch. Build should probably have some nice "shortcuts" available (maybe in spork or something) but is entirely package defined. Extract probably needs a "what's that file end with" dispatcher. I think having Dependency and conflict resolution is a difficult thing and it'd be nice if there was something like a Having Janet able to manage installing packages out of the box sounds like a pretty good idea, but once someone needs more than that, perhaps a library (spork, install-utils, whatever) to provide for managing such at a higher level, handling all the fetching, dependencies, conflicts, and so forth would be the best of both, especially if that library is self contained with no other dependencies other than Janet so it'd be as simple as fetching it manually and doing The one issue I see is, if Janet has a On another note... something like (when (not (bundle/installed? "package"))
(bundle/install package))Having all this easily available from within Janet, REPL or Script or Project/Package, would likely feel more fluid and more easily approachable. |
|
I think the main idea would be to have the base module handling (steps 3 & 4) in janet itself and keep the other parts in spork. |
|
I'm not too keen on all of spork being preinstalled with janet. It now has a fair number of pieces by various authors at various levels of "completeness" / "stability". I wonder if there is a way to get "just the necessary bits"...or perhaps there could be some other arrangement. Edit 2: Perhaps a jpm-like tool could depend on code in spork but not cause spork to be installed as a library (so like what is possible now with So the tool could still end up as part of an installation of janet but it could remain unaffected for its basic functionality by what happens to live in janet's syspath. I hope I'm making sense here (^^; Perhaps I didn't quite understand what was meant by "janet releases" though... Edit 1: It occurs to me that there may be projects that depend on specific versions of spork. It seems possible that some of these might break if a newer version of spork gets installed when an upgrade to janet occurs. Not sure if this is a valid concern, but FWIW. |
|
Hm, these are some valid concerns, I'm not sure what the best solution would be regarding those. |
|
I'm unsure if spork should be an automatically included "batteries" library that essentially achieves a status as a de-facto part of the Janet standard library. It's starting to sound like package management shouldn't be part of spork, if one's goal is to in essence say "use latest version of package management and/or Janet". Which almost sounds like keeping a jpm of sorts somewhere though it might be different than the jpm of today, being that it relies on things built in to the Janet standard library. @sogaiu wrote:
This would be nice but then package management wouldn't be very REPL-ish aside from low level concerns of uninstall/remove/reinstall. If it makes sense for package management to be done via REPL and for the higher level package management tool depend on spork then spork might need a split between "stable" and "evolving", that or the REPL higher level package management library should intern spork basically (privatize all the symbols or namespace them behind a prefix like Having wrote all this it almost seems like we need many small libraries instead of a few conglomerates but I don't really like that idea either (that's Javascript-esque... there's a library for that for everything). So now we have a better and more difficult idea that might satisfy all the things in the next sections. Versioned imports. If we have a directory structure something like: Then we could let spork be installed by default, the higher level package management tool could pin to a version, a user could request the latest, and unversioned imports from within Janet default to the latest. This means one could do something like With that idea, spork could be included as well as a higher level package management library for the REPL, without clashing with what users need. Whether or not it's a good idea is another question, and even if it's a good idea, it's questionable if it's worth the extra complexity outside of the fact that this saga we're currently participating in will not be the only case where such a need is likely to come up wherein one needs 2 versions of the same library to service tools and service user code. |
|
I don't really plan on spork being released with Janet. If that was the goal, it would just be merged into Janet. However, spork contains a lot of the core functionality of JPM, just better. On windows, it might make sense to have a spork setup option in the installer but ultimately it should be pretty easy to setup manually. Currently we have a Makefile target to try and setup JPM, so something like that might make it in. However, the key idea is that you could have different versions of spork per project because it's just another pacakage in the tree |
|
Is this going to create breaking changes? |
No, this should be purely additive. JPM will still work for the foreseeable future, and won't require any changes to existing projects. Existing packages that use JPM will still work as well, and I see no reason why a project could not support both besides a bit of extra work. There are various levels of integration that we could go for here (should jpm installed packages be visible to As per usual though, to avoid breaking changes, stick with release versions of Janet rather than latest master. |
So I'm open to ideas, but at first glance versioned imports seems like something I'm very much not going to implement. Rather, it would be a versioned install and the import would be something like (bundle/install "/downloads/spork" "spork-1.2.3") # install script dynamically chooses what root directory to install to based on bundle name
(import spork-1.2.3 :as spork)
(require (string "spork-" (calculate-needed-version)))This looks awful to me in most cases, but its implementable via the usual mechanisms. |
|
Perhaps, can project.janet specify something like this? This is the haskell way of specifying dependencies. In ~/.cabal, various versions of spork are available. |
IIUC, at least this could help mitigate the problem of installation of one project clobbering the dependencies of another. As an example, with how things work with jpm currently, suppose:
Assume further that X < Y [1] and Y has changes that are not compatible with some functionality in X. If someone has project A installed and then installs project B, project A may break because version X of spork has been replaced with version Y of spork. With the proposed "versioned install" idea, if various projects make use of the mechanism, it seems like the aforementioned problem could be avoided (or at least lessened) -- at least by projects that make use of the mechanism. [1] Strictly speaking, I don't think one needs any kind of version "number" ordering to express the problem mentioned above. IIUC, all that is necessary is that there are two different versions with some incompatibilities (e.g. imagine the case of two different commits from a particular repository where there are no version "numbers" associated with either commit). As a side note, vendoring can also protect against the issue of disappearing dependencies (e.g. this recent PR). It looks like other communities have "immutable" stores that can address this. May be that's not so important at this stage. |
|
@bakpakin wrote:
It does look awful, but versioned install solves the issue I had in mind. Someone could potentially "monkey-patch" You are pretty spot on that built-in low-level @sogaiu wrote:
If various projects make use of it soon it'll be the canonical thing, and somewhere in there someone's going to make it look much nicer probably. @sogaiu wrote:
The immutable store thing sounds awesome, until going about the implementation thereof. Vendoring is a lot easier at than making an immutable store that supports $favorite-vcs, $archive, $something-phenomenally-ridiculous-but -somehow-necessary-to-support. |
I messed up a bit with terminology above -- probably I should have written something like (mostly) "append-only". A specific example of this might be maven central. Git repositories on GitHub with the default branch being protected are somewhat close but IIUC these can just "go away" if a user removes the repository (or like in the recent xz incident, MS decides to just make something unavailable). May be there was no confusion, but FWIW.
Yeah, probably it is not a simple matter, but I suppose it depends on the specifics a bit regarding how complicated things can get. Over time, depending on popularity, "append only" things seem like they could get costly though because the stored content just keeps growing...and growing (^^; Edit: Sorry if this is getting a bit off-topic. May be it wouldn't be bad to make a list somewhere of related "concerns" (e.g. installed dependency clobbering, dependency disappearance, etc.) though. Possibly it would be helpful for present and future discussions? |
|
Been doing some thinking about dependencies and bundle/install and bundle/uninstall, and I do think we should allow users to prevent breaking dependencies. Essentially, this is something that bundle/uninstall and bundle/install would check about what's currently installed before doing anything.
This does some what require that we distinguish between "incidentally" installed bundles and "directly" installed bundles. (defn bundle/install
...
(def deps (read-deps-data-jdn))
(def missing (seq [d :in deps :unless (bundle/installed? d)] d))
(when (next missing) (errorf "missing required dependencies: %s" (string/join missing ", ")))
...)
(defn bundle/uninstall
...
(def deps (read-installed-deps-data-jdn))
(def dependents (seq [d :in deps :if (bundle/installed-directly? d)] d))
(when (next dependents) (errorf "cannot uninstall due to dependent bundles: %s" (string/join dependents ", ")))
# uninstall my files, then uninstall dependent files
(each d deps
(if (= 1 (bundle-dep-ref-count d))
(bundle/uninstall d)))
This works without needing to know anything about how to fetch dependencies or versions - the constraint is a function of what's in the syspath so we could definitely put it into the EDIT: The concept of directly installed bundles vs. incidentally installed bundles is basically "autoremove" in apt-get, or "orphaned packages" in pacman. |
|
I've been trying to figure out how this:
would be done. Is this a matter of checking existing manifest files? |
I was thinking so, just check that existing bundles are installed first. It could also include a manual "deps" hook where you could also search for system dependencies or even do feature testing a la autoconf. |
|
Thanks for the clarification.
That sounds quite interesting. |
|
As a user of Janet and someone who thinks a lot about package managers, I've got some opinions/feedback about this. Since some of what I have to say is critical, I would like to preface this with a statement of my gratitude for all the thought and work that's gone into Janet including that of its community members helping answer questions on the forums as well as publishing excellent-quality information about various specific Janet topics. Having used almost every parens language, this one is absolutely the best and easiest to use/understand except for one enormous papercut: the relation between and usage of janet, jpm and spork. As it stands, it's way more difficult than it should be to get all three of these things up and running. The standard operation of jpm also has some serious issues. My apologies for a bit of an aside here about spork since this thread is supposed to be about jpm. These three pieces of software are treated as separate but, due to their entanglement and some of the functionality in spork, I would argue are all actually de facto a part of the "standard Janet distribution", if you will.
I primarily use Arch and Alpine. Currently, Alpine packages the most recent release of Janet but without jpm. In order to get spork in an Alpine container you have to install git and an entire C toolchain, build jpm and install it to your root tree not-under-package-management, and then use it to install spork. Arch does not package janet in its official repos. In the Arch AUR, there are several incompatible packages that either do or do not include jpm and thus some attempts to individually package spork will fail to build because of cross-confusion between their respective maintainers about the relationship between janet, jpm, and spork. Just look how many times spork is mentioned in this thread. Spork documentation is included in the standard library documentation. Some of the functionality (such as In addition to what's mentioned above about hidden state via env variables creating problems, my primary issue with jpm is that the default workflow is to use superuser privileges to install packages to my root file tree. For me this absolutely non-negotiably does not fly (hence running it in a container) nor should it for anyone. Even in the absence of malicious intent, the potential for a critical mistake somewhere is enormous. Installing and uninstalling jpm packages will also not necessarily clean up all the files, allowing pollution to accumulate in the root tree. The other two installation options are not much better. They are to either specify some environment variables (which the original go package manager proved to be extremely brittle) or to use At the top level, the interface to the whole thing could be along the lines of Under the hood it would, upon first run/build use perhaps
As for building and installing, having a just-data I realize that some or all of these ideas may not be that great. Having deps build/install lazily and recursively on first run/build might not be ideal. Maybe it's too different of a workflow for existing projects to want to adopt. The functionality of jpm is currently covering two separate concerns (building janet projects and getting other people's code) which may be better left separate. However, Janet is still young enough to do something better without an enormous amount of friction. Many other languages have failed miserably at this admittedly extremely difficult problem and are locked in with their pile of crap. I implore you to please not reproduce the faults of luarocks, pip, gem, npm, or go's first package managers. For lack of better ideas, just copy cargo. Cargo only needs one metadata file per crate and doesn't clobber differently-versioned deps. It needs no env variables, virtual environments, per package dependency folders, or superuser privileges and has such sane defaults that it rarely requires configuration. It uses lock files and requires packages to be versioned so that builds are reproducible. If not any of that, a robust and modular build system that's part of the stdlib like zig that lets distro packagers or CI easily package/build releases and so that vendoring is viable (which seems where this thread is tending) would be most welcome. Once again, thank you all for all the hard work, excellent programming language, and effort around its community. |
|
@stutonk, agree with most of what is said here. It's not that well documented yet, but the Show and tell: The bundle-tools branch adds the following functions to Janet core: For managing your local bundles (packages):
For inspecting current bundles:
For building a manifest:
There is also example usage of the bundle/* functions in Lately I've been working more on the compile-opt branch, but I think I will likely merge the bundle-tools branch to master soon, as I don't know if there is much more to take away that wouldn't really hinder usability. |
That would be nice, but I think that is the step after integrating JPM, as spork has C dependencies and I want it to be able to be built and developed independently of Janet. The bundle-hooks branch of spork can build itself without JPM. Currently, we don't distribute anything with Janet that isn't baked into the binary. Originally jpm and spork were all separate so I didn't need to make so many frequent releases, but that is less of an issue now. |
|
I thought there might be some benefit in summarising where things are. It looks like the following:
Additionally:
This issue, as originally opened by @tionis, read to me as being about whether JPM should be integrated into Janet. The resolution of the above is that JPM won't be integrated into Janet. Rather, a collection of functions will be built-in to core. These functions could be used by users to install dependencies on their own (thereby avoiding any use of JPM) or to create their own package manager (thereby obviating JPM entirely). |
|
Bump. What is the current status of this? I'm looking into organizing my workflow, and maybe automating some of the builds and project mainenance, and therefore it would be good to know if I should base it on the jpm, or if there's going to be some redesigned tooling soon. |
|
There has definitely been work starting in June and through the most recent commit regarding the This repository's Lines 363 to 367 in 3d93028 A bit that shows building and installing spork using Possibly of use are some of the tests to get a sense of what pieces are being exercised [1]. In short, I think it depends on specifically what one wants to do whether it'd be good to start using [1] This directory has spork's use of |
|
Thanks for the references! That's actually really cool, I browsed the code, and it really looks like something I'd use. I think it would also be cool to have a declarative DSL like the one jpm provides, but maybe as an option that works together with the script. All in all, in a lisp, declarative vs imperative is a bit blurred distinction, and the declarative form works more like a common-case template/snippet for the scripted API rather than the full API of itself. As I understand, that's mostly what jpm does anyway, but it seems to be using some different One thing I'd note though, is the hard coupling of Windows<->MSVS. It is very often that GCC is used on Windows, and while it works with |
Regarding ### Improved version of the C Compiler abstraction from JPM that should be more correct, composable, and
### have less configuration.So may be it's ok to view spork's Re: windows remarks -- I don't know if it makes any difference, but there is some |
|
Is |
|
@noncom That is a pretty accurate summary of the current state of things - cc.janet in spork is basically a wrapper around various C compilers to make it easier to build both Janet native module as well as arbitrary C and C++ based programs. I am using this for some of my own projects where I want more control over the build/compile processes without needing to write a lot of shell script or rely on Make/ninja. As far as the DSL for making things easier, that would also be nice and is just something I haven't gotten around to. My eventual goal is probably to pretty closely copy the jpm build interface into spork.
Because it was the first version and is frankly not as good. It makes some assumptions as to what compilers you are using and how your code is structured.
True, gcc on windows though is usually used with mingw. Clang is also commonly used. This is just something we should fix and have an interface for auto-detecting. That initial build.janet needs some better auto-detection logic for the current toolchain, as well as an override. |
No, jpm is mostly unchanged and will remain unchanged for a while. |
Many janet installs out there are broken or only partially correct, simplifying the setup might help improve this.
To quote @bakpakin :
Any approach bringing a tighter integration with the runtime and package management should reduce possible misconfigurations and make it simpler for people to get started with janet.
There are still a few open questions like:
The text was updated successfully, but these errors were encountered: