Package manager

[asterite]

This is the main issue tracking this feature from the Roadmap.

Crystal will provide a built-in package manager. We really want this to be the only package manager so it's easier to build a collaborative community.

We want a truly decentralized package manager. These ideas could make it work:

• No global directory where all deps are installed: it's local to each project (lib/libs directory)
• Each repo has a special branch (maybe _releases) with metadata for dependencies for each version. These are cummulative, so version 0.2 contains metadata for 0.1 and 0.2.
• Crystal provides a command to release a new version, pushing to that special branch, and creating a tag
• Use Semantic Versioning
• Automatically download recursive dependencies
• Resolve conflicts
• Remove Projecfile, use YAML, both for project.yml and for the metadata file
• Include name and version in project.yml
• The name above is used for the directory in which it is installed, and used by require "...", so for example "webmock.cr" will be installed in "lib/webmock", and that directory will contain the direct checkout of the project (so it has the src directory in it)
• The require logic changes to that if you do require "foo", we check if there's foo.cr in CRYSTAL_PATH, or foo/foo.cr, or foo/src/foo.cr (this last one is the one that will be used for dependencies installed via the package manager). With this logic, the current CRYSTAL_PATH value doesn't need to change
• Probably rename "libs" to "lib"

With the above, when you do crystal deps install, all first-level dependencies are gathered. From there we go to each depednency's repository and check out the special _releases branch to get all metadata for all versions of that first-level dependency. We apply this recursively until we get all the metadata for all involved libraries. The previous process should be fast, because only that metadata branch must be checked out, and only once for each library (and we can parallelize the requests). Then we can solve conflicts and install what's needed.

For discoverability, we can list github/bitbucket repositories that have crystal code and that also have that special _releases branch, which in turn contains all the information for every version of the library.

For all of this, the easiest thing would be to build on top of @ysbaddaden's shards, which already has the desired YAML format, probably has some logic for semver, etc.

[ysbaddaden]

As discussed with @asterite Shards shall supersede the current crystal deps command.

Development on the package manager will continue there for the time being. We'll eventually figure out if it shall be merged into the Crystal source code or be kept separated (as long as it's tightly integrated/embedded with Crystal's distributions).

The main feature still missing is the lock file (crystal-lang/shards#12) then to integrate it as crystal deps.

[PragTob]

Looking forwards to this very much 👍 👍 🚀 🚀 💃

Comments/questions:

• will semantic versioning be enforced/is that on the roadmap? elm claims that they do it, which I finrd to be fascinating
• it seems common to me that the lib directory contains code you wrote yourself but that is applicable to other projects (that's how I use it in my ruby projects and I know other do too), hence personally I'd prefer another name for the directory. Unless in crystal we'll have another convention for where to put that code (somewhere under src) ?
• Steve Klabnik once told me that he considers toml to be a much better format for that sort of task (compared to yml), which is why it is used for cargo (Rust) - I don't have any details, though and I like yml, just throwing it in here
• Do I understand it right that this will all be based on checking out git repos or is that a first step and in the distant future there might be something else (like gems/packages pushed to some infrastructure) - I understand that own infrastructure would be sort of central but so is relying on github/bitbucket (sort of) in my understanding

also thanks @ysbaddaden for the work so far! 👍

[ysbaddaden]

• semver: not enforced, but supported and expected (mostly by developers);
• I like lib for dependencies better than libs and it's 3 letters like src so it's nice, but no strong opinion here (it could be vendor);
• YAML is distributed in the stdlib and did the job. I have no strong opinion against TOML and it may prove better. We shall just try it. Maybe it's better than YAML which needs quotes around some version requirements (eg: ~> if I remember correctly).
• There won't be any registry that would distribute packages. Source code shall be downloaded from repositories (either GIT or whatever we want to support). That being said, but there may be a nice registry site to index and search libraries (ala http://crystalshards.herokuapp.com/) to index and search libraries, available versions, etc.

[0x1eef]

probably you won't need many of the features that YAML provides, and I guess you have to think about security when loading YAML because it can deserialize complex types, where as toml is more simple and just supports arrays, hash, string, & number types.

[jhass]

Is there a defined serialization of TOML, given abstract input data? Last time I checked I couldn't find it, but I think it's important to have in order to support tooling that automatically edits the file to bump the version number, add/remove dependencies etc.

Stdlib YAML doesn't have serialization either yet, but it should be comparatively easy to add.

[asterite]

In TOML you have to put quotes around every string, so having to put quotes in YAML from time to time is not an issue.

There's also the thing that TOML isn't stable. We have a parser for it, it's just not in the standard library because it's not standard enough.

We can also compare a shard.yml with a Cargo.toml file. To me, YAML looks much cleaner without all those quotes. Also, those [[...]] look strange. Well, once you learn TOML you understand it, but it's something else that needs to be learned (I'm sure many more people know YAML).

There's also the thing that @jhass mentions, there's many ways to represent a TOML and I can't figure a way to do a toml_mapping because you have to parse things to an intermediate structure before converting that to a model. In JSON that works pretty well, and in YAML it might work well too.

[0x1eef]

@jhass what do you mean by "Stdlib YAML doesn't have serialization either yet" ? When I say serialization I mean turning a Crystal string into yaml, and then back again(deserialize). YAML can just do much more than only a string, and sometimes it's a security issue (like being able to create Ruby symbols from YAML).

[jhass]

I mean that the serialization API is not available yet, but the standard does define a canonical serialization and the backend libyaml does have an API for it.

With TOML, if there's a canonical serialization, we also have to write the complete serializing code. But I couldn't even find a definition for such.

[refi64]

Ugh...please not TOML. It's the worst markup language. Ever.

The issue with YAML is that, if you want auto-tooling, you're going to store a heck of a lot in the tree structure (the indentation the user used, the whitespace surrounding the colon, etc.), as well as use some sort of ordered hash to avoid literally wrecking the order of the file, if you want the tooling to not change the whole file around every time one "bumps the version number".

What about a ridiculously simple INI-ish thing, like:

[meta]
name = shards
version = 0.1.0

[sources]
https://shards.crystal-lang.org/
https://shards.example.com/

[dependencies]
pg >= 1.2.3
memcached = *
openssl = github+datanoise/openssl.cr#master

[dependencies-dev] ## Like Pip's requirements-dev.txt; you don't minitest installed to actually run shards
minitest = git+https://github.com/ysbaddaden/minitest.cr.git ~> 1.0.0

[trans]

I would like to offer a modified approach that would provide a few additional benefits. My main concern is with the _releases branch. I do not think it is a good solution to the problem b/c the mechanism of a branch doesn't really correspond to the need. Another issue is the usability of the project.yml file. Either the user will have to follow a very strict schema or the parser will have to have plenty of heuristics to make the format flexible. The former isn't very user friendly, and the later would generally require use of the same parser library whenever the file needs to be read to ensure proper info. But other tools might want to use that info too, perhaps some not even written in Crystal.

To overcome these issues, I'd like to propose an idea I came up with some time ago that I use for my Ruby projects. I have a humanized form of metadata, in an easily edited YAML file, though it can be a Ruby script or other format, like TOML, or combination of files actually, it doesn't really matter. Than there is a canonical metadata file. A tool takes the editable file(s) and generates the canonical file. The canonical format follows a strict schema with no exceptions. This makes it readable by other programs without any heuristic processing. To solve the _releases issue, this tool can also look back at past tagged versions and pull the relevant dependency info into the canonical file. This approach gives maximum flexibility to the developer while still providing a single strict metadata file for use by the package manager and other tools.

[ysbaddaden]

The _releases branch hasn't been implemented. This is open for discussion, and as I understand it, it would mostly allow to speed things up, yet hide the gory details behind a few tools —that would work as you described.

For now the different shard.yml come from version tags.

The format of the YAML file will indeed be specified and I'm thinking about a validate command (Travis does that for instance).

[vyp]

@trans:

My main concern is with the _releases branch. I do not think it is a good solution to the problem b/c the mechanism of a branch doesn't really correspond to the need.

I don't understand what you mean, could you elaborate?

@asterite I have a question:

With the above, when you do crystal deps install, all first-level dependencies are gathered.

Where are these dependencies 'installed', in ./.crystal?

[ysbaddaden]

The src folder of a foo dependency is installed as libs/foo.

The .crystal is where we cache cloned repositories.

[kumpelblase2]

If I may throw in my 5ct:

• I'd certainly be in favor of something along the lines of yaml or json for the project file. I find it way easier to read than anything ini-ish/toml (because it most of the times looks weird with arrays or sub-properties).
• I also do like the decentralized approach, but having a single registry to look up what's available is still something I'd want to have. Because not everything will be stored on github and we end up having half the software (I'm exaggerating here) on self hosted repositories where you won't easily find them. Probably a hard problem to takle for a registry as well. But I see it on the Shards roadmap, so I'll just hope something like that will be available.
• Liking the Semver versioning approach, would be even more happy if it got enforced, but I don't have a good argument to do that.
• On the name of the source directory for those dependencies I have to agree with the concern of @PragTob , as I think several people would use a lib directory for something else. So I would prefer something like a deps (or dep for that matter) directory or maybe even something completely different.

Otherwise it looks solid and I could certainly work with it, so I'm happily waiting for it. 👍

[jhass]

If we go for something else than lib I'd vote for vendor, but I think it's fine if we establish lib for third party and src for own stuff as a community convention.

[vyp]

Thank you @ysbaddaden.

I vote for enforcing semver but only because it just ensures to keep all the stuff that handles version numbers simple (or simpler).

About format, I vote yaml because it's 'prettier' to read, and I guess crystal users in general also like nice looking (i.e. readable) syntax, because that's one of the selling points of this language. But I'm really not too opposed to the other options either.

About third party directory name, I don't really see an issue with the community just establishing lib as the name. But if another one has to be chosen, I'd go with @ysbaddaden's vendor suggestion > deps/dep.

@asterite About discoverability... In addition to automatically listing github/bitbucket repos with crystal code (I assume you mean automatically), could we also have it that random people can submit their project to this list (as a 'hardcoded' project)?

Obviously, this would only be for those who do not host on github/bitbucket. And the submission should have their git repo url somewhere, so that the presense of a _releases branch can be detected. There shouldn't be any conflicts even if two projects have the same name because the url should always be different. (As in, conflicts in the list itself. I think the problem of actually using conflicting projects simultaneously still exists, but that's another issue altogether.)

Oh and also, what about gitlab.com? (if they consider crystal as a language)

[jwaldrip]

Question:

The releases branch sounds like a pain to handle. Why not just rely on git tags, and ensure each released version has the correct dependencies specified with each tag?

[asterite]

@jwaldrip The releases branch will be managed by a tool, so it won't be a pain. And you need to have fast access to all dependencies of every version released. It's not enough to have access to the dependencies of a particular tag, you need them all to resolve conflicts.

[ibash]

Hi there, just started to explore crystal recently and wanted to lend a couple of thoughts:

1. While I like the idea of decentralized dependencies, I think a centralized repo is going to emerge whether you want it to or not, at the very least to provide an easy way to discover packages and have reliable installs. The question is really whether you want it to emerge organically or here.
2. I really do not like the idea of the _releases branch. The reason why is that it makes versioning inherently tied to git. I would much prefer not having git be an implicit requirement if it can be avoided. In this instance it could be avoided by by having a _releases folder within the same directory.

[vyp]

@ibash

1. There is going to be a list to help with this as said in the OP.

2. A _releases folder (or better a .releases folder) will not work because it means crystal will need to download the entire project for each dependency just to do dependency resolution. Understandably, this process should not have to do that. Because there's no central registry holding all metadata*, the next best thing is to use a dedicated git branch per project for this information.

   And besides, @ysbaddaden, doesn't shards currently only support git anyway, meaning all crystal projects have to use git if they want to be compatible with shards? I think enforcing using git is an easy and simple solution.

*Of course, there will be a list as said in point 1 (probably something like https://crystalshards.herokuapp.com/), but that's obviously not a reliable source of metadata information because crystal projects don't have to be on this list.

[oprypin]

I am very very strongly against _releases.
Let me just leave this here.

/tmp/blaxpirit $ git ls-remote --tags https://github.com/blaxpirit/crsfml
701047a301c09740e1409b0f557212c27373d3c2        refs/tags/v2.2.0
f9253349b25c1f0e01f5839b95c470f334e85d7c        refs/tags/v2.2.1
2ff4bcb2adcbca95ee1b4e4ddd11f7c4f7c03b56        refs/tags/v2.3.0
9b7534de0af941b73da06f65a3a57c1afe65c4be        refs/tags/v2.3.1

/tmp/blaxpirit $ git clone --branch v2.2.1 --depth 1 https://github.com/blaxpirit/crsfml
Cloning into 'crsfml'...
remote: Counting objects: 52, done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 52 (delta 2), reused 24 (delta 1), pack-reused 0
Unpacking objects: 100% (52/52), done.
Checking connectivity... done.
Note: checking out 'f9253349b25c1f0e01f5839b95c470f334e85d7c'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

/tmp/blaxpirit $ cd crsfml

/tmp/blaxpirit/crsfml (v2.2.1) $

1. To list all tags a clone isn't needed at all.
2. To see one specific version only one commit needs to be cloned.

Not to mention GitHub API which turns (1) into a JSON download and (2) into an archive download (URL provided in JSON).
GitHub also allows you to download a single file like so: https://raw.githubusercontent.com/BlaXpirit/crsfml/v2.2.1/README.md (replace README.md with project.yml or whatever).
I'm sure GitHub is worth specialcasing because of this. With the added benefit that Git doesn't need to be installed!

[vyp]

@BlaXpirit

1. Not only the version numbers, but a dependency needs to be checked if it has any dependencies of its own. So I don't think just listing the tags will work?
2. Even with depth=1, it still downloads all the rest of the files of the project which is unnecessary. Using a _releases branch will be more faster the more dependencies there are, compared to using just cloning a release (even with depth=1).

(@asterite I'm just guessing here though, I may be wrong?)

But why are you very strongly against a _releases branch per se?

[oprypin]

@vyp

People provided multiple reasons to not like the _releases branch. I don't have much to add.
And I haven't seen a single compelling reason why it's necessary.

Do dependencies really need to be checked beforehand? I mean, if you're gonna be downloading the whole library anyway, why bother with having a way to get just the release-info files?

If you're concerned about speed. The vast majority of dependencies will be on GitHub. If it's specialcased like I mentioned then this is not a concern anymore.

The size of the _releases branch can also get big, by the way.

[ysbaddaden]

No nested dependencies ala node_modules. That won't happen. It doesn't fit the way Crystal requires dependencies (globally, not scoped). Dependencies are installed flattened in the libs folder.

The lock file is being implemented. It's a flattened list of all dependencies and only applies to the project. See crystal-lang/shards#27

[ysbaddaden]

Just reading myself: sorry for the tone. I didn't mean to be rude!

[asterite]

@jwaldrip

Semantic versioning: we could enforce that. But does this mean just the "1.x.x" syntax? Elm seems to check changes in the code, although I'm not sure how would you do that.

DSL: The problem with using a DSL is that you'll have to fire the compiler multiple times to compile that. And then it becomes hard to communicate data between these processes. In Ruby it's easier because you can eval, instance_eval and share state in the same single running process. Also, what would the shard method do? It can't do anything by itself, this info needs to be gathered and then used in conflict resolution, so at most that would just register this info in a data structure... which is the same as using JSON, YAML or some other format.

Command line: I also think doing crystal shards or crystal deps will be much more intuitive than just shards (a separate command).

Child dependencies: as @ysbaddaden mentions, in node.js you do some_module = require("...") and then you use it through some_module, so you don't have a global name conflict. But in Crystal it's just require "..." and everything gets clamped together. This sounds good in node, until you find out you have a lot of duplicated, almost the same code, in your dependencies directory for no real reason. I have a node_modules directory for a relatively small app that occupies 200 MB, ugh... Crystal's drawback is that you have a single namespace, but I find that to be more intuitive and simple if used correctly (no need to fill you code with multiple require or import statements at the top of your files).

[PragTob]

semantic versioning: I meant the code checks, no idea how they do it - I just wanted to put it in as an idea as I found that to be amazing about elm. Not sure how it plays out in practice. I'm perfectly fine with putting it on a "future wishlist" or abandoning it as too complex :)

[jwaldrip]

@asterite

_Semantic Versioning:_ yes as the format for semver, nothing more.

_DSL:_ I don't know why the compiler would have to run multiple times. I would see that the only thing that uses the shardfile and its lock would be the shards/deps command itself. Once the dependencies are downloaded they are in the directory tree and can be required using "..." or their name as defined in the CRYSTAL_PATH. When cloning a project, it just customary to run a dependencies command.

_Child Dependencies:_ I see your point, but my counter point is that in bundler it gets annoying when two libraries child dependencies conflict. This could certainly be the exception, but I don't think it lends to as much flexibility... unless there is a way to both.

[waterlink]

On semver standard is described here: http://semver.org/ Format is pretty
strict, but allows for labels after patch version: '1.0.0-alpha' for
example.

Best Regards,
Alexey Fedorov,
Sr Ruby, Clojure, Crystal, Golang Developer,
Microservices Backend Engineer,
+49 15757 486 476

2015-09-06 18:32 GMT+02:00 Jason Waldrip notifications@github.com:

@asterite https://github.com/asterite

semver: yes as the format for semver, nothing more.
DSL: I don't know why the compiler would have to run multiple times. I
would see that the only thing that uses the shardfile and its lock would be
the shards/deps command itself. Once the dependencies are downloaded they
are in the directory tree and can be required using "..." or their name as
defined in the CRYSTAL_PATH. When cloning a project, it just customary to
run a dependencies command.

—
Reply to this email directly or view it on GitHub
#1357 (comment).

[jwaldrip]

@waterlink I agree with that too.

[trans]

@ysbaddaden

We can't have different formats for the metadata file, like one project uses TOML, another one is using YAML or JSON... Let's avoid aliases!

There would only be one definitive file, but the developer can work with whatever source format they'd like. The distinction between them is similar to the difference between Gemfile and Gemfile.lock.

[tomchapin]

👍 for the shard ideas presented by @jwaldrip

[ibash]

On the topic of source formats.
I think it's better to have a single format than many. The reason why is that when looking at another library or project, you immediately know what file to pick out and how to read it.

What the specific format is does not matter much, developers will learn it and most of the formats mentioned are human friendly enough to be useable.

I'm partial to json, but if it's yaml, toml, or something else, it's not going to bother me.

[refi64]

Ok...this is kind of off-topic...but I can't help it...

Guess what the full name of Kirby 64 was?

Kirby 64 and the Crystal Shards

(notice the bolded text)

Sorry, but it seemed kind of funny to me. :)

[ibash]

I vote next tool be named Kirby.

[asterite]

Starting from b86a5b8 crystal deps now delegates to shards \o/

The idea is that eventually shards will be distributed with the compiler, so it's always found.

[ozra]

+1 for "Kirby", haha.

@jwaldrip - as already mentioned the hierarchical dependencies á la npm/iojs does not work with Crystal. This is the reason I strongly believe semver should be enforced. The semantics of the scheme makes it straight forward to deduce which latest possible versions are compatible for all dependants - or if there's a conflict in version needs that needs to be worked out!

"Flat namespace" == great need for a good convention of dependency compatibility checks == semver.

The "strictness" of the semver rules aren't a problem at all - they're super straight forward.

[ysbaddaden]

This is the last time I express myself about semver: it's nice, follow it if you can and want, but I won't enforce it.

1. Depending on the project, having 1.2 or 1.2.3.4 scheme can be perfectly valid.
2. You can't trust minor won't break your code. A bug fix will almost always break something.
3. Hell, you can't even trust a patch! A security fix may lead to a breaking change. Will you bump major for a tiny fix?! I won't.

That being said. Semver is very good. It helped focus to keep a sane version scheme and a stable code that can can somehow rely on. You should try to apply its principles as much as possible.

But you can't follow it blindly, you can't enforce it, and I won't. This is developer responsibility.

[ozra]

Fair enough, the point that developers efforts of versioning correctly might not match reality is a good one. I still don't see how a good unification of dependencies can be achieved without that "promise" of a strife to get it as right as humanly possible though. In any event, this will be my last commenting on SemVer too. Having "official" modules managing in crystal is great!

[refi64]

Despite not being a big SemVer fan, you need to define a standard versioning systems if you want dependencies to be managed correctly. If someone adds package B version >=1.2.1 as a dependency to package A, but package B tags the releases in some weird way, it'll screw everything up.

Maybe not as strict as SemVer, but some kind of control.

[jwaldrip]

So what is the status on the package manager?

[jhass]

The current status is that crystal deps delegates to shards, which you can install from http://github.com/ysbaddaden/shards

Maybe we should close this issue now in favour of more specific ones?

[asterite]

Yes, let's close this and add issues in shards as they appear :-)

[jwaldrip]

Since one of the goals was to make it part of the core, is there any plan to merge shards into crystal?

• Jason

---

Jason Waldrip
Chief Technology Officer
Brandfolder.com
email| jason.waldrip@brandfolder.com
office |720-744-0311(tel:720-744-0311)
mobile |646-460-5959(tel:646-460-5959)

On Oct 14, 2015, 8:17 AM -0600, Ary Borenszweignotifications@github.com, wrote:

Closed#1357(#1357).

—
Reply to this email directly orview it on GitHub(#1357 (comment)).

[jhass]

Yes, once everything stabilizes we'll look into that again.

[glyh]

I've scan through the shards project, finding there's no plan for support command like cargo install, I wonder would this be a thing for crystal?

[caspiano]

Hi @glyh! Here's an issue with some discussion on that topic crystal-lang/shards#81

Come join our forum, it may be more suited for discussion on questions like this :)

[glyh]

@caspiano Thanks. I don't think that's the place though. The author is reluctant to implement this feature.

BTW, this language's competitors have great support for package managing. I'm talking about Nim and Go. V doesn't seem to have a good one though.