Discussion: Crystal Install - Install Crystal Command Line Tools

[jwaldrip]

I wanted to start this discussion before I continued down the wrong path....

tl;dr
Crystal needs an easy way to install command line packages from local and remote sources.

Why?
Command line tools are at the heart of the developer ecosystem and are one of the reasons why many languages become popular. Let's have Crystal take a lesson from it's ruby cousin and allow for a native way to install global command line packages in the path with crystal install.

How?
Crystal would use a similar method as shards for fetching remote dependencies and would use the shard.yml to actually build and install the binary. The follow commands should work:

crystal install git:https://github.com/ysbaddaden/minitest.cr.git
crystal install github:ysbaddaden/minitest.cr
crystal install /some/local/path

The follow commands would:

1. Pull the repo into a temporary directory (download if remote).
2. Run shards install within the directory, therefore fetching all the dependencies.
3. Run crystal build --release -o $PATH and install each of the shards binaries into the local directory.

Changes to crystal shards:
In order for this to work, I would suggest adding a bin section to the shard file that would correlate with the bin name and the crystal file that contains the code for said binary.

# ...
bin:
  kemal: ./commands/kemal-cli.cr
# ...

What's next?
I would like this to drive a discussion with the Crystal core team and the crystal community before I move forward.

[ysbaddaden]

There is a discussion about using Shards as a build system, and another one about installing binaries.

I don't know of any CLI tool in Crystal at the time of writing. Since Crystal is compiled, it's more complicated to have CLI tools loading code at runtime. Also, there is no kemal-cli nor any binary in minitest.

I.e. we need actual use cases.

[ysbaddaden]

Allright, I found one usecase: ICR. It could be nice to be able to install/upgrade it with a single command. Thought I believe it would be better baked into Shards.

[RX14]

I have quite a strong opinion on the proliferation of language-specific package managers that have sprung up in the past few years. Having to have a distro-specific package manager plus npm, gem+rvm (which never works right), cargo, pip etc just gets tiring. You end up having to put a lot of work into installing packages using these systems because they rarely work first try (pip3 or pip2?, node gyp error what the f*** does that mean?), especially when they depend on system libraries. Especially because I usually want to install commandline tools locally, instead of as root (it is my opinion that only the system package manager should be the only package manager to install things outside my user directory). It appears to me that installing command line tools is a difficult problem, because every OS has a different structure, and even between linux distros there is various interpretations of the FHS. So I propose we don't do it.

I think we should leave package management of command line tools written in crystal to the system package manager. Then, as a user, I don't have to care about which language my tools are written in, and I don't have to know what package manager they were installed with, or attempt to keep my system up to date using 12 different package managers. I just have to use one. Unfortunately it's a bit late for that vision to be true, but I don't think crystal should contribute to this mess.

People who write tools in crystal should provide install instructions for their tool, or attempt to get the tool into various package repositories. Ideally, crystal tools would be installed by downloading the source from github and using make and make install, which could even handle downloading a crystal omnibus automatically. The point is that it's up to the tool developer, which means the tool developers are more free to automate the install process end to end. For example, providing an interactive installation script. When such tools make it into binary package repositories, you don't need crystal or shards on your system at all, because it's precompiled. You also get automatic tool updates with the rest of your system, with one package management interface!

Such an approach does make it a little harder for the tool developer, but I think that with appropriate tooling (standardized makefiles, or a little binary that's included in VCS which bootstraps the crystal toolchain e.g. gradlew but for use in makefiles) this can be reduced.

Unfortunately, this approach isn't so practical when you're developing tools for other people in the crystal community. They already have crystal and shards installed, and just want a simple command to run to install the tool they need to develop their crystal project (think rails-style tools, or icr). But the difference with such tools is that they are tied to the projects you're working on. If you look at the success of rvm gemsets, there are only a few tools that you want installed across every gemset, and even then you often need different versions for different ruby versions. Back to crystal, I suggest we standardize a directory for placing "tools". This bin directory would be inside the project directory, like libs/ is now. Then, developers simply use shards tool install x in their project to install a tool, and tools needing to be preinstalled can be listed in the shards.yml file. Developers would then call crystal tool foo <args>, or maybe just make the tools directory bin/, then it just becomes bin/icr instead of crystal tool icr. This approach to installing tools per-project preserves shards simple, small and opinionated nature, and avoids cluttering system directories and the path with tools. If users really want the tool on their path, tools such as crenv can add it for them, as crenv is already "in" the user's shell.

In summary, I propose:

• For people who create generic tools/daemons not targeted for crystal developers:
  • shards be updated to be able to build binaries and place them in a standardized directory inside the project dir (shards#110).
  • A small crystalw wrapper script be created which downloads a crystal omnibus for the version specified in .crystal-version, and unpacks it, then calls exec ~/.crystal/<version>/bin/crystal <args>. It should be able to detect if the correct crystal version is in the path already, and execute that instead. This script would be checked into the source of projects which wish to be installed on a target system, and used in the installation instructions (or makefile) to build the project binaries without requiring the user to manually install crystal.
• For crystal developers who want to use a tool with their project:
  • shards be updated with a tool command which can install and uninstall development tools into a bin directory at the project root. Tools could also be specified in the shards.yml file, and installed using shards install.
  • Possibly add crystal tool foo == bin/foo? Not 100% sure on this one.

Opinions welcome.

[david50407]

I think this can be another program out of Crystal.
Shards only maintains Crystal libraries now (right?), unlike npm or gem, shards doesn't run a post-/pre-install script, in the other words, shards doesn't really install a library or program in your system, it just downloads the source code and add the path into Crystal compiler searching-path.
I think if a Crystal program needs to be spread to another user's computer, it may not need crystal, it just need a binary release with/without system-specific package manager (like Homebrew, apt, pacman, choco, etc).

[RX14]

@david50407 shards can run post install scripts.

I think that library management, and tool management is in scope for shards. Anything that's outside the scope of a single project such as installing binaries outside the project isn't though.

[ysbaddaden]

I share the (extended) opinion of @RX14. Installing an application globally should mean installing a system package (or decompressing a tarball).

Maybe Shards could help build packages for the most common ones, but FPM and Omnibus already do just that.

[asterite]

Another usecase micrate, so I can run migrations in my project. But I don't know how would that work, does it get installed relative to the current project, or globally? And how is it added to the PATH?

[RX14]

@asterite My vision is that with micrate, you could use crystal tool install juanedi/micrate (or more likely add it under tools: in the shard.yml) to install the micrate binary to bin/micrate. It's isn't added to the PATH, because bin/micrate isn't that much longer than micrate. If you wanted it to be on the PATH you could use something like crenv to add the bin folder to the PATH automatically in crystal projects.

[bcardiff]

I agree with @RX14.

The use cases I imagine is template expansion. Create a class / specs together. I would expect that the shard's post install might symlink a wrapper to a crystal script that will do the work. If used as a script it might be simpler to for example customize which .ecr template to use. For small tools developers use I feel comfortable with this approach.

If the tool is a framework with some utilities, it will need some part of the shard on runtime. So the binary delivery won't make it here.

Maybe some guidance can appear for doing such post install symlinks and we can work from there.

[jwaldrip]

@RX14 I agree with the tools being project specific, the use case is definitely supporting rails-esk clis and for things like micrate. I like the tools section in the shard file... although I think if packages supply their own tools then they should be referenced as the name in the shardfile. The following example assumes that an application has a shard that has a cli tool. This example also assumes each shard can only have 1 cli package.

Application

name: my-app
version: 0.1.0

dependencies:
  micrate:
    github: juanedi/micrate
    branch: master

development_dependencies:
  icr:
    github: greyblake/crystal-icr
    branch: master

Child Shard

name: micrate
version: 0.1.0
tool: micrate.cr 
license: MIT

Command
crystal tool micrate [options] [arguments]
_or_
shards tool micrate [options] [arguments]

The second argument would link to the name referenced in applications shard.yml.

[jwaldrip]

@RX14 @asterite @ysbaddaden Anything else to add here?

[ysbaddaden]

Yes, I don't think a crystal install will pop up anytime soon. Shards should be improved to link executables (crystal-lang/shards#95) and maybe become a build tool (crystal-lang/shards#110).

I don't think the whole "clone a repository / download a tarball; pull dependencies; build executables; install into $PATH" is really need. As @RX14 stated, OS packages are a better solution, especially since the compiler isn't required to run the executables.

That being said, Cargo has this feature, so maybe we'll eventually want it. Maybe when Crystal has lots of global and very useful developer tools that don't have to be dependencies (like ICR, unlike micrate). For now, I believe this is still too premature.

One may build such a tool, though.

[jwaldrip]

1. @ysbaddaden: Good discussions going on in shards in relation to this matter.
2. I COMPLETELY disagree on OS packages. By leveraging other methods to create command line tools you are asking the developer to know something outside the realm of crystal.
3. IMHO, A great tool breeds a great eco system. Crystal is AMAZING, but something is preventing the ecosystem from growing. I think CLI tools for crystal developers is just ONE of those things.
4. I think this functionality should be part of the crystal or shards command line tools.

@RX14 stated his dislike of language specific package managers, but are they not successful? They ask the developer to only know about their developer eco-system. What is the point of cross compiling if you still have to do the leg work for apt, yum, brew etc. This seems to defeat the point of having a language that can do that for you.

[jwaldrip]

Closing in favor of crystal-lang/shards#95 and crystal-lang/shards#110. @ysbaddaden looking for any development assistance there?

[RX14]

@jwaldrip Just because an idea is successful doesn't mean it's the right thing to do. Developer tools should have a clean crystal-specific package manager because they can stay self-contained to a project.

More general system tools and daemons often have very different, more complex requirements than developer tools in terms of dependency management. They may want to be installed system-wide, they may want to run scripts on the host system, they may have system libraries they depend on, they may want to install init scripts. Such requirements are well out of scope for shards, and end up creating more work for the user and developer, as they both go for the "obvious solution" of using shards and later along find it limiting. By limiting the scope of shards now we avoid creating problems later.

We should instead focus on the problem of installing system tools and daemons as separate and orthogonal to shards' mission. I covered some of my ideas for addressing this problem above. The end goal would be to make installing crystal daemons and tools that aren't in your package manager as easy as installing C tools: acquire source, make, sudo make install. Is that really harder than downloading and unpacking the crystal tarball, using an opaque install command, then manually changing your PATH?

Using simple tools like make, and not providing an official package manager also aids getting into official package repositories. Getting packages into distro's official package repositories is a huge boon because the packages just work. They have init scripts, they install binaries in the right place, the default config is edited if required for the distro, and you have a more mature package manager in general, with people handling security updates and distro-specific issues.

In effect: installing developer tools needs to be easy, installing system tools and daemons needs to be correct.

[jwaldrip]

Installing developer tools needs to be easy, installing system tools and daemons needs to be correct.

100% Agree. My focus was ONLY on developer tools.

[RX14]

@jwaldrip maybe I got a little confused by this comment. Were you not refuting my point?

[jwaldrip]

@RX14 I was refuting in the context of dev packages.

[ysbaddaden]

For the anecdote, Shards was initially called Harbor, and it was a compiled Bower client. A single binary file, without any requirements. Why? Because I was fed up of having to install Node to install NPM to install Bower to install JS/CSS dependencies in pure Ruby projects that didn't use Node, at all.

[didactic-drunk]

What about sysadmin or productivity tools that are non project specific where

• The distribution is LTS or unupgradable. Years old and not getting updated packages
• Or the user doesn't have root

Examples (via cargo install):

• ripgrep
• bat
• exa, lsd
• hyperfine

While few tools exist in crystal there are some and more coming.