Precompiled diesel_cli binaries for major operating systems

[lpil]

Hello! Thank you for this excellent project!

I wish to make use of diesel_cli on machines that are both resource constrained and do not have Rust installed.

Would it be possible to make precompiled binaries availible so they can be downloaded and run without needing to install Rust and compile them ourselves?

Thank you,
Louis

[weiznich]

In theory this it should be possible to build binaries for all major operation systems. As this should be as easy as possible for us maintains I would like to have this as part of our CI workflow. https://github.com/rust-analyzer/rust-analyzer/blob/master/.github/workflows/release.yaml As we are planing to migrate our CI to github actions anyway sometime soon this should probably be integrated into #2269. rust-anaylzer is already doing something like this for their release process, so maybe some parts from there could be reused? At least I personally do not have currently the bandwidth to work on that, so if someone wants to do that feel free to submit a corresponding PR integrating that with our current CI.

Open questions that needs to be answered:

• How to handle dependencies like libpq, libmysqlclient and libsqlite3? If we link them dynamically we need to assume something about their location and name on the target system, right? Statically linking them will require a lot of fiddling with the build system.

[benedictjohannes]

The way Diesel define migration in pure SQL statements makes source control of DB schema extremely easy to deploy. One just need to check out SQL files, have the Diesel CLI binary, execute migrations.

However, having to compile Diesel CLI for every system in which it needs to run is a dealbreaker.

(forgive my ignorance) Being a nodeJS/web developer I'm checking whether I can use Diesel CLI to handle migrations. I usually handle migrations with knexJS, but it requires node and package installation to be able to do its job. Both Knex and Golang's implementation require code in the framework's language (.js or .go).

However, Diesel CLI's approach of migrations makes it (almost) language agnostic: Any DBA and database-connected app developer can write pure SQL (I'm not aware of other small binary that does the job). Diesel (at least its CLI) can then potentially reach beyond Rust developers.

Is it possible to static link all DB client libraries (I mean, bake it in the compiled binary)? I had no problem connecting to PG11.x server using psql v12. Arch Linux has (precompiled) binaries as well (requiring dependencies of client libs, though).

[weiznich]

It is certainly possible to link all DB client libraries statically. For sqlite this can be done via the sqlite-bundled feature. For libpq and libmysqclient some configuration of the corresponding sys crates (libpq-sys, libmysqlclient-sys) via environment variables is required. Checkout their README's and build.rs scripts for details.

As mentioned above: I won't work by myself on this issue, but I will accept a well written PR here.

[lpil]

However, Diesel CLI's approach of migrations makes it (almost) language agnostic: Any DBA and database-connected app developer can write pure SQL (I'm not aware of other small binary that does the job). Diesel (at least its CLI) can then potentially reach beyond Rust developers.

To clarify this is my use-case, I'm working with projects written in Erlang rather than Rust and find Diesel an attractive alternative to Flyway as it requires us to install and run the JVM.

[benedictjohannes]

@lpil Probably the only part giving away Diesel CLI's power as not a migration toolkit is that it requires Cargo.toml to operate. I bet this feature can gain traction and will probably benefit both Rust and Diesel in terms of developer recognition.

I tried do dive into Rust toolchain the best that I could, but with setting target to linux musl. Diesel CLI freshly cloned from master branch compiled successfully, but the binary is still not portable (musl target only has glibc statically linked - build log)
I even played around for a few hours on fresh GCP VM with Docker Rust Musl builder image but I still failed to build Diesel CLI for musl. It must be me, as the container image passed tests building Diesel. I believe providing such binaries is a small challenge for true Rustaceans.

@weiznich I understand that this might not be a priority, but I believe that this would be a great feature.

[weiznich]

Probably the only part giving away Diesel CLI's power as not a migration toolkit is that it requires Cargo.toml to operate.

With #2069 it does not require a Cargo.toml anymore. Using a diesel.toml that optionally configures the migration directory also works now.

I tried do dive into Rust toolchain the best that I could, but with setting target to linux musl. Diesel CLI freshly cloned from master branch compiled successfully, but the binary is still not portable (musl target only has glibc statically linked - build log)
I even played around for a few hours on fresh GCP VM with Docker Rust Musl builder image but I still failed to build Diesel CLI for musl. It must be me, as the container image passed tests building Diesel. I believe providing such binaries is a small challenge for true Rustaceans.

https://github.com/emk/rust-musl-builder might be helpful as they have explicit examples how to build diesel applications using musl and diesel-cli is in the end just a diesel application.

[weiznich]

@GopherJ I do not see what this adds to the discussion, that's already written in my comment above. If you've nothing to say, please stop adding comments here.

[GopherJ]

ok @weiznich I'll delete my comment above, I just want to say it's important while seeing so many .so

However, pls don't use github as dictatorship platform, I know you are a good contributor of diesel, but this shouldn't stop others speaking...

being a good contributor doesn't mean:

You are superior to others

[weiznich]

@GopherJ It's not about stopping someone from speaking, it's just about increasing the signal to noise ratio for potential contributors. I'm happy to hear out comments that add something to the discussion, but I dislike going though my notifications and spend my time with reading basically "me too messages" coming up in my post box.

[GopherJ]

After applying the following changes:

I'm able to use rust-musl-builder to compile a static linked diesel_cli

alias rust-musl-builder='docker run --rm -it -v "$(pwd)":/home/rust/src ekidd/rust-musl-builder'

rust-musl-builder cargo build --release --manifest-path=diesel_cli/Cargo.toml --no-default-features --features postgres

[benedictjohannes]

@GopherJ Your last comment looks helpful. Can you share in more detail, maybe on a separate branch on your fork or maybe using github gist? It's quite hard for me to understand what you've done by looking at a screenshot.

[weiznich]

I would rather not like to have a direct dependency on openssl-sys. That's an internal libpq dependency, so I think that should be handled there. Additionally it seems that libpq + openssl + musl is causing some issues in general. See for example #813 or #700 for example.

[GopherJ]

@benedictjohannes Hi, basically what I did comes from: https://github.com/emk/rust-musl-builder#making-diesel-work and https://github.com/emk/rust-musl-builder#making-openssl-work

without these I ran into: emk/rust-musl-builder#69

fork:https://github.com/GopherJ/diesel (not tested, just added patch for you)

[jman-schief]

I'd like to confirm that the patch GopherJ@26060a4 allows to statically compiling diesel_cli against postgres+openssl, thank you @GopherJ so much for helping here!

I've made a little progress and could also compile in the SQLite3 client using the above mentioned docker container (emk/rust-musl-builder) and just changing the sqlite3 crate used.

My final goal is to have a diesel_cli binary also with the MySQL/MariaDB client statically compiled so I have one binary to cover all three DB supported by the cli. But the MySQL/MariaDB client is a tough nut to crack, I couldn't compile yet compile it with musl.

I have published my progress in this repo and will be happy to share my learnings. Also super happy to receive suggestions and/or patches.

[nanoqsh]

I apologize for this possibly inappropriate question, but why does Diesel need dynamic linking at all? (At least in the case of Postgres)
I used a rust-postgres library and I wrote SQL queries manually and it works fine even though I don't have Postgres installed on my system. I use Diesel to get rid of manual queries. But I don't quite understand why I need to have Postgres on the local machine if my application is still running in the database remotely?

[weiznich]

@nanoqsh I think you are mixing up a few things here. Let me try to clarify those points.
First of all to use diesels postgres backend you don't need a complete postgres installation. You only need a copy of libpq, the official postgresql client implementation. Additionally it is possible to link this library statically, so no dynamic linking is required. (Checkout the readme of pq-sys for details about this.)

rust-postgres on the othet hand is a pure rust implementation of a postgres client library. This means they do not depend on libpq internally as they implement similar functionality in pure rust.
Now why does diesel not just use rust-postgres internally? There are quite few reasons for this. I think the main point is just that rust-postgres wasn't mature enough at the time the diesel development started. Also there are some concerns about performance and compatibility compared to the official client implementation, but those are likely solvable.
If you are interested in changing this: There are two open PR's with exemplary pure rust implementations. For me itself its not a priority to fix them and merge them, but I would likely accept a fixed version of such a PR.

[nanoqsh]

@weiznich Thanks for the clarification

[tifrel]

Since the last comment here is almost a year ago, I wonder if there has been any progress achieved on this?

[weiznich]

@tifrel This issue is not closed, so nothing has changed. Also please stop asking these questions on such issues.

[weiznich]

This is now fixed with the release of diesel 2.2. The github release now contains prebuild binaries for several platforms: https://github.com/diesel-rs/diesel/releases/tag/v2.2.0