add to package repositories

[BurntSushi]

It'd be nice to at least get it into Ubuntu and homebrew. Sadly, I think either one of those will be quite difficult since Rust isn't packaged in either one of them.

The lowest hanging fruit is the Archlinux User Repository (AUR).

Fedora is getting Rust packaged soon, so that may be plausible.

Sadly, we may need to live with binaries distributed here for the time being.

[BurntSushi]

I now have a brew formula and an Archlinux PKGBUILD. I also added ripgrep to the Archlinux user repository.

Sadly, the brew formula isn't actually in Homebrew, and it downloads a binary, which is really bad form. I'd like to fix it, but I'm not quite sure how.

[tekacs]

rust does appear to be packaged in Homebrew (http://braumeister.org/formula/rust) - just a heads-up.

The current approach uses a 'bottle' and so pulls binaries and should work just fine. It /does/ install cargo. It currently builds from source on the latest macOS Sierra, because the formula hasn't been updated to include a bottle for that yet.

[BurntSushi]

@tekacs Oh, cool, somehow I missed that! I'll check that out, because I definitely want to get ripgrep into brew proper if that's possible.

[mwean]

I tried to create a formula for it:

class Ripgrep < Formula
  desc "ripgrep combines the usability of The Silver Searcher with the raw speed of grep"
  homepage "https://github.com/BurntSushi/ripgrep"
  url "https://github.com/BurntSushi/ripgrep/archive/#{version}.tar.gz"
  version "0.1.17"
  sha256 "b558b6650bfa9cf0fd0fa58a8617cafc7c819ee25a26d15ca2ce39979dd18a18"

  depends_on "rust" => :build

  def install
    system "cargo", "build"
  end

  test do
    system bin/"rg", "--help"
  end
end

But couldn't get it to build. The first time I tried, it was stuck on make for over an hour before I killed it. Then I got this error:

Last 15 lines from /Users/Matt/Library/Logs/Homebrew/rust/02.make:
[ 22%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/SearchForAddressOfSpecialSymbol.cpp.o
[ 23%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/Signals.cpp.o
[ 23%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/TargetRegistry.cpp.o
[ 23%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/ThreadLocal.cpp.o
[ 23%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/Threading.cpp.o
[ 23%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/TimeValue.cpp.o
[ 23%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/Valgrind.cpp.o
[ 23%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/Watchdog.cpp.o
[ 23%] Linking CXX static library ../libLLVMSupport.a
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: ../libLLVMSupport.a(ThreadPool.cpp.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: ../libLLVMSupport.a(ThreadPool.cpp.o) has no symbols
[ 23%] Built target LLVMSupport
[ 23%] Built target obj.llvm-tblgen
make[1]: *** [all] Error 2
make: *** [/private/tmp/rust-20160924-69301-1t6mv0r/rustc-1.11.0/x86_64-apple-darwin/llvm/llvm-finished-building] Error 2

READ THIS: https://git.io/brew-troubleshooting
If reporting this issue please do so at (not Homebrew/brew):
  https://github.com/Homebrew/homebrew-core/issues

These open issues may also help:
rust: disable stable on 10.12 https://github.com/Homebrew/homebrew-core/pull/4962

I'm not familiar with Rust at all, but maybe some of this will help you.

[est31]

Rust is packaged for both ubuntu and debian.

[priyadarshan]

Ripgrep is such an excellent tool. It would be nice to have it in FreeBSD ports, too.

https://www.freebsd.org/doc/en_US.ISO8859-1/books/porters-handbook/porting-submitting.html

Do I understand correctly that, if one install rust:

http://www.freshports.org/lang/rust/
http://www.freshports.org/devel/cargo/

then ripgrep is easy to install with cargo install ripgrep?

[mwean]

It looks like there's some problem installing rust on Sierra, so I won't be able to get the formula working. @BurntSushi can you try adding the formula a posted locally and trying to install?

[Stebalien]

It looks like a trusted user just put ripgrep into Arch Linux's [community] repo.

[BurntSushi]

Yup, many many thanks to @svenstaro!

[patrickdepinguin]

Please consider a gentoo ebuild too.

[passcod]

@BurntSushi could you mention that the Arch package is in [community]? It was confusing that it 404'd on the AUR and didn't install without refreshing package databases.

[svenstaro]

@passcod: @BurntSushi just merged #92 which fixed that.

[passcod]

Awesome and thanks again @svenstaro!

[archon810]

A vote to add to OpenSUSE repos here.

[nihathrael]

Rust is packaged for both ubuntu and debian.

At least for Ubuntu this repository currently doesn't build with the Rust/Cargo versions in the official repository (rust: 1.7.0, cargo: cargo 0.8.0 (built 2016-03-22))

Message:

> cargo build --release                                                                                                                                                                                                                                                                                     master [2b15832]
failed to parse manifest at `<path>/Cargo.toml`

Caused by:
  could not parse input as TOML
Cargo.toml:42:9 expected a key but found an empty string
Cargo.toml:42:9-42:10 expected `.`, but found `'`

[est31]

@nihathrael it works on rust 1.9.0, which will be the one that ubuntu 16.10 ships with in a few weeks.

[BurntSushi]

@nihathrael Right, ripgrep requires Rust 1.9.

[wezm]

@priyadarshan yes, that's correct.

[hmeine]

I filed a request for MacPorts here: https://trac.macports.org/ticket/52416

[sambrightman]

Isn't it good form to download bottles (binaries) in Homebrew-land?

[BurntSushi]

Go check out homebrew core. Aren't most formulas compiled from source?

On Sep 28, 2016 3:12 AM, "Sam Brightman" notifications@github.com wrote:

Isn't it good form to download bottles (binaries) in Homebrew-land?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAb34rt2uQgRnUr3OUVjfJl2rW9v6Y4mks5quhNZgaJpZM4KDG1y
.

[sublee]

Is there any way to get a pre-built binary for Ubuntu? I really want to use ripgrep for my all environments but I don't want to install Rust just for this reason.

Furthermore, I think Linuxbrew is not stable enough so I'm not using it.

[BurntSushi]

@sublee The README contains links to binaries that I distribute.

[sublee]

@BurntSushi Oh, why didn't I find the link. I was a fool. Anyway, thank you for letting me know!

[BurntSushi]

No worries! :-)

[moshen]

Opened Homebrew/homebrew-core#5268 , but it seems to be hung up on Sierra issues.

[carlwgeorge]

I just submitted this for Fedora and EPEL (to be used in RHEL and CentOS).

[BurntSushi]

e.g. find a solution to the hyphenation issue

Yeah it sounds like the textwrap maintainer is amenable to getting this straightened out, so I think we should be good here. Is there a timeline I should be aware of here?

It just means it's harder for other people to take advantage of it. Not only do we have to compile with "--features simd-accel" but we have to condition this on the architectures this actually works on. This logic really should be part of the crates' Cargo.tomls themselves, package maintainers shouldn't be burdened with having to figure this out. Specifically encoding_rs/simd are the crates at issue here, but it affects ripgrep via dependencies.

My problem with this whole thing generating more work is that the simd crate was, and for the foreseeable future, will be an experimental nightly only thing. The Rust ecosystem is moving through a transition period in how we handle SIMD, so there are growing pains here. The thing you desire is exactly where the ecosystem should be headed towards. For example, the regex crate used by ripgrep will auto-configure the use of SIMD at runtime, so that nobody but that code needs to worry about platform support. Unfortunately, some of ripgrep's dependencies (encoding_rs and bytecount, namely) haven't moved to this new model yet, but AFAIK, that is the goal. So for the time being, SIMD support in Rust is split between runtime support (auto-configured) and compile time support (burden pushed to the person running cargo build). I think I agree with you that even compile time support should be handled better, but there's nothing on the roadmap for that AFAIK and nobody has even thought to work on it. Probably because runtime detection is so much nicer when possible.

In other words, if it were up to me, I'd probably forget about packaging optional nightly only Rust features. (For ripgrep, that includes both simd-accel and avx-accel.) In particular, since I imagine you're only packaging stable Rust, and stable Rust cannot build ripgrep with either the simd-accel or avx-accel features anyway, I don't think I can even conceptualize how those features would be used in the Debian package ecosystem.

I thought features [ "pcre2" ] in [package.metadata.deb] meant "the Debian package should compile with --features pcre2". We're not doing that with debcargo at the moment, for any binary crate. My point was how to let crate developers specify which features they want us to build for the Debian package, if not the default feature set. I was suggesting just adding a "debian" feature for that purpose, rather adding that to a tool-specific section.

I see. So let me try to play this back to you... What you want to know is whether, when someone does, apt-get install ripgrep, whether that should automatically come with PCRE2 support or not? My feeling there is that yes, it should, because it's better for users. So I think what you're saying is that you'd like to see a signal for that? So is this all you would want?

[package.metadata.debian]
features = ["pcre2"]

Do you need/want any of the other metadata that's in the package.metadata.deb section?

I'm so sorry that I don't understand Debian's policy better. I'm sure that's part of the cause of what's making this difficult. I really really appreciate your time and patience with me as we figure this out. :-)

[joshtriplett]

@BurntSushi Regarding SIMD: ideally the encoding_rs and bytecount crates will switch to the new SIMD model that works with stable Rust, and until then I don't think it makes sense for Debian to attempt to use those features. I'm assuming that once that happens, we won't need a simd feature at all, and SIMD will Just Work?

Regarding pcre2: I think @infinity0 was suggesting that you could just have a debian feature available, and make that feature depend on pcre2, then have package.metadata.deb use that same feature (or teach cargo-deb to use such a feature automatically). We could then teach debcargo to look for a debian feature and enable it by default if found. In general, we'd always prefer autodetection over configuration when possible. :)

(I'm also hoping one day we can integrate support for things like manpages and completions into cargo so you don't have to do something specific to cargo-deb to install them.)

[BurntSushi]

Regarding SIMD: ideally the encoding_rs and bytecount crates will switch to the new SIMD model that works with stable Rust, and until then I don't think it makes sense for Debian to attempt to use those features. I'm assuming that once that happens, we won't need a simd feature at all, and SIMD will Just Work?

That is my understanding, yes. This is already true for regex. bytecount is in the process of moving to that model. I know less about encoding_rs though and what their specific plans are regarding runtime dispatch of optimized SIMD routines (cc @hsivonen). I know they want to moved to the portable SIMD API, which isn't stable yet, but I don't actually know what their plans are surrounding runtime dispatch.

w.r.t. to adding a Debian Cargo feature, that seems reasonable, yeah. @infinity0 Does that sound good?

[infinity0]

Yeah it sounds like the textwrap maintainer is amenable to getting this straightened out, so I think we should be good here. Is there a timeline I should be aware of here?

4-5 months is the estimated time for the next Debian Stable judging by the previous few releases. But this deadline includes many potential other issues, including future issues not yet discovered, so getting this one out of the away ASAP would be best.

[..] Probably because runtime detection is so much nicer when possible.

I see, thanks for explaining the background of SIMD in rust. I agree runtime detection is best, and this should hopefully include graceful fallbacks for architectures like ppc64 that have no SIMD at all. OK then for now I'll just ignore these optional simd features.

In other words, if it were up to me, I'd probably forget about packaging optional nightly only Rust features. [..] I don't think I can even conceptualize how those features would be used in the Debian package ecosystem.

I just agreed to ignoring the simd-accel feature for now, but in general if absolutely necessary we can enable nightly features for specific crates that require it like the simd crate, simply by setting RUSTC_BOOTSTRAP=1 during the build.

w.r.t. to adding a Debian Cargo feature, that seems reasonable, yeah. @infinity0 Does that sound good?

That sounds good yes.

Do you need/want any of the other metadata that's in the package.metadata.deb section?

No thanks it's OK, it's useful for us to read it as reference, but we have other ways of doing that that fits better with official Debian tooling.

[ignatenkobrain]

I would be slightly against having "debian" feature, because then we will end up with "fedora", "mageia", "opensuse" and thousand of others. it is really up to distribution to decide which features to use.

said that, IMO the most useful features should be in "default" feature.

[joshtriplett]

Long-term, is there a plan to enable pcre2 in default? Or is it likely to always be non-default due to the additional library dependency?

[BurntSushi]

It will always be non-default because of the additional library dependency. I'd like to keep the default build pure Rust for now. Having lots of distro specific features wouldn't be good, I agree.

…

On Mon, Sep 10, 2018, 02:14 Josh Triplett ***@***.***> wrote: Long-term, is there a plan to enable pcre2 in default? Or is it likely to always be non-default due to the additional library dependency? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAb34jLIz0nuc9GFxEvN-BYQ_tjD4cmCks5uZgMugaJpZM4KDG1y> .

[hsivonen]

I know less about encoding_rs though and what their specific plans are regarding runtime dispatch of optimized SIMD routines (cc @hsivonen). I know they want to moved to the portable SIMD API, which isn't stable yet, but I don't actually know what their plans are surrounding runtime dispatch.

The plan is not to add run-time dispatch. encoding_rs relies heavily on inlining, and I don't want to add complexity to support hardware that's so old as to be desupported by major browsers.

Instead of turning off SIMD for all 32-bit x86 users, I hope that Debian decides, like Fedora, that supporting non-SSE2 x86 is no longer worthwhile.

The non-NEON case on ARMv7 is a bit less obsolete than the non-SSE2 case on x86. However, ARMv7 support in encoding_rs is primarily for Android where requiring NEON practically excludes only old Tegra devices. The situation for Linux distros isn't ideal.

fallbacks for architectures like ppc64 that have no SIMD at all.

Do you mean the SIMD support in Rust isn't ready? Surely 64-bit POWER can be trusted to always have SIMD available. Even 32-bit PowerPC had it since G4. (I'm hoping to attempt SIMD-enabling encoding_rs on 32-bit PowerPC once the compiler and standard library support is there.)

[BurntSushi]

The plan is not to add run-time dispatch. encoding_rs relies heavily on inlining, and I don't want to add complexity to support hardware that's so old as to be desupported by major browsers.

That's unfortunate. That means the only way users of ripgrep will get encoding_rs's SIMD optimizations is by enabling compile time flags. (Which is the case today, but I figured Rust's support for runtime detection of CPU features would cause folks to use it so that >SSE2 optimizations can be enabled automatically on CPUs that support it.)

[BurntSushi]

@hsivonen Ah, apologies! I retract my previous complaint. encoding_rs appears to only use sse2 on Intel, so I guess there is no runtime dispatch required on the platform I care most about (x86_64). :-)

[infinity0]

I would be slightly against having "debian" feature, because then we will end up with "fedora", "mageia", "opensuse" and thousand of others. it is really up to distribution to decide which features to use.

It will always be non-default because of the additional library dependency. I'd like to keep the default build pure Rust for now.
Having lots of distro specific features wouldn't be good, I agree.

It sounds like you're reserving the "default" feature to avoid installing extra stuff, how about a standard name like "default+packaged" for distros / package managers that can expect to package everything?

Instead of turning off SIMD for all 32-bit x86 users, I hope that Debian decides, like Fedora, that supporting non-SSE2 x86 is no longer worthwhile.

The non-NEON case on ARMv7 is a bit less obsolete than the non-SSE2 case on x86. However, ARMv7 support in encoding_rs is primarily for Android where requiring NEON practically excludes only old Tegra devices. The situation for Linux distros isn't ideal.

All of this is fine, my point is that this should be clearly documented in the rust-simd crate - so crates depending on rust-simd, such as encoding-rs and all of its dependents like ripgrep, can supply the correct cfg conditionals in their Cargo.toml. Or if they don't, distros like Debian have enough documentation to patch it in so that builds don't break on ppc64 etc. Currently I really have no idea, I have to guess by enabling --feature simd+accel on ripgrep and figure out whether build failures due to encoding_rs, on each of the 10 Debian architectures that have rust, are "my fault" or "your fault".

(edit: then some build failures are even invisible, e.g. some of our x86 autobuilders have sse2 and some don't, and then it's up to cargo test to actually exercise these instructions which we don't do by default due to the possibility of dev-dependency loops)

[infinity0]

Do you mean the SIMD support in Rust isn't ready? Surely 64-bit POWER can be trusted to always have SIMD available. Even 32-bit PowerPC had it since G4.

To clarify this point, which I may not have made clear enough in my previous point - I think rust's own SIMD does work on ppc64 but some crates are using your crate simd as a dependency, which doesn't build on ppc64. However neither ripgrep nor encoding_rs give any architecture conditionals in their Cargo.toml. If I wanted to experiment with enabling simd+accel for ripgrep in Debian, I have to figure out these cfg conditionals myself. But this information is not obvious and should be documented by simd. For example I had no idea it required sse on 32-bit x86 since the build worked on Debian 32-bit x86, probably because that particular builder happened to support sse, and I generally don't investigate successful builds.

[hsivonen]

Ah, apologies! I retract my previous complaint. encoding_rs appears to only use sse2 on Intel, so I guess there is no runtime dispatch required on the platform I care most about (x86_64). :-)

Right. encoding_rs uses very basic 128-bit SIMD, so the kind of SIMD it needs is always present on x86_64 and aarch64.

The flag is there just because SIMD still requires nightly. Once std::simd graduates to the release channel, the plan is to not require any feature flag and to instead look at target features only, so in the future I expect SIMD to be enabled if targeting x86_64, i686, aarch64 or thumbv7neon/armv7neon and be disabled if targeting i586 or armv7/thumbv7. (i686 and i586 used in the Rust target sense and not in the GCC target sense.)

some crates are using your crate simd as a dependency, which doesn't build on ppc64

Sorry about that. I added a note to the README of simd on trunk. The crate was that way when it was transferred to me and I've tried to keep changes to the minimum while waiting for std::simd. (This was already documented in the README for encoding_rs.)

[sylvestre]

And ripgrep is now also available in Ubuntu https://launchpad.net/ubuntu/+source/rust-ripgrep starting from Cosmic! (18.10)

[okdana]

btw, i didn't notice before because the file listing on Debian's package site doesn't work, but these packages are missing the zsh completion function (and the other non-bash ones, i guess). Should i file a bug for that?

ETA: Filed bug # 908807

[sylvestre]

Yes, please report that on the Debian bug tracking system

[BurntSushi]

@sylvestre Woohoo! Awesome!!!

[michalfita]

In Alpine Linux:

nikoleta:~# apk add ripgrep
ERROR: unsatisfiable constraints:
  ripgrep (missing):
    required by: world[ripgrep]
nikoleta:~#

Not present here: https://pkgs.alpinelinux.org/contents?file=ripgrep&path=&name=&branch=edge&repo=main&arch=x86_64

[kpcyrd]

@michalfita this is blocked by alpinelinux/aports#1250 which in turn would need better support for musl libc based systems by rust

[citrus-it]

ripgrep is now available for OmniOS in the official extra repository - https://pkg.omniosce.org/r151028/extra/info/0/ooce%2Ftext%2Fripgrep%400.10.0%2C5.11-151028.0%3A20181208T100739Z

On OmniOS r151028 or above, it's as simple as pkg install ripgrep

[BurntSushi]

@citrus-it A PR that adds it to the README would be great!

[BurntSushi]

I think ripgrep is now in most or all of the major package repositories that I can personally think of, so I'm going to close this. 🎉

PRs are of course welcome as ripgrep gets added to other package repositories.

[lousyd]

@carlwgeorge , the last comment from you is here, discussing a ripgrep tarball with bundled dependencies in order to circumvent the "with rich dependencies" requirement. Did anything ever move forward on that?