doesn't compile - clap issue

[alperyilmaz]

Hi,
I'm not a rust programmer so I might be doing something wrong here but I couldn't compile frawk.

I tried following ways to install

cargo +nightly install --path .
cargo  install frawk
cargo +nightly install --path . --no-default-features --features use_jemalloc,allow_avx2,unstable
cargo +nightly install --path . --no-default-features --features allow_avx2,unstable

all failed with the following message:

error[E0599]: no method named `multiple` found for struct `clap::Arg` in the current scope
   --> src/main.rs:306:15
    |
306 |              .multiple(true)
    |               ^^^^^^^^ method not found in `clap::Arg<'_>`

error[E0599]: no method named `multiple` found for struct `clap::Arg` in the current scope
   --> src/main.rs:329:15
    |
329 |              .multiple(true)
    |               ^^^^^^^^ method not found in `clap::Arg<'_>`

error[E0599]: no method named `multiple` found for struct `clap::Arg` in the current scope
   --> src/main.rs:350:15
    |
350 |              .multiple(true))
    |               ^^^^^^^^ method not found in `clap::Arg<'_>`

For more information about this error, try `rustc --explain E0599`.
warning: `frawk` (bin "frawk") generated 2 warnings
error: failed to compile `frawk v0.4.2 (/usr/src/myapp)`, intermediate artifacts can be found at `/usr/src/myapp`

I'm using ubuntu 20.04 and I ran rustup default nightly for nightly compilation. Here are the version numbers in case they are needed:

# cargo --version
cargo 1.56.0-nightly (e96bdb0c3 2021-08-17)
# rustc --version
rustc 1.56.0-nightly (ad02dc46b 2021-08-26)
# rustup --version
rustup 1.24.3 (ce5817a94 2021-05-31)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.56.0-nightly (ad02dc46b 2021-08-26)`

[alperyilmaz]

In different environments, there are different problems I guess..
I tried compiling frawk in Docker and i got another error. Here's the Dockerfile contents

FROM rustlang/rust:nightly-bullseye

RUN echo "deb http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye main" >> /etc/apt/sources.list && \
    echo "deb-src http://apt.llvm.org/bullseye/ llvm-toolchain-bullseye main" >> /etc/apt/sources.list

RUN wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -

RUN apt update && install llvm-11-dev liblld-11-dev

ENV LLVM_SYS_100_PREFIX=/usr/lib/llvm-11

RUN cargo install frawk

and this is the error I got:

error: could not find native static library `Polly`, perhaps an -L flag is missing?

error: could not compile `llvm-sys` due to previous error
warning: build failed, waiting for other jobs to finish...
error: failed to compile `frawk v0.4.2`, intermediate artifacts can be found at `/tmp/cargo-install19Kek1`

Caused by:
  build failed

Not sure if llvm-10 is required but I couldn't find one in debian-bullseye repos, thus installed llvm-11. I'll try another container which can install llvm-10..

[alperyilmaz]

I tried another environment and got the clap error back again.
Here's the Dockerfile

FROM rustlang/rust:nightly-alpine3.12

RUN apk add alpine-sdk llvm10 llvm10-dev

ENV LLVM_SYS_100_PREFIX=/usr/lib/llvm10

RUN cargo install frawk

and same error message:

error[E0599]: no method named `multiple` found for struct `clap::Arg` in the current scope
   --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/frawk-0.4.2/src/main.rs:306:15
    |
306 |              .multiple(true)
    |               ^^^^^^^^ method not found in `clap::Arg<'_>`

error[E0599]: no method named `multiple` found for struct `clap::Arg` in the current scope
   --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/frawk-0.4.2/src/main.rs:329:15
    |
329 |              .multiple(true)
    |               ^^^^^^^^ method not found in `clap::Arg<'_>`

error[E0599]: no method named `multiple` found for struct `clap::Arg` in the current scope
   --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/frawk-0.4.2/src/main.rs:350:15
    |
350 |              .multiple(true))
    |               ^^^^^^^^ method not found in `clap::Arg<'_>`

For more information about this error, try `rustc --explain E0599`.
error: failed to compile `frawk v0.4.2`, intermediate artifacts can be found at `/tmp/cargo-installMFmKQX`

Caused by:
  could not compile `frawk` due to 3 previous errors

a binary of frawk would be great.. since its compilation seems not straigntforward.. eager to compile and test frawk soon..

[ezrosent]

Thanks for the detailed report! Providing binaries for frawk is absolutely on my roadmap, so far I've had trouble producing static binaries with comparable performance but it is something I'm looking into. Your earlier comments are also a good reminder for me to update the LLVM dependency.

In the short term I'll work to fix the clap compilation issue. This is not something I'd generally expect to be a barrier to compiling from source, but the dependency frawk has is clap v3, which is in beta and hence less stable. If this continues to happen I may migrate back to clap v2, which will make this less common.

Thanks again, I'll try to get this sorted soon.

[alperyilmaz]

hi, clap issue has been resolved with commit 81a67738. Also, I was able to build it within Docker successfully.

Here's the Dockerfile contents

FROM ubuntu:20.04

RUN apt update && apt install -y wget ca-certificates build-essential zlib1g-dev libllvm10 llvm-10-dev && rm -rf /var/lib/apt/lists/*

ENV RUSTUP_HOME=/usr/local/rustup \
    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:$PATH \
    RUST_VERSION=1.56.0 \
    LLVM_SYS_100_PREFIX=/usr/lib/llvm10

RUN set -eux; \
    dpkgArch="$(dpkg --print-architecture)"; \
    case "${dpkgArch##*-}" in \
        amd64) rustArch='x86_64-unknown-linux-gnu' ;; \
        arm64) rustArch='aarch64-unknown-linux-gnu' ;; \
        *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
    esac; \
    \
    url="https://static.rust-lang.org/rustup/dist/${rustArch}/rustup-init"; \
    wget "$url"; \
    chmod +x rustup-init; \
    ./rustup-init -y --no-modify-path --profile minimal --default-toolchain nightly --default-host ${rustArch}; \
    rm rustup-init; \
    chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
    rustup --version; \
    cargo --version; \
    rustc --version;

The official image with Alpine was using musl, so probably not compatible with gnu. The official rust-lang images were based on Debian images and they didn't have llvm-10. So, I had to create an image based on Ubuntu.

After cloning the repo, the compiling can be done with the following command (replace the image name before running):

docker run --rm -it --user "$(id -u)":"$(id -g)" -e CARGO_TARGET_DIR=/tmp/myapp -v "$PWD":/tmp/myapp -w /tmp/myapp alperyilmaz/rust-nightly-llvm cargo +nightly install --path .

The executable file frawk is around 48MB big. I thought it was because the executable is static but I checked it and it's dynamically linked.

Thanks for this great piece of software. Awk was already great and now it will be greater when it's running faster and in parallel.

[ezrosent]

Glad this works, and thanks for the kind words! I've uploaded 0.4.3 to crates.io for good measure. And thanks as well for the Docker commands; those look extremely useful. I'll close this issue for now; let me know if you run into more issues with frawk.

[alperyilmaz]

Although issue is closed, I wanted to document compile instructions for frawk 0.4.4
Earlier Dockerfile fails since cargo nightly is asking for LLVM-12

FROM ubuntu:20.04

RUN apt update && apt install -y wget ca-certificates build-essential zlib1g-dev libllvm12 libllvm-12-ocaml-dev llvm-12-linker-tools libclang-common-12-dev && rm -rf /var/lib/apt/lists/*

ENV RUSTUP_HOME=/usr/local/rustup \
    CARGO_HOME=/usr/local/cargo \
    PATH=/usr/local/cargo/bin:$PATH \
    RUST_VERSION=1.56.0 \
    LLVM_SYS_120_PREFIX=/usr/lib/llvm-12

RUN set -eux; \
    dpkgArch="$(dpkg --print-architecture)"; \
    case "${dpkgArch##*-}" in \
        amd64) rustArch='x86_64-unknown-linux-gnu' ;; \
        arm64) rustArch='aarch64-unknown-linux-gnu' ;; \
        *) echo >&2 "unsupported architecture: ${dpkgArch}"; exit 1 ;; \
    esac; \
    \
    url="https://static.rust-lang.org/rustup/dist/${rustArch}/rustup-init"; \
    wget "$url"; \
    chmod +x rustup-init; \
    ./rustup-init -y --no-modify-path --profile minimal --default-toolchain nightly --default-host ${rustArch}; \
    rm rustup-init; \
    chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
    rustup --version; \
    cargo --version; \
    rustc --version;

build the image

docker build . -t alperyilmaz/rust-nightly-llvm

then go to frawk source folder and run the following command:

docker run --rm -it -e CARGO_TARGET_DIR=/tmp/myapp -v "$PWD":/tmp/myapp -w /tmp/myapp alperyilmaz/rust-nightly-llvm cargo +nightlyinstall --path .

due to permission issues the compilation is run as root, so you might need to run chown user:user frawk with your own user name.

[ezrosent]

Thank you for updating these instructions! I'll look into linking to your comments on this thread from the README.

[alperyilmaz]

that would be great.. I'll try to maintain it as well. But, can not promise 100% success since llvm is kind of tricky.

[eggplants]

@alperyilmaz I'm curious, so I'll ask why do you delete /var/lib/apt/lists/*? I am wondering if it is for speed.

[alperyilmaz]

Hi, it's for making images slightly smaller. When you run "apt update" it downloads bunch of files which might take up 300-500 minutes of space depending on your repo list. So, I was trying to make the image as small as possible.