Build docs

[gwbres]

Bug report

Any library that depends on ANISE + embed_ephem cannot be documented, because cargo --build doc fails when the crates.io docker tries to deploy it:

# rustc version
rustc 1.82.0-nightly (feeba198f 2024-08-17)# docs.rs version
docsrs 0.6.0 (a6155e2c 2024-08-13)# build log
[INFO] running `Command { std: "docker" "create" "-v" "/home/cratesfyi/workspace-builder/builds/gnss-rtk-0.7.0/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace-builder/builds/gnss-rtk-0.7.0/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace-builder/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace-builder/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "DOCS_RS=1" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "6442450944" "--cpus" "6" "--user" "1001:1001" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:923055f121b5182466d55868a8b05e67af8ba4a3a3f6bad814e953ca3cd3ac2a" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "rustdoc" "--lib" "-Zrustdoc-map" "--config" "build.rustdocflags=[\"--cfg\", \"docsrs\", \"-Z\", \"unstable-options\", \"--emit=invocation-specific\", \"--resource-suffix\", \"-20240817-1.82.0-nightly-feeba198f\", \"--static-root-path\", \"/-/rustdoc.static/\", \"--cap-lints\", \"warn\", \"--extern-html-root-takes-precedence\"]" "--offline" "-Zunstable-options" "--config=doc.extern-map.registries.crates-io=\"https://docs.rs/{pkg_name}/{version}/x86_64-unknown-linux-gnu\"" "-Zrustdoc-scrape-examples" "-j6" "--target" "x86_64-unknown-linux-gnu", kill_on_drop: false }`
[INFO] [stdout] 97e29512810038c770652daf170c1482052e7d980c88a0ba07e5b8a4f97c8cb4
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] running `Command { std: "docker" "start" "-a" "97e29512810038c770652daf170c1482052e7d980c88a0ba07e5b8a4f97c8cb4", kill_on_drop: false }`
[INFO] [stderr] warning: target filter specified, but no targets matched; this is a no-op
[INFO] [stderr]    Compiling libgit2-sys v0.17.0+1.8.1
[INFO] [stderr]    Compiling anise v0.4.2
[INFO] [stderr]     Checking platform-dirs v0.3.0
[INFO] [stderr]     Checking rand v0.8.5
[INFO] [stderr]     Checking arrow-csv v52.2.0
[INFO] [stderr]     Checking portable-atomic v1.7.0
[INFO] [stderr]     Checking tabled v0.15.0
[INFO] [stderr] error: failed to run custom build command for `anise v0.4.2`
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/anise-6a27233e82fd655c/build-script-build` (exit status: 101)
[INFO] [stderr]   --- stderr
[INFO] [stderr]   thread 'main' panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/anise-0.4.2/build.rs:31:40:
[INFO] [stderr]   failed to create directory "/opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/anise-0.4.2/../data": Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" }
[INFO] [stderr]   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[INFO] [stderr] warning: build failed, waiting for other jobs to finish...
[INFO] running `Command { std: "docker" "inspect" "97e29512810038c770652daf170c1482052e7d980c88a0ba07e5b8a4f97c8cb4", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "97e29512810038c770652daf170c1482052e7d980c88a0ba07e5b8a4f97c8cb4", kill_on_drop: false }`
[INFO] [stdout] 97e29512810038c770652daf170c1482052e7d980c88a0ba07e5b8a4f97c8cb4

what the build script does for embed_ephem should not happen when building the docs. I'm not sure what options we have to either bypass the build script or detect within that script that docrs is running

[ChristopherRabotin]

This is a problem. Would a reasonable fix be to print a major warning but not fail?

[gwbres]

The problem comes from attempting to create a folder, and apparently, they deploy a docker without "write" permissions.
All we need is to bypass this step somehow

[gwbres]

Hello,

is there any plan to fix this and propose a new version ?
the following crates cannot be documented because of this:

• gnss-rtk
• rinex-qc
• sp3 once upgraded to Orbit (about to be completed)
• rinex once upgraded to Orbit (not completed yet)

[ChristopherRabotin]

Yes, I'll look into it and it will be resolved by the end of September, probably sooner. Ideally, we could release a stable version of hifitime, anise, and nyx all at once, but I don't think that's realistic with the current timelines.

[gwbres]

no emergency on my side. I was interested in releasing new rinex versions, it is now feasible since your latest tags, I marked them -alpha as well.
I still have a few tasks to complete until 0.17 is out, some will take a while. I will also need the documentation to build fine, and it's probably better to wait for -stable nyx for us to move to -stable as well.

• I need to re-establish some operations that are broken in rinex-cli, like CSV export
• conclude the orbital projections
• conclude OBS RINEX reporting
• conclude Orbit transition (SP3[90%] + RINEX)

[ChristopherRabotin]

Let me know if the changes in the commit above aren't enough. I think that they are.

[gwbres]

Let me know if the changes in the commit above aren't enough. I think that they are.

Seems good enough ! It will most likely never happen to real users

[ChristopherRabotin]

This bug is likely still present: https://docs.rs/crate/anise/0.4.3/builds/1345906

[gwbres]

It's hard to tell what the actual problem is, it seemed more obvious previously

is that ?

error: #[derive(RustEmbed)] folder '/opt/rustwide/workdir/../data/' does not exist. cwd: '/opt/rustwide/workdir'

if so, the data folder is either

• missing when the docker is deployed, due to either deployment options or Cargo.toml setup (is /data excluded and not shipped ?)
• $CARGO_MANIFEST_DIR is not currectly defined, but I doubt so, it would not end up as '/opt/rustwide/workdir/../data/'

personnaly, I'd rather try to bypass build::main() completely when docrs is running

[gwbres]

the problem is in the RustEmbed macro itself, and what it's doing (where it is doing it) in the docker setup.

RustEmbed most likely creates a file, and that location does not exist.
build.rs is actually not the problem ! (or, no longer is)

[ChristopherRabotin]

I think there are two problems:

1. There is a typo in the proc macro: "warning: unexpected cfg condition name: docrs". That should be docsrs.
2. The data folder does not exist which is why the RustEmbed fails. I think that to fix this, we just need to add an empty file in the data folder and include it in the crate build. Then the data folder will exist when the crate is downloaded from crates.io.

So it should be a quick fix.

[ChristopherRabotin]

Used the include instruction as per the docs to explicitly include a .cargokeep folder in the data folder. I also added a relevant test in the CI where the data folder is deleted and then rebuilt with that .cargokeep file.

[ChristopherRabotin]

I don't know why this didn't work ... the data folder should exist: https://github.com/nyx-space/anise/blob/master/Cargo.toml#L27 .

# rustc version
rustc 1.83.0-nightly (18b1161ec 2024-10-02)# docs.rs version
docsrs 0.6.0 (d5a37845 2024-09-24)# build log
[INFO] running `Command { std: "docker" "create" "-v" "/home/cratesfyi/workspace/builds/anise-0.4.4/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace/builds/anise-0.4.4/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "DOCS_RS=1" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "6442450944" "--cpus" "6" "--user" "1001:1001" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:6252d7ea7fb8caaf7af6d800c5fb250a9cd862b9a7f9508afb3c54fa7fe1102e" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "rustdoc" "--lib" "-Zrustdoc-map" "--all-features" "--config" "build.rustdocflags=[\"--cfg\", \"docsrs\", \"-Z\", \"unstable-options\", \"--emit=invocation-specific\", \"--resource-suffix\", \"-20241002-1.83.0-nightly-18b1161ec\", \"--static-root-path\", \"/-/rustdoc.static/\", \"--cap-lints\", \"warn\", \"--extern-html-root-takes-precedence\"]" "--offline" "-Zunstable-options" "--config=doc.extern-map.registries.crates-io=\"https://docs.rs/{pkg_name}/{version}/x86_64-unknown-linux-gnu\"" "-Zrustdoc-scrape-examples" "-j6" "--target" "x86_64-unknown-linux-gnu", kill_on_drop: false }`
[INFO] [stdout] 94a4085e5b203e903330ef6499f36ff954fcf20cfb96a5e110c3c0d87c60dd8b
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] running `Command { std: "docker" "start" "-a" "94a4085e5b203e903330ef6499f36ff954fcf20cfb96a5e110c3c0d87c60dd8b", kill_on_drop: false }`
[INFO] [stderr] warning: target filter specified, but no targets matched; this is a no-op
[INFO] [stderr]  Documenting anise v0.4.4 (/opt/rustwide/workdir)
[INFO] [stderr] error: #[derive(RustEmbed)] folder '/opt/rustwide/workdir/../data/' does not exist. cwd: '/opt/rustwide/workdir'
[INFO] [stderr]   --> src/almanac/embed.rs:11:1
[INFO] [stderr]    |
[INFO] [stderr] 11 | / #[folder = "$CARGO_MANIFEST_DIR/../data/"]
[INFO] [stderr] 12 | | #[include = "de440s.bsp"]
[INFO] [stderr] 13 | | #[include = "pck11.pca"]
[INFO] [stderr] 14 | | struct AstroData;
[INFO] [stderr]    | |_________________^
[INFO] [stderr] 
[INFO] [stderr] error: could not document `anise`
[INFO] running `Command { std: "docker" "inspect" "94a4085e5b203e903330ef6499f36ff954fcf20cfb96a5e110c3c0d87c60dd8b", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "94a4085e5b203e903330ef6499f36ff954fcf20cfb96a5e110c3c0d87c60dd8b", kill_on_drop: false }`
[INFO] [stdout] 94a4085e5b203e903330ef6499f36ff954fcf20cfb96a5e110c3c0d87c60dd8b

https://docs.rs/crate/anise/0.4.4/builds/1378203

[gwbres]

include and exclude are tied to the crates io publication right ?
to my understanding, it specifies what should exist in crates io database

It's probably related to how they setup their docker environment, maybe we should reach out to the rust community.

FYI "-v" "/home/cratesfyi/workspace/builds/anise-0.4.4/source:/opt/rustwide/workdir:ro,
it seems the entire workspace is read only (first errors we encountered), should be bypassed if include does what we think it does.

The docrs job is probably tied to anise and not the workspace, are we sure the steps happens at the location we think it happens ?

[gwbres]

From the first response, I would recommend wrapping the folder creation inside:

if std::env::var("DOCS_RS").is_err() {
    // ... your code here ...
}

[ChristopherRabotin]

Thanks for opening the issues on the rust organization. I had not seen them. From the tarball, it's clear that the data folder does not exist. Would it be acceptable if the embed_ephem was simply not documented?

Essentially, change:

#[cfg(feature = "embed_ephem")]
#[cfg_attr(docsrs, doc(cfg(feature = "embed_ephem")))]
mod embed;

To:

#[cfg(feature = "embed_ephem")]
#[cfg(not(feature = "docsrs"))]
mod embed;

[ChristopherRabotin]

Redone in #335