Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add exhaustive/extensive tests #364

Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

Conversation

tgross35
Copy link
Contributor

Builds on top of #348 so it includes those commits.

@tgross35 tgross35 marked this pull request as draft December 21, 2024 06:55
@tgross35
Copy link
Contributor Author

tgross35 commented Dec 21, 2024

A start at "extensive" tests that run up to u32::MAX inputs, only on requested functions. Requested tests are set via env with a few different options:

# Run one or more exhaustive test
LIBM_EXTENSIVE_TESTS=sinf,cosf cargo t --features test-multiprecision --release -- extensive
# Run all `f32` or `f64` tests
LIBM_EXTENSIVE_TESTS=all_f32 cargo t --features test-multiprecision --release -- extensive
# Run everything
LIBM_EXTENSIVE_TESTS=all cargo t --features test-multiprecision --release -- extensive

For single-arg f32 it tests with all possible values in the domain. For f64, it runs u32::MAX tests with a logspace. For functions with N > 1 arguments, it currently does a logspace for each arg, each with u32::MAX^(1/N) steps.

I ran a handful of tests on my 24-core machine, the current setup with Rayon ranges about 10m-1h depending on the test. That can probably be improved, I haven't looked at this from a perf perspective at all (though one thing I noticed quickly from htop is there is a lot of time spent in kthreads, maybe I need to double check if Rayon is doing batching).

After I get this a bit better sorted out, I am going to hook up CI to run relevant tests when source files change. Current state is very scratchy.

@beetrees you may be interested since I know you suggested this a few times.

@tgross35
Copy link
Contributor Author

I'm using the mimic harness so I can skip tests at runtime and show a progress bar, sample output.

running 90 tests
test mp_extensive_acos       ... ignored, extensive tests are only run if specified in RUST_LIBM_EXTENSIVE
test mp_extensive_acosf      ... starting extensive tests for `acosf`
[9m  100%] #################### acosf 4,294,967,295/4,294,967,295: 7,672,836.6102/s eta 0s
ok
test mp_extensive_acosh      ... ignored, extensive tests are only run if specified in RUST_LIBM_EXTENSIVE
test mp_extensive_acoshf     ... starting extensive tests for `acoshf`
[5m  73 %] ###############----- acoshf 3,139,620,000/4,294,967,295: 3,544,521.3401/s eta 5m (1.6715916e36,)

@tgross35
Copy link
Contributor Author

(though one thing I noticed quickly from htop is there is a lot of time spent in kthreads, maybe I need to double check if Rayon is doing batching)

Way faster with chunking, acosf (sample test with a small domain) completes in two minutes on my machine, asinhf (sample with an unbounded domain) completes in 5 minutes.

@tgross35 tgross35 force-pushed the extensive branch 3 times, most recently from ee66993 to 6da3286 Compare December 22, 2024 05:53
@tgross35 tgross35 changed the title [WIP] Add exhaustive/extensive tests Add exhaustive/extensive tests Dec 22, 2024
@tgross35 tgross35 force-pushed the extensive branch 17 times, most recently from b0ace52 to 233f651 Compare December 26, 2024 10:51
tgross35 and others added 4 commits December 26, 2024 10:57
Add a constant for negative pi and provide a standalone const
`from_bits`, which can be combined with what we already had in
`hex_float`. Also provide another default method to reduce what needs to
be provided by the macro.
Introduce `f8`, which is an 8-bit float compliant with IEEE-754. This
type is useful for testing since it is easily possible to enumerate all
values.
Create a type representing a function's domain and a test that does a
logarithmic sweep of points within the domain.
Introduce a generator that will tests various points of interest
including zeros, infinities, and NaNs.
For visualization, add a simple script for generating scatter plots and
a binary (via examples) to plot the inputs given various domains.
This will enable us to `include!` the file to access these types in
`libm-test`, rather than somehow reproducing the types as part of the
macro. Ideally `libm-test` would just `use` the types from `libm-macro`
but proc macro crates cannot currently export anything else.

This also adjusts naming to closer match the scheme described in
`libm_test::op`.
These types from `libm_macros` provide a way to get information about an
operation at runtime, rather than only being encoded in the type system.
Include the file and reexport relevant types.
New API to support upcoming tests.
Rather than passing names or identifiers, just pass `CheckCtx` in a few
more places.
@tgross35 tgross35 force-pushed the extensive branch 2 times, most recently from f2a17f9 to 3b82822 Compare December 26, 2024 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants