Add distribution plots to `rand_distr` documentation #1434

MichaelOwenDyer · 2024-04-09T16:50:26Z

Added a CHANGELOG.md entry

Summary

This PR adds some graphs to the documentation of different distributions in rand_distr to illustrate their behavior.

Motivation

Make the distributions in this crate easier to understand via visual aids.

Details

The diagrams will be hosted in https://github.com/rust-random/charts (see this PR) and links will be inserted into the documentation.

Closes #131.

Documentation Progress:

MichaelOwenDyer · 2024-04-09T16:58:02Z

Need to look into it more, but maybe whatever solution we decide on here could also be used for #278

dhardy · 2024-04-10T09:08:41Z

However, this also adds a transitive dependency to syn, which is not great.

Not an issue: you already feature-gated it.

The diagrams are generated with Python code, which is included in the PR for reference.

Good to have in the same repo, but it doesn't need to be in the generated crate package. Could you try investigating why this happens? We already have an explicit include rule in the Cargo.toml, which seems like it should be enough.

We should also talk about the best place to put the diagrams (currently there is one assets folder for rand and another for rand_distr, but this might not work).

We don't package any of these files except by embedding, so we don't need an assets dir and we don't need these separated by crate, but I prefer that the generated resources are next to the generating script. Having the assets in crate-specific dirs but not the script is especially weird. Maybe just move diagrams/src to assets/src, also splitting by crate?

Or, simpler: forget Bernoulli (doesn't really need it) and put all the rest under rand_distr/plots. (Also solves my nit below.)

Plots

Bernoulli

X-axis values should be false, true; not 0, 1.

Binomial

Can we add a second parameterisation, e.g. n=6, p=0.5?

Cauchy

👍 (straight copy from WP, I see)

newpavlov · 2024-04-10T09:13:08Z

I think SVG would be a better fit (separate files, not embedded into HTML). You should be able to generate the same plots with Python without issues.

dhardy · 2024-04-10T09:58:58Z

@newpavlov you missed the discussions in #131 (specifically, how to make this work on docs.rs).

SVG appears larger in some cases, smaller in others. It does scale better.

newpavlov · 2024-04-10T10:19:08Z

Ah, I misread the discussion and thought it was about embedding SVG into HTML. I don't think the terrible diff objection is important, plus we probably should use compression either way (i.e. use SVGZ instead of plain SVG). Though there may be some issues with browser support of SVGZ files.

In RustCrypto we use links to images which reside in the RustCrypto/media repository, e.g. see docs for the ctr crate. It means that we rely on raw.githubusercontent.com instead of packaging the image together with crate, but since the images are purely for illustrative purposes, I think it's a reasonable risk to take.

…agrams

MichaelOwenDyer · 2024-04-10T12:27:13Z

Good to have in the same repo, but it doesn't need to be in the generated crate package. Could you try investigating why this happens? We already have an explicit include rule in the Cargo.toml, which seems like it should be enough.

I think it may be because the folder was named src. I've moved it and renamed it to py, could you please check again?

forget Bernoulli (doesn't really need it) and put all the rest under rand_distr/plots

Got it. That also removes the need for the added dependency on rand, now it only remains in rand_distr.

Binomial
Can we add a second parameterisation, e.g. n=6, p=0.5?

I went ahead and added n=10, p=0.6 because I think it looks better next to n=10, p=0.2. If you think having a demonstration of a different n is important, then I could experiment some more. Maybe something like n=20, p=0.5 would look better.

I think SVG would be a better fit (separate files, not embedded into HTML). You should be able to generate the same plots with Python without issues.

I prefer SVG as well. However, I just did the switch and the diff count exploded to +11,000. Wowza.

newpavlov · 2024-04-10T12:37:56Z

The Python script folder should probably be explicitly excluded from packaged crate (check if it gets included by running cargo package).

It also may be worth to consider the separate repo approach described above. It will be a simpler solution than relying on embed-doc-image and will result in a smaller package size. I will leave the final decision to @dhardy.

MichaelOwenDyer · 2024-04-10T13:40:17Z

The Python script folder should probably be explicitly excluded from packaged crate (check if it gets included by running cargo package).

Ah, thanks for the tip! According to this include and exclude should not be used at the same time. But I found the problem, had to change src/ to /src :)

It also may be worth to consider the separate repo approach described above. It will be a simpler solution than relying on embed-doc-image and will result in a smaller package size. I will leave the final decision to @dhardy.

Certainly. I'm not attached to embedding them if we find a better solution.

dhardy · 2024-04-10T14:38:32Z

I prefer SVG as well. However, I just did the switch and the diff count exploded to +11,000. Wowza.

Diffs normally ignore binary files; this is just because SVG is text based? If so, switching to SVGZ may avoid this?

had to change src/ to /src

Aha!

It also may be worth to consider the separate repo approach described above.

The only real issue is versioning. I see usage of commit hashes in the URLs that ctr uses; this or tags works.

There is also the minor issue that repo clones don't include all docs, but we already have that with the book.

The other issue is that anyone trying to make themselves a copy of the docs for offline usage likely won't have the plots, but that's not critical.

Considering these plots aren't directly tied to the Rust code in any way, a separate repo probably is the way to go.

MichaelOwenDyer · 2024-04-10T15:27:08Z

Diffs normally ignore binary files; this is just because SVG is text based? If so, switching to SVGZ may avoid this?

Yes, SVGZ is better in this regard. However embed-doc-image doesn't support it.

a separate repo probably is the way to go.

Okay then! Want to create it?

dhardy · 2024-04-10T15:43:43Z

Okay then! Want to create it?

https://github.com/rust-random/charts

MichaelOwenDyer · 2024-04-10T16:35:19Z

Can you authorize me to push in that repo, or should I make a fork?

dhardy · 2024-04-10T16:37:19Z

@MichaelOwenDyer you should already have admin privileges on that repo.

MichaelOwenDyer · 2024-04-10T16:48:04Z

Hm, strange, I am getting a 403 whenever I try to push.

Edit: Nevermind, I logged in using a GitHub token and then it worked.

vks · 2024-04-26T10:26:12Z

Diffs normally ignore binary files; this is just because SVG is text based?

We can tell Git via attributes to treat the files as binary if necessary.

# Conflicts: # rand_distr/src/geometric.rs # rand_distr/src/hypergeometric.rs

dhardy · 2024-06-19T16:03:29Z

@dhardy, could you take a look at the docs for the Geometric distribution and tell me if you think the plot is consistent?

The way Wikipedia puts it is that there are two definitions of the distribution. The code and the PMF count the number of failures, while your plot and description count until the first success.

dhardy · 2024-06-19T16:15:47Z

Also, is there a reason we use σ for the scale parameter in Gumbel instead of β which I see in the literature?

That's presumably an error in the documentation since the code just uses scale internally.

MichaelOwenDyer · 2024-06-19T21:34:00Z

The way Wikipedia puts it is that there are two definitions of the distribution.

Okay, yes I saw that too. I will adjust the x-axis of the plot to start at 0.

That's presumably an error in the documentation since the code just uses scale internally.

I'll fix it then 🤠

MichaelOwenDyer · 2024-06-19T21:58:23Z

Another thing I noticed: We are using a in the documentation and implementation for the Zeta distribution, but s is what I am seeing online. Any reason for this?

Also, should I make an effort to embed some more links to Wikipedia across the documentation? Some already exist.

dhardy · 2024-06-20T06:55:23Z

Another thing I noticed: We are using a in the documentation and implementation for the Zeta distribution, but s is what I am seeing online. Any reason for this?

@vks implemented this (#1136). @vks?

Also, should I make an effort to embed some more links to Wikipedia across the documentation? Some already exist.

WP is a useful resource; I guess this would be convenient for some people but not important.

vks · 2024-06-20T14:52:31Z

@MichaelOwenDyer Can you be more specific? The notation and naming are not very consistent across the literature. For example, what we call Zeta is sometimes called Zipf, or vice versa.

MichaelOwenDyer · 2024-06-20T14:59:10Z

@vks I'm referring to the struct at zipf::Zeta. zipf::Zipf uses s already, but zipf::Zeta uses a. I didn't see anything online which lists the Zeta parameter as a, see wikipedia: https://en.wikipedia.org/wiki/Zeta_distribution

vks · 2024-06-20T20:50:43Z

I guess the a comes from the reference, but I no longer have access to it to check. I think I did not want to change it in the implementation to avoid getting confused when implementing the algorithm from the paper.

I'm fine with changing it to s now, as you say it's more consistent with our Zipf docs, and the usual parameter name chosen for the Riemann zeta function.

MichaelOwenDyer · 2024-06-21T21:20:55Z

Okay. We may want to do that in a different PR since ZetaError::ATooSmall is pub, and I believe renaming that would be a breaking change.

dhardy · 2024-07-08T08:46:08Z

@MichaelOwenDyer I think this just needs a cargo fmt and then can be merged.

MichaelOwenDyer · 2024-07-08T09:45:07Z

Thanks for the nudge, I've had a lot on my plate recently. Will finalize this today

- Change parameter of Gumbel from σ to β - Skew normal now uses the correct symbols ξ, ω, α - Other tweaks

MichaelOwenDyer · 2024-07-08T11:31:30Z

Okay @dhardy, I've made my final changes. I added Wikipedia links to the documentation and tried to make it look as consistent as possible. I would appreciate you taking one last look at it and if it looks good to you then I would say its ready to merge :)

dhardy · 2024-07-08T10:34:11Z

rand_distr/src/pareto.rs

-//! The Pareto distribution.
+//! The Pareto distribution `Pareto(α, xₘ)`.

 use crate::{Distribution, OpenClosed01};
 use core::fmt;
 use num_traits::Float;
 use rand::Rng;

-/// The Pareto distribution `Pareto(scale, shape)`.
+/// The Pareto distribution `Pareto(α, xₘ)`.
 ///
 /// The Pareto distribution is a continuous probability distribution with
-/// parameters `scale` (`α`) and `shape` (`x`<sub>`m`</sub> or `k`).
+/// parameters `scale` (`α`) and `shape` (`xₘ` or `k`).
 ///
 /// # Plot
 ///
 /// The following plot shows the Pareto distribution with various values of
 /// `scale` and `shape`.
 /// Note how the scale parameter `α` corresponds to the height of the jump
-/// in density at `x = x`<sub>`m`</sub>, and to the rate of decay in the tail.
+/// in density at `x = xₘ`, and to the rate of decay in the tail.


I think the parameter names got confused somewhere — looks like α is shape. Also, match argument order of the constructor: scale comes first.

Okay, I think I fixed it; please take another look. The plot still lists them in the other order (α, xₘ) but I guess that's okay.

rand_distr/src/skew_normal.rs

rand_distr/src/frechet.rs

MichaelOwenDyer · 2024-07-08T18:15:25Z

Okay, I'm going to go ahead and merge this :D

MichaelOwenDyer added 2 commits April 9, 2024 01:09

Intermediate solution

9b02d66

First few diagrams added

2ea1da3

MichaelOwenDyer added 4 commits April 10, 2024 13:30

Remove bernoulli.png, move python src to diagrams folder, add more di…

3187495

…agrams

Embed more diagrams

e4cfd9d

Add back python code in new dir

52719b4

Switch to svg files

f656d82

MichaelOwenDyer added 2 commits April 10, 2024 15:40

Exclude py src from package

bcc0228

Add python method stubs for all dists

6dc035c

MichaelOwenDyer mentioned this pull request Apr 10, 2024

Add diagrams for rand_distr rust-random/charts#1

Merged

32 tasks

MichaelOwenDyer added 3 commits April 10, 2024 19:02

Remove python and embedding

00ba897

Remove plots

6bc8145

Documentation binomial to hypergeometric

741d82e

MichaelOwenDyer added 3 commits May 24, 2024 14:09

Add rest of plots

af53cb2

Merge branch 'refs/heads/master' into dist-diagrams-in-docs

91028d2

# Conflicts: # rand_distr/src/geometric.rs # rand_distr/src/hypergeometric.rs

Update documentation

92010f2

Update documentation

2d8de9a

MichaelOwenDyer mentioned this pull request Jun 19, 2024

Update x-axis of Geometric distribution rust-random/charts#5

Merged

Update Inverse Gaussian and Normal

2319a65

dhardy approved these changes Jun 23, 2024

View reviewed changes

MichaelOwenDyer added 4 commits July 8, 2024 12:25

Update documentation

9829ca1

- Change parameter of Gumbel from σ to β - Skew normal now uses the correct symbols ξ, ω, α - Other tweaks

Add Wikipedia links

74ec228

Consistency

e105c59

fmt

6165645

dhardy requested changes Jul 8, 2024

View reviewed changes

dhardy reviewed Jul 8, 2024

View reviewed changes

rand_distr/src/frechet.rs Outdated Show resolved Hide resolved

MichaelOwenDyer added 4 commits July 8, 2024 13:41

Change order of Frechet parameters

b1f5252

Fix Pareto parameters

00503f4

Fix ξ name

4428023

Add skew normal wikipedia link

4a880b8

dhardy approved these changes Jul 8, 2024

View reviewed changes

MichaelOwenDyer merged commit 17746a1 into rust-random:master Jul 8, 2024
14 checks passed

MichaelOwenDyer deleted the dist-diagrams-in-docs branch July 9, 2024 11:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add distribution plots to `rand_distr` documentation #1434

Add distribution plots to `rand_distr` documentation #1434

MichaelOwenDyer commented Apr 9, 2024 •

edited

Loading

MichaelOwenDyer commented Apr 9, 2024

dhardy commented Apr 10, 2024

newpavlov commented Apr 10, 2024

dhardy commented Apr 10, 2024

newpavlov commented Apr 10, 2024 •

edited

Loading

MichaelOwenDyer commented Apr 10, 2024

newpavlov commented Apr 10, 2024 •

edited

Loading

MichaelOwenDyer commented Apr 10, 2024

dhardy commented Apr 10, 2024

MichaelOwenDyer commented Apr 10, 2024

dhardy commented Apr 10, 2024

MichaelOwenDyer commented Apr 10, 2024

dhardy commented Apr 10, 2024

MichaelOwenDyer commented Apr 10, 2024 •

edited

Loading

vks commented Apr 26, 2024

dhardy commented Jun 19, 2024

dhardy commented Jun 19, 2024

MichaelOwenDyer commented Jun 19, 2024

MichaelOwenDyer commented Jun 19, 2024

dhardy commented Jun 20, 2024

vks commented Jun 20, 2024

MichaelOwenDyer commented Jun 20, 2024

vks commented Jun 20, 2024 •

edited

Loading

MichaelOwenDyer commented Jun 21, 2024

dhardy commented Jul 8, 2024

MichaelOwenDyer commented Jul 8, 2024

MichaelOwenDyer commented Jul 8, 2024

dhardy Jul 8, 2024

MichaelOwenDyer Jul 8, 2024

MichaelOwenDyer commented Jul 8, 2024

Add distribution plots to rand_distr documentation #1434

Add distribution plots to rand_distr documentation #1434

Conversation

MichaelOwenDyer commented Apr 9, 2024 • edited Loading

Summary

Motivation

Details

MichaelOwenDyer commented Apr 9, 2024

dhardy commented Apr 10, 2024

Plots

Bernoulli

Binomial

Cauchy

newpavlov commented Apr 10, 2024

dhardy commented Apr 10, 2024

newpavlov commented Apr 10, 2024 • edited Loading

MichaelOwenDyer commented Apr 10, 2024

newpavlov commented Apr 10, 2024 • edited Loading

MichaelOwenDyer commented Apr 10, 2024

dhardy commented Apr 10, 2024

MichaelOwenDyer commented Apr 10, 2024

dhardy commented Apr 10, 2024

MichaelOwenDyer commented Apr 10, 2024

dhardy commented Apr 10, 2024

MichaelOwenDyer commented Apr 10, 2024 • edited Loading

vks commented Apr 26, 2024

dhardy commented Jun 19, 2024

dhardy commented Jun 19, 2024

MichaelOwenDyer commented Jun 19, 2024

MichaelOwenDyer commented Jun 19, 2024

dhardy commented Jun 20, 2024

vks commented Jun 20, 2024

MichaelOwenDyer commented Jun 20, 2024

vks commented Jun 20, 2024 • edited Loading

MichaelOwenDyer commented Jun 21, 2024

dhardy commented Jul 8, 2024

MichaelOwenDyer commented Jul 8, 2024

MichaelOwenDyer commented Jul 8, 2024

dhardy Jul 8, 2024

Choose a reason for hiding this comment

MichaelOwenDyer Jul 8, 2024

Choose a reason for hiding this comment

MichaelOwenDyer commented Jul 8, 2024

Add distribution plots to `rand_distr` documentation #1434

Add distribution plots to `rand_distr` documentation #1434

MichaelOwenDyer commented Apr 9, 2024 •

edited

Loading

newpavlov commented Apr 10, 2024 •

edited

Loading

newpavlov commented Apr 10, 2024 •

edited

Loading

MichaelOwenDyer commented Apr 10, 2024 •

edited

Loading

vks commented Jun 20, 2024 •

edited

Loading