Add EntropySource wrapper #235

pitdicker · 2018-01-14T19:49:21Z

This is just an idea I am trying. The PR is based on top of #233, only the last 3 commits are new.

I have added an EntropySource wrapper that uses OsRng, and if it errors falls back to JitterRng. I hope this will help in three situations:

The fallback mechanism in NewSeeded can also be used by user code.
It is no longer necessary to have a Reseeder trait, providing an RNG for reseeding is enough.
We can hopefully add a feature flag to let EntropySource use an external RNG. This will make it and NewSeeded available for no_std uses.

dhardy

This actually makes NewSeeded redundant.

dhardy · 2018-01-14T20:15:20Z

src/lib.rs

+}
+
+impl EntropySource {
+    pub fn new() -> Result<Self, Error> {


I think it's worth making this infallible and delaying errors so that X::from_rng(EntropySource::new()) works. Since performance isn't very important here we could potentially just try all sources each call, though that's not optimal.

JitterRng has a pretty high set-up cost, because it runs the timer for every initialization.

Do you mean to add something like a third variant to EntropySourceInner that none are available, with the Error?

X::from_rng(EntropySource::new()) works really nice! It will take me until Wednesday or Thursday before I can update this PR.

dhardy · 2018-01-14T20:17:02Z

src/lib.rs

+    }
+
+    fn fill_bytes(&mut self, dest: &mut [u8]) {
+        self.try_fill_bytes(dest).unwrap();


unwrap has to go — but sure, this is a quick implementation

I was even thinking of just using unimplemented!() here. The situations where both OsRng and JitterRng fail should be extremely rare, so I didn't care much about the unwrap. What do suggest to replace it with?

Okay for now; revisit in #249

dhardy · 2018-01-14T20:19:52Z

src/lib.rs

+                    let mut rng2_result = JitterRng::new();
+                    if let Ok(mut rng2) = rng2_result {
+                        result = rng2.try_fill_bytes(dest); // Can't fail
+                        switch_rng = Some(EntropySourceInner::Jitter(rng2));


Probably simpler to recurse here (i.e. call new_self.try_fill_bytes()).

It took some thinking to get the logic right and the borrow checker happy. But I'll think about it!

dhardy · 2018-01-14T20:24:38Z

Um, so did you come up with this idea independently or see my comment before I finished the review? Either way I like it, plus we can leave out NewSeeded for now (unless you feel strongly about including it).

It might be nice to have a global generator function like entropy() instead of EntropySource::new(); I'm not sure. May be better to name this EntropyRng?

pitdicker · 2018-01-14T20:39:02Z

Um, so did you come up with this idea independently or see my comment before I finished the review? Either way I like it, plus we can leave out NewSeeded for now (unless you feel strongly about including it).

I was surprised by you comment. We thought of the same thing at the same time 😄.
Did I write somewhere that NewSeeded was the thing I liked most about rand when looking at it for the first time (somewhere in august)? That {Rng}::new() just does the right thing?

pitdicker · 2018-01-14T20:57:58Z

I am partial about the name. Do you want to pick between EntropySource and EntropyRng?

I feel that adding an entropy() function will make this to 'prominent'. Except for initializing RNG's and a handful of special cases like generating keys entropy should not be needed much. It does not do any real harm except for performance, but I think it is better to steer users towards normal PRNG's.

pitdicker · 2018-01-14T21:03:34Z

(nothing changed yet, just a rebase to hopefully get the CI happy)

dhardy · 2018-01-15T06:11:52Z

To quote the OED:

partial to: Having a liking for.
impartial: Treating all rivals or disputants equally.

More on topic, I prefer EntropyRng over EntropySource because so far all our Rng impls are *Something*Rng.

You are right that XRng::new() is more user-friendly than XRng::from_rng(entropy()), though it does have two issues: requirement for use rand::NewSeeded; and relatively little flexibility in the case that a user wants to (or needs to) use their own entropy source. But yes, probably NewSeeded is the best option for most users, so I guess we should go with that.

But having this as well, and making NewSeeded a simple wrapper around EntropyRng? I think it's a good idea since it allows users to use external generators directly with the same fallback logic.

pitdicker · 2018-01-15T19:23:05Z

Ah, thank you. I was impartial. Now I am starting to get partial to EntropySource 😄. Bikeshedding!

IsaacRng, ChaChaRng, XorshiftRng and Hc128Rng all follow the pattern {algorithm name} Random number Generator.
OsRng and JitterRng do something similar with {source} Random number Generator.
And StdRng and ThreadRngalso make sense to me as the 'standard RNG' and 'thread-local RNG'.

EntropyRng 'Entropy Random Number Generator' does not really sound like a thing... Something like FreshRng (not a serious suggestion!) would follow the pattern of including the source in the name. EntropySource breaks with it all and just names the intended use.

You are right that XRng::new() is more user-friendly than XRng::from_rng(entropy()), though it does have two issues: requirement for use rand::NewSeeded; and relatively little flexibility in the case that a user wants to (or needs to) use their own entropy source.

Yes, I don't like the extra import. But there seems to be no way around it? Without the trait it is necessary to import entropy/EntropySource/EntropyRng.

What did you think of the idea to add a feature flag (in the future) that allows crates to provide a different source for EntropySource at link time, so we have better no_std support?

dhardy · 2018-01-15T19:33:54Z

Um... I think the extra import could be avoided if new was part of the Rng trait and the entropy source was in the same crate. But it would still be necessary to import Rng, which isn't always necessary otherwise (though with the current plan, users must import rand::{NewSeeded, Sample} — I wonder if we should make that NewRng and SampleRng).

Anyway, this "entropy source" will be a full Rng; it just may be slow (like JitterRng), so I don't see why it can't be EntropyRng?

I don't think a feature flag can do that; at best it could enable support for setting another source at run time.

pitdicker · 2018-01-15T20:22:15Z

I also though of renaming NewSeeded to NewRng. Shall we just do that?

If we make new part of Rng or SeedableRng it will be difficult to add a default implementation that uses EntropyRng when we also split the traits off into rand-core. Also RNG's would be able to implement new, do we want that?

I just thought of an issue: ReseedableRng has it's own new function. If it also implements the SeedableRng trait, that would lead to two conflicting new methods because it is also automatically provided by NewRng?

I don't think a feature flag can do that; at best it could enable support for setting another source at run time.

I was thinking of something like this (but have to learn a lot about FFI and linking first):

#[cfg(feature = "extern_entropy_source")]
extern {
    fn extern_fill_bytes(dest: *mut u8, length: usize) -> u32;
}

#[cfg(feature = "extern_entropy_source")]
impl Rng for EntropySource {
    fn try_fill_bytes(&mut self, dest: &mut [u8]) -> Result<(), Error> {
        /* call extern_fill_bytes */
    }
}

Linking in a crate that provides a different source would have to be done by the build script of the final crate.

dhardy · 2018-01-15T22:58:12Z

I was playing the devil's advocate; I didn't mean putting new in Rng would be a good idea.

You're right, linking like that should be possible, but it's not very "Rustic". It relies on extern_fill_bytes being defined exactly once and on mutable global state (probably, for init), and it potentially means that any crate in the build chain could override the default entropy source. The entropy source isn't designed to be fast, so I would prefer the option to append extra external RNGs as fallbacks, and an optional feature not to add OsRng and JitterRng (with those available by default).

pitdicker · 2018-01-19T16:15:55Z

(I have been busy writing this comment for four days...)

I was playing the devil's advocate; I didn't mean putting new in Rng would be a good idea.

One of the nicest ways to explore options 😄.

What do you think of SourceRng as a name?

You're right, linking like that should be possible, but it's not very "Rustic". It relies on extern_fill_bytes being defined exactly once and on mutable global state (probably, for init)

Would this be more "Rustic"? Moving EntropyRng to a separate crate, so that it can be replaced in the Cargo.toml of the final crate? That would need the rand-core split, and OsRng and JitterRng to be split out of rand to avoid a cyclic dependency.

it potentially means that any crate in the build chain could override the default entropy source. The entropy source isn't designed to be fast, so I would prefer the option to append extra external RNGs as fallbacks, and an optional feature not to add OsRng and JitterRng (with those available by default).

Good point. We don't want an application that relies on a source like OsRng to have it silently overridden to some weak source, just because a dependency of a dependency thought it was good idea to do so. Crates with a [replace] or a [patch] section can't be uploaded to crates.io, and for workspaces only the root crates can set the override. So if we go that route it is not a problem. With linking I would have to think some more about it.

The second part of the puzzle I had in mind was to make ThreadRng a wrapper around EntropyRng if the std feature is not available. Hopefully together this provides solution for WebAssembly, no_std and embedded that does not require changes to rand for every situation.

Something like this had been proposed/tried before in #109, #133, #78. It is not optimal, no use of ThreadRng with no_std is.

As argued in #109 (comment) this could help make crates to be available for no_std that have no way for the user to configure the RNG. For the randomness sources we can then think in the direction of JavaScript's window.crypto.getRandomValues, the difference between two hardware timers, RDRAND, or other device-specific randomness hardware.

dhardy · 2018-01-20T11:36:29Z

If we go in that direction —

OsRng is inherently useless when its not available (or for WASM where it currently always fails), but having it always available (but on unsupported platforms always failing) would simplify no_std support.
Moving OsRng to a separate rand-os crate would as you suggested make it easy to add an alternative source for specific platforms.
The JitterRng code should be available when OsRng isn't and the built-in jitter time sources are also unavailable. Since it might be used to [replace] a rand-os crate, this implies that JitterRng should not be in rand or rand-os, hence we need a rand-jitter crate.
We still want EntropyRng for situations where OsRng should be available but fails at run-time; since this depends on both OsRng and JitterRng it can't be in either of those crates, so may as well be in rand.
Since this allows users to replace either source of EntropyRng, more configuration isn't needed (we could have a third source, doing nothing by default, allowing user replacement — but I see little use).

The above is quite a big redesign, but on first impressions it sounds good.

SourceRng? No, I don't like the name. ExternalRng possibly, but it sounds more like a trait name.

I was experimenting with making EntropyRng always available regardless of OsRng and JitterRng and having some optional third (user-set) source, but the above sounds better.

pitdicker · 2018-01-21T20:28:18Z

I have rebased on top of #233 again (basically recreated the branch...). The last commit renames EntropySource to EntropyRng. I also made new infallible.

The logic to select an entropy source now is something like this:

if None
    if OsRng.new is ok
        if Osrng.fill is ok
            save OsRng
    else if JitterRng.new is ok
        fill with JitterRng
        save JitterRng
    return OsRng error
if OsRng
    fill with OsRng
    if OsRng fill is err
        if JitterRng is ok
            fill with JitterRng
            save JitterRng
    return OsRng error
if JitterRng
    if counter > 8
        if OsRng.new is ok
            if Osrng.fill is ok
                save OsRng
        else
            fill with JitterRng
    else
        fill with JitterRng

Maybe I just made things to difficult, but couldn't quickly see a better way. Will you have another look?

I would like to keep future directions for EntropyRng in the back in mind, but not explore it more at the moment.

dhardy · 2018-01-22T09:58:24Z

The logic in the comment above sounds reasonable to me.

I didn't look at the commits themselves; the order seems to be mixed up with commits from #233? Can you do git rebase -i seedable_rng?

pitdicker · 2018-01-22T11:33:29Z

I am not sure what is happening with the order of the commits, locally and on github (https://github.com/pitdicker/rand/commits/entropy_source) they aren't mixed.

I did not change the commits you already reviewed, only added one extra.

dhardy · 2018-01-22T11:41:38Z

That's odd. Anyway, your new approach looks fine, although it doesn't check the error kind.

If OsRng reports ErrorKind::Unavailable, should we ever retry it?

If OsRng reports NotReady or Transient, should we report that or immediately switch to JitterRng? Your approach (switch but try OsRng again later) is probably the best compromise, although I'm less sure about the counter (we might simply retry OsRng each time?).

pitdicker · 2018-01-22T12:15:00Z

Good point about checking the error kind, I only thought about it partially.
If the error from OsRng is Transient it is better not to switch but to retry.
With Unavailable I was thinking about long running (server) processes. We don't want to have it unavailable at some point (maybe because lack of file descriptors), and remain unavailable for the rest of the year.
NotReady may take tens of seconds or even minutes, using JitterRng in the interim seems like a good idea.

I am also not sure using a counter is best. The overhead of the extra system call is only 2%. Especially when doing larger reads with try_fill_bytes it makes sense to just check each time. On the other hand OsRng is not going to quickly recover from whatever made it fail, maybe even never. Checking each time just feels strange to me.

I just realized JitterRng generates u64s natively. When using next_u32 in EntropyRng where it is implemented with impls::next_u32_via_fill will leave 50% of the performance on the table. I think it makes sense to optimize the fill_bytes implementation of JitterRng a bit for small slices, not to make EntropyRng more complex.

I will add a description of the fall-back strategy to the documentation of EntropyRng.

dhardy · 2018-01-22T12:28:29Z

Performance isn't critical here, so using JitterRng::next_u32 directly is probably not worth it. Also if the system call overhead is only 2% then IMO we should retry often when OsRng was NotReady.

Unavailable was supposed to mean permanently unavailable, but it's difficult to correctly differentiate in all cases, so retrying occasionally does make sense. Use a count-down counter to allow different delays?

pitdicker · 2018-01-22T19:09:58Z

Use a count-down counter to allow different delays?

Yes, that could work. And every time we retry, we get the reason for failing again from the OS, so we can reset the counter. Something like this?

            EntropySource::Jitter(ref mut rng) => {
                if self.counter == 0 {
                    match try_os_new(dest) {
                        Ok(os_rng) => {
                            switch_rng = Some(EntropySource::Os(os_rng));
                        }
                        Err(e) => {
                            if e.kind() == ErrorKind::Unavailable {
                                self.counter = 8; // Wait 8 calls before retry
                            } else {
                                self.counter = 0; // Retry on next call
                            }
                            return rng.try_fill_bytes(dest); // use JitterRng
                        }
                    }
                } else {
                    self.counter -= 1;
                    return rng.try_fill_bytes(dest); // use JitterRng
                }
            }

dhardy · 2018-01-22T19:47:07Z

Yes. Although if we only wait 8 calls I wonder if it's worth using a counter at all; I was thinking a much higher count, but there's probably no reason to do so. I guess given how slow JitterRng is a system-call is relatively fast.

pitdicker · 2018-01-22T20:13:46Z

I don't think using a counter or not matters much.
Also setting the counter to a small or a large value doesn't matter much.

If we set the counter to a high value, and EntropyRng is used rarely, there can be a long time between rechecks. But because it is used rarely this is not performance critical, so no problem.

If we set the counter to a low value, and EntropyRng is used often, there will be too many checks because the OS didn't have time to sort out itself. But also not a problem.

I'll leave it to you: keep things as they are now, remove the counter, or increase it?

dhardy · 2018-01-23T11:14:27Z

Good argument. So I guess I'll let you decide, while I take another look at #233.

dhardy · 2018-01-23T11:32:22Z

Looks like this needs a rebase

pitdicker · 2018-01-24T06:25:26Z

Rebased and removed the counter

dhardy · 2018-01-24T11:24:09Z

Okay, I made my own commit to replace usage of unwrap and improve error messages, from:

---- test::test_entropy stdout ----
        thread 'test::test_entropy' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Unavailable, msg: "OsRng is disabled", cause: None }', /checkout/src/libcore/result.rs:906:4

---- test::test_thread_rng stdout ----
        thread 'test::test_thread_rng' panicked at 'could not initialize thread_rng: Error { kind: Unavailable, msg: "OsRng is disabled", cause: None }', /checkout/src/libcore/result.rs:906:4

---- test::test_random stdout ----
        thread 'test::test_random' panicked at 'could not initialize thread_rng: Error { kind: Unavailable, msg: "OsRng is disabled", cause: None }', /checkout/src/libcore/result.rs:906:4

to:

---- test::test_entropy stdout ----
        thread 'test::test_entropy' panicked at 'all entropy sources failed; first error: RNG error [permanent failure or unavailable]: OsRng is disabled', src/lib.rs:1035:12

---- test::test_random stdout ----
        thread 'test::test_random' panicked at 'could not initialize thread_rng: RNG error [permanent failure or unavailable]: OsRng is disabled', src/lib.rs:944:34

---- test::test_thread_rng stdout ----
        thread 'test::test_thread_rng' panicked at 'could not initialize thread_rng: RNG error [permanent failure or unavailable]: OsRng is disabled', src/lib.rs:944:34

With that change I'm happy to merge this (although I did notice thread_rng() initialisation seemed to be slow when forced to use JitterRng).

dhardy · 2018-01-24T11:40:41Z

Thinking about it some more, the above messages are still not ideal: we never show errors from JitterRng. If we use logging to do this, we need some way to log each error type at least once without spamming the log every time the EntropyRng is used (i.e. if OsRng is tried and fails every time).

I keep wondering whether we want a single shared state for these RNGs (i.e. accessed via a function) or not; so far it doesn't appear necessary for OsRng (which is for many platforms stateless anyway), but JitterRng has a lot of state, and if we want limited logging in EntropyRng we may need state for that too.

dhardy · 2018-01-27T08:30:54Z

I think you need something like a static Option<Mutex<Box<[u8; MEMORY_SIZE]>>> if you want to dynamically allocate a shared box. Use a std::sync::Once to create the mutex and memory on first use (see my PR to limit OsRng to a single file handle, which is similar but has a second Option).

I guess this brings us back to this discussion too.

pitdicker · 2018-01-31T11:15:07Z

I would like to make the changes to JitterRng in a separate PR. Should it hold up this one?

dhardy · 2018-01-31T13:50:40Z

No, it shouldn't hold this up. Sorry, I'll try to get this merged.

dhardy · 2018-01-31T18:59:33Z

I tweaked this and merged with master, but it needs a bit more work really. Feel free to rebase, or I might try reworking this a bit more.

I'm not sure about the error handling; e.g. if reseeding failed it might be permissible to continue without reseeding?

pitdicker · 2018-01-31T20:13:14Z

Thank you. I wanted to say no need to hurry, it is not as if I was able to do much last week.

I tweaked this and merged with master, but it needs a bit more work really. Feel free to rebase, or I might try reworking this a bit more.

Is there a way to see what changes you made?
Don't put too much effort in the reseeding code, I have ported the error handling changes to it from your branch, and was just waiting with the PR until this was merged.

pitdicker · 2018-02-01T07:27:54Z

I have to say I am again not happy with this messy merging. Not all changes from 2278d44 and ee815de were merged. I will make a new PR this evening.

dhardy · 2018-02-01T11:58:17Z

Sorry, I've been using kdiff3 to do the merges, and apparently trusting it too much.

pitdicker · 2018-02-01T12:03:57Z

I found a nice way to reduce the size of JitterRng by the way. There is no reason the memory array has to be preserved across runs. Also prev_time is initialized again for every round of gen_entropy. And the two delta's for the stuck test are questionable to preserve across runs. So we can just store all these values on the stack in gen_entropy, instead of making them part of the type of JitterRng.

dhardy · 2018-02-01T14:44:25Z

Why did you close — are you going to start fresh with a new PR?

pitdicker · 2018-02-01T14:58:30Z

Yes, I will make a fresh PR. I did not realize re-using this one was even an option...

dhardy · 2018-02-01T15:08:44Z

You can force-push and re-open if you like — depends whether you want to keep these comments in the same thread really.

pitdicker · 2018-02-01T17:36:39Z

Apologies, I understood things wrong and blamed you and merging. I rebased and reopened this PR. Also added an extra commit with logging for EntropyRng similar to what NewRng had.

And it turned out reopening a PR after a force-push is difficult.

dhardy · 2018-02-05T11:10:48Z

I'm still not quite sure we've got this right, though re-reading the comment thread the design has evolved significantly since we started!

The OsRng object now requires a whopping 1 bit on Linux, and has zero size on all other platforms. JitterRng by contrast is still significantly larger, after merging #251. This isn't really an issue but what is significant is that both generators store all non-trivial configuration in static data guarded by atomics, and other memory is only used as a cache and for working memory. EntropyRng however stores the decision of which RNG to use locally.

Since the current code tries to use OsRng every time I guess this doesn't matter, though if in the future we changed this it might; I suppose this is only a concern if OsRng failure is slow to detect on any platform (relative to JitterRng performance this is unlikely; if we were to use e.g. RDRAND as a fallback it might be somewhat significant). Summary: theoretical concern only, no change needed.

The logging doesn't seem quite right; currently error details should be printed via Debug but this uses Display in a few places; if we merge #249 the unwrap and expect need to be changed to use Display; also the messages may be a bit long. Further, using info level to report switching back to OsRng only if a warn message was previously reported seems strange, but isn't a big deal.

Since the above issues are very minor and we've been waiting for this a while, I'll go ahead and merge, and leave #249 and #251 to follow up.

Add EntropySource wrapper

dhardy reviewed Jan 14, 2018

View reviewed changes

pitdicker force-pushed the entropy_source branch from fd7c131 to e32b4c9 Compare January 14, 2018 21:01

pitdicker mentioned this pull request Jan 20, 2018

Port new SeedableRng trait #233

Merged

pitdicker force-pushed the entropy_source branch from e32b4c9 to 7a07c36 Compare January 21, 2018 20:18

pitdicker force-pushed the entropy_source branch from 41fc616 to b6339f5 Compare January 24, 2018 06:25

pitdicker closed this Feb 1, 2018

pitdicker added 7 commits February 1, 2018 18:07

Add EntropySource

60ed6b6

Use EntropySource in NewSeeded

200d8b9

Remove Reseeder trait

730b853

Rename to EntropyRng and make new infallible

8008f7a

Use fill_bytes_via_u32 in JitterRng

be2b886

Remove retry counter

88c4a53

Add logging to EntropyRng

69eee53

pitdicker reopened this Feb 1, 2018

pitdicker force-pushed the entropy_source branch from b6339f5 to 69eee53 Compare February 1, 2018 17:33

Silence unused variable warning

8c500a4

pitdicker mentioned this pull request Feb 2, 2018

Port reseeding #252

Merged

dhardy merged commit fc187b3 into rust-random:master Feb 5, 2018

pitdicker deleted the entropy_source branch March 1, 2018 19:37

dhardy mentioned this pull request Mar 11, 2018

Fallback source of entropy dhardy/rand#22

Closed

pitdicker mentioned this pull request Mar 24, 2018

Implement CryptoRng for EntropyRng and StdRng #323

Merged

pitdicker pushed a commit that referenced this pull request Apr 4, 2018

Merge pull request #235 from pitdicker/entropy_source

4120882

Add EntropySource wrapper

Add EntropySource wrapper #235

Add EntropySource wrapper #235

Conversation

pitdicker commented Jan 14, 2018

dhardy left a comment

Choose a reason for hiding this comment

dhardy Jan 14, 2018

Choose a reason for hiding this comment

pitdicker Jan 14, 2018

Choose a reason for hiding this comment

pitdicker Jan 15, 2018

Choose a reason for hiding this comment

dhardy Jan 14, 2018

Choose a reason for hiding this comment

pitdicker Jan 14, 2018

Choose a reason for hiding this comment

dhardy Feb 4, 2018

Choose a reason for hiding this comment

dhardy Jan 14, 2018

Choose a reason for hiding this comment

pitdicker Jan 14, 2018 • edited Loading

Choose a reason for hiding this comment

dhardy commented Jan 14, 2018

pitdicker commented Jan 14, 2018

pitdicker commented Jan 14, 2018

pitdicker commented Jan 14, 2018

dhardy commented Jan 15, 2018

pitdicker commented Jan 15, 2018

dhardy commented Jan 15, 2018

pitdicker commented Jan 15, 2018

dhardy commented Jan 15, 2018

pitdicker commented Jan 19, 2018

dhardy commented Jan 20, 2018 • edited Loading

pitdicker commented Jan 21, 2018

dhardy commented Jan 22, 2018

pitdicker commented Jan 22, 2018

dhardy commented Jan 22, 2018

pitdicker commented Jan 22, 2018

dhardy commented Jan 22, 2018

pitdicker commented Jan 22, 2018 • edited Loading

dhardy commented Jan 22, 2018 • edited Loading

pitdicker commented Jan 22, 2018

dhardy commented Jan 23, 2018

dhardy commented Jan 23, 2018

pitdicker commented Jan 24, 2018

dhardy commented Jan 24, 2018

dhardy commented Jan 24, 2018

dhardy commented Jan 27, 2018

pitdicker commented Jan 31, 2018

dhardy commented Jan 31, 2018

dhardy commented Jan 31, 2018

pitdicker commented Jan 31, 2018

pitdicker commented Feb 1, 2018

dhardy commented Feb 1, 2018

pitdicker commented Feb 1, 2018

dhardy commented Feb 1, 2018

pitdicker commented Feb 1, 2018

dhardy commented Feb 1, 2018

pitdicker commented Feb 1, 2018 • edited Loading

dhardy commented Feb 5, 2018

pitdicker Jan 14, 2018 •

edited

Loading

dhardy commented Jan 20, 2018 •

edited

Loading

pitdicker commented Jan 22, 2018 •

edited

Loading

dhardy commented Jan 22, 2018 •

edited

Loading

pitdicker commented Feb 1, 2018 •

edited

Loading