[tests] fix brittle deviation test for tracer

[majorgreys]

The RateSamplerTest.test_sample_rate_deviation was failing for sample_rate = 0.1. The actual deviation was 0.0216 though the manual threshold is set to 0.02. While we could increase the number of iterations or do more convergence testing, we are increasing the deviation threshold to 0.05 to keep a reasonable upper bound in place while making the test less brittle.

__________________ RateSamplerTest.test_sample_rate_deviation __________________

self = <tests.test_sampler.RateSamplerTest testMethod=test_sample_rate_deviation>

    def test_sample_rate_deviation(self):
        for sample_rate in [0.1, 0.25, 0.5, 1]:
            tracer = get_dummy_tracer()
            writer = tracer.writer
    
            tracer.sampler = RateSampler(sample_rate)
    
            random.seed(1234)
    
            iterations = int(1e4 / sample_rate)
    
            for i in range(iterations):
                span = tracer.trace(i)
                span.finish()
    
            samples = writer.pop()
    
            # We must have at least 1 sample, check that it has its sample rate properly assigned
            assert samples[0].get_metric(SAMPLE_RATE_METRIC_KEY) == sample_rate
    
            # Less than 2% deviation when 'enough' iterations (arbitrary, just check if it converges)
            deviation = abs(len(samples) - (iterations * sample_rate)) / (iterations * sample_rate)
>           assert deviation < 0.02, 'Deviation too high %f with sample_rate %f' % (deviation, sample_rate)
E           AssertionError: Deviation too high 0.021600 with sample_rate 0.100000
E           assert 0.0216 < 0.02

tests/test_sampler.py:37: AssertionError

Since we introduced random.SystemRandom for generating unique identifiers for spans, here we've removed random.seed calls as they do not affect the state of underlying random.SystemRandom.

[jd]

Can you explain in the commit message why thix fix the issue and which issue it fixes?

[majorgreys]

Ok, removing random.seed does not fix this issue:

            # Less than 2% deviation when 'enough' iterations (arbitrary, just check if it converges)
            deviation = abs(len(samples) - (iterations * sample_rate)) / (iterations * sample_rate)
>           assert deviation < 0.02, 'Deviation too high %f with sample_rate %f' % (deviation, sample_rate)
E           AssertionError: Deviation too high 0.020500 with sample_rate 0.100000
E           assert 0.0205 < 0.02

I see three approaches to fixing this moving forward:

1. Increase the threshold, say to 2.5%
2. Increase the number of iterations for lower sample rates since we might not have collected enough samples
3. Test convergence rather than a single rate test

Updated comment