HealthSystem. Convert Excel files to CSV

[mnjowe]

This PR seeks to convert all excel files in healthsystem resource folder to CSV equivalent.

[tbhallett]

Thanks @mnjowe

This looks good to me, and the checks are passing --- so I'm sure all good.

I'm asking @marghe-molaro to review though, as these files were introduced by her, and it would be great for he to cast her eye over it too.

[marghe-molaro]

Hi @mnjowe, from a quick look it all looks in order, and if the tests are passing it should all be fine!

Only thing I've noticed is that in the priority policy sheets all variables should be integers, but in the new files all those related to the fast-tracking of individuals appear to have a decimal place. Might be safer to turn them back to integers, mainly to ensure that this doesn't confuse modellers interested in adding a new policy (they might get a sense that these can be assigned non-integer value, which would however be rounded up). What do you think? Many thanks!

[mnjowe]

Thanks @marghe-molaro. Not sure how it came to that but I agree with you that it should be integers so as it was in excel file. I will look into it

[mnjowe]

Hi @marghe-molaro . So it turns out the floats are as a result of null values(empty rows) in the excel file columns(see attached). How do you want me to fill these columns?

[marghe-molaro]

Hi @mnjowe, thanks for spotting this. Those values are never considered (see how lowest_priority_considered is loaded in the load_priority_policy fnc in the healthsystem module) so I think you can go ahead and make those "-1"s, for safety. Many thanks!

[mnjowe]

great, thanks!

[tbhallett]

@marghe-molaro -- is then 'Approved' for merging then?

[tbhallett]

This is now showing conflict!

[mnjowe]

Let me look into it

[tbhallett]

Another conflict (sorry), but as soon as this is resolved, let's merge immediately.

[mnjowe]

Resolving it now

[mnjowe]

@tbhallett can you please try running the failing test(tests/test_simulation.py) on your machine? Seems its running just fine on my laptop even after updating and merging master in this branch. I wonder what is making it to fail here?

[tbhallett]

@tbhallett can you please try running the failing test(tests/test_simulation.py) on your machine? Seems its running just fine on my laptop even after updating and merging master in this branch. I wonder what is making it to fail here?

Yes, working on my machine too.

Maybe a network issue with Azure!?!? I'll restart the tests

[tbhallett]

Same thing again! -- very weird. Let's ask @tamuri and @matt-graham, why these tests all pass locally but not on Azure.

(I see it's in test_continuous_and_interrupted_simulations_equal so I wonder if some non-determinism has crept in somehow!?)

[tamuri]

I wonder if some non-determinism has crept in somehow!?

Yes, it appears so. I ran the test 20 times, and it failed twice.

[mnjowe]

Hm. @tbhallett, could this be as a result of the new changes in CSV files? @tamuri did you also try running the master version of this test? I can try running it if you didn't already

[tamuri]

It's failing intermittently on master too. I suggest we continue merging the csv PRs and raise a new issue for this failing test.

[matt-graham]

I think this non-determinism has probably been around for a while as I suspect it's what caused the non-reproducible failure of this same test mentioned in #1507 (comment). I'll see if I can do a git bisect with a longer / more sensitive version of the failing test to narrow down.