Delays and EventProp

[neworderofjamie]

This PR improves handling of delays and allows them to be learned using EventProp (using new functionality from genn-team/genn#629).

However, incorporating delays means that you need to be careful between trials to empty dendritic delay buffers (as well as outPost) and make sure variables accessed with delays (LambdaI) are also reset correctly. This PR modifies the existing reset logic using the new functionality implemented in genn-team/genn#634, namely:

• The create_reset_custom_update helper now uses VarAccessMode.BROADCAST wherever possible so resets will be applied across all delay slots if necessary (also saves an unnecessary memory read)
• Added new Compiler.add_out_post_zero_custom_update helper to create custom updates that zero outPost and, if necessary denDelay on device.

This PR also includes fixes for several issues which somehow only came to light when trying to train SSC.

• Gradients were being optimised and reduced in 1st batch when they are invalid
• During validation, gradients are still accumulated - this is fine but they weren't being zeroed before the next epoch of training. An additional custom update is now generated to do this.

Also, this PR modifies CompiledTrainingNetwork.train to take a seperate list of validation callbacks, preventing irritating issues around e.g. checkpointing and learning rate schedules which I think Tom, Balazs and everyone else who's tried to use mlGeNN hit!

Fixes #90

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 26.92308% with 114 lines in your changes missing coverage. Please review.

Project coverage is 65.18%. Comparing base (4e39040) to head (60e2293).
Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
ml_genn/ml_genn/compilers/event_prop_compiler.py	19.76%	69 Missing ⚠️
ml_genn/ml_genn/compilers/compiler.py	37.03%	17 Missing ⚠️
ml_genn/ml_genn/compilers/eprop_compiler.py	0.00%	9 Missing ⚠️
ml_genn/ml_genn/callbacks/custom_update.py	50.00%	4 Missing ⚠️
...enn/ml_genn/compilers/compiled_training_network.py	20.00%	4 Missing ⚠️
ml_genn/ml_genn/optimisers/adam.py	33.33%	4 Missing ⚠️
...genn/ml_genn/callbacks/optimiser_param_schedule.py	0.00%	3 Missing ⚠️
ml_genn/ml_genn/compilers/inference_compiler.py	50.00%	2 Missing ⚠️
ml_genn/ml_genn/connection.py	50.00%	1 Missing ⚠️
ml_genn/ml_genn/utils/data.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #104      +/-   ##
==========================================
- Coverage   66.11%   65.18%   -0.93%     
==========================================
  Files         101      101              
  Lines        4123     4220      +97     
==========================================
+ Hits         2726     2751      +25     
- Misses       1397     1469      +72     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tnowotny]

I think I was able to follow most of it and it looks ok. Just to confirm, delays are not reset after an epoch, right? Only the gradients either for weights or for delays. (otherwise the reset would be fatal if learning delays, obviously).