WIP: Adding Domain adaptation

[bruce-edelman]

!!This PR is still a WIP!!

Adding Domain Adaptation following what was done for SIA and ReLEARN from https://www.biorxiv.org/content/10.1101/2023.03.01.529396v1 (their code lives at https://github.com/ziyimo/popgen-dom-adapt)

This requires two major changes to diploshic:

• Forking the network architecture after feature extraction
  • Discriminator fork of model has GRL to encourage the training to do a bad job at discriminating real/fake data
  • use masked loss functions so that each task is only done for data that makes sense
• Adjusting the data generators to enable the inclusion of empirical data (target data) that your simulated data (source data) wants to 'adapt' to
  • this involves setting up a second 'Y' or target values for the discriminator prediction outputs

That should be it for the major implementation changes. The rest of this PR is small changes to the interfacing script that handles the logic of using the original model by default and then switching to the domain adaptive model with the CLI argument --domain-adaptation

Currently by default if you turn on domain adaptation then the code assumes that you have .fvec feature vector files created from your target domain data and stored in your training directory named empirical.fvec

Current steps left undone:

• Construct different simulated data that is 'mis-matched' with current training data to see test the increased performance with domain adaptation if there is a mis-specification of your simulated data and data you want to do predictions on. -- this needs to be simulated data so we have labels to evaluate any changes in performance
• Compare DA model with original in the mis-specification experiment
• Compare predictions on the REAL data from soup to nuts example with original and DA data.

[bruce-edelman]

Just added small fixes to the bugs you found @andrewkern -- one of the bugs was because train_test_split needs all the same length arrays input so this requires the number of your observations in emprical.fvec need to be the same as your training sets.

For the current hack of using the neut.fvec as our fake target domain data I just copied these 2000 data points 5 times to give 10000 obs to match the simulations.

With this change and a few array shape fixes the model begins training with --domain-adaptation on just fine for me now

[bruce-edelman]

This is the copy 5x line that should be removed in the future when user passes in empirical target domain data the same length of their training set simulations

[andrewkern]

running this now! one warning I'm getting is

WARNING:tensorflow:Early stopping conditioned on metric `val_accuracy` which is not available. Available metrics are: loss,predictor_loss,discriminator_loss,predictor_accuracy,discriminator_accuracy

this has to do with the metrics on the early stopping criterion.

[bruce-edelman]

Fixed the callback issue -- have code change from 'val_accuracy' to 'val_predictor_accuracy' for checkpointing and early stopping when using domain adaptation