This work was performed at The Allen Institute of Artificial Intelligence.
This project is constantly being improved. Contributions, comments and suggestions are welcome!
Experiment | Subset | Data |
---|---|---|
Implicit Knowledge of Taxonomic Relations | Training | train, dev |
Implicit Knowledge of Taxonomic Relations | Hypothesis-only | dev, test |
Implicit Knowledge of Taxonomic Relations | ExplicitReasoning | dev, test |
Implicit Knowledge of Taxonomic Relations | ImplicitReasoning | dev, test |
Counting over Implicit Facts | Training | train, dev |
Counting over Implicit Facts | Hypothesis-only | dev, test |
Counting over Implicit Facts | Counting (1, K-1) | dev, test |
Counting over Implicit Facts | Counting K | dev, test |
-
First, clone the repository:
git clone https://github.com/alontalmor/LeapOfThought.git
-
Change your directory to where you cloned the files:
cd LeapOfThought
-
Create a virtual environment with Python 3.6 or above:
virtualenv venv --python=python3.7 (or python3.7 -m venv venv or conda create -n multiqa python=3.7)
-
Activate the virtual environment. You will need to activate the venv environment in each terminal in which you want to use LeapOfThought.
source venv/bin/activate (or source venv/bin/activate.csh or conda activate multiqa)
-
Install the required dependencies:
pip install -r requirements.txt
You can test all artisets using pytest, or using pycharm tests directory (pytest-pycharm added):
pytest tests
The allennlp caching infra is used, so be sure to have enough disk space, and control the cache directory using ALLENNLP_CACHE_ROOT env variable.
This will take create a new artiset python file and test based on the copy_from artiset, output is placed in artiset_module dir.
python LeapOfThought/run.py -c Hypernyms -o create_new_artiset --copy_from ExampleArtiset --artiset_module soft_reasoning
From please use:
python LeapOfThought/run.py --help
To recreate the hypernyms training data
python LeapOfThought/run.py -c Hypernyms -o build_artificial_dataset -out s3://aigame/data/taxonomic_reasonings/taxonomic_reasonings.jsonl.gz -v trainin_mix
For training the counting experiment (hypernym experiment has similar hyperparams):
python -m allennlp train LeapOfThought/allennlp_models/config/transformer_binary_qa.jsonnet -s YOUR_OUTPUT_DIR -o "{'data_loader': {'batch_sampler': {'batch_size': 4}}, 'dataset_reader': {'pretrained_model': 'roberta-large', 'sample': 35000}, 'model': {'pretrained_model': 'roberta-large'}, 'random_seed': 2, 'train_data_path': 'https://aigame.s3-us-west-2.amazonaws.com/data/counting/counting_training_mix_no_total_train.jsonl.gz', 'trainer': {'checkpointer': {'num_serialized_models_to_keep': 1}, 'cuda_device': GPUNUM, 'learning_rate_scheduler': {'cut_frac': 0.1, 'num_epochs': 4, 'num_steps_per_epoch': 729}, 'num_epochs': 4, 'num_gradient_accumulation_steps': 12, 'optimizer': {'lr': 1e-05, 'weight_decay': 0.1}}, 'validation_data_path': 'https://aigame.s3-us-west-2.amazonaws.com/data/counting/counting_training_mix_no_total_dev.jsonl.gz', 'validation_dataset_reader': {'pretrained_model': 'roberta-large'}}" --include-package LeapOfThought.allennlp_models
Example for evaluating the Implicit Knowledge of Taxonomic Relations, ExplicitReasoning test set:
python -m allennlp evaluate MY_TRAINED_MODEL_PATH.tar.gz https://aigame.s3-us-west-2.amazonaws.com/data/hypernyms/hypernyms_explicit_only_short_neg_hypernym_rule_test.jsonl.gz --output-file MY_OUTPUT_FILE_results.json -o "{'trainer': {'cuda_device': GPUNUM}, 'validation_data_loader': {'batch_sampler': {'batch_size': 20, 'type': 'bucket'}}}" --cuda-device -1 --include-package LeapOfThought.allennlp_models
A caching infra is used, so make sure to have enough disk space, and control the cache directory using LEAPOFTHOUGHT_CACHE_ROOT
env variable.
see teachyourai/common/file_utils.py