Skip to content

uma-pi1/GraSH

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

GraSH: Successive Halving for Knowledge Graphs

This is the code and configuration accompanying the paper "Start Small, Think Big: On Hyperparameter Optimization for Large-Scale Knowledge Graph Embeddings" presented at ECML-PKDD 2022. The code extends the knowledge graph embedding library for distributed training Dist-KGE. For documentation on Dist-KGE refer to the Dist-KGE repository. We provide the hyperparameter settings for the searches and finally selected trials in /examples/experiments/.

UPDATE: GraSH was recently merged into our main library LibKGE. All configs from this repository, except the ones for Freebase that require distributed training, can be executed in LibKGE. Please use LibKGE for your own experiments with GraSH.

Table of contents

  1. Quick start
  2. Configuration of GraSH Search
  3. Run a GraSH hyperparameter search
  4. Results and Configurations
  5. How to cite

Quick start

Setup

# retrieve and install project in development mode
git clone https://github.com/uma-pi1/grash.git
cd grash
pip install -e .

# download and preprocess datasets
cd data
sh download_all.sh
cd ..

Training

# train an example model on a toy dataset (you can omit '--job.device cpu' when you have a gpu)
python -m kge start examples/toy-complex-train.yaml --job.device cpu

This example will train on a toy dataset in a sequential setup on CPU

GraSH Hyperparameter Search

# perform a search with GraSH on a toy dataset (you can omit '--job.device cpu' when you have a gpu)
python -m kge start examples/toy-complex-search-grash.yaml --job.device cpu

This example will perform a small GraSH search with 16 trials on a toy dataset in a sequential setup on CPU

Configuration of GraSH Search

The most important configuration options for a hyperparameter search with GraSH are:

dataset:
  name: yago3-10
grash_search:
  eta: 4
  num_trials: 64
  search_budget: 3
  variant: combined
  parameters: # define your search space here
job:
  type: search
model: complex
train:
  max_epochs: 400
  • eta defines the reduction factor during the search. Per round the number of remaining trials is reduced to 1/eta
  • search_budget is defined in "number of full training runs". The default choice search_budget=3, for example, corresponds to an overall search cost of three full training runs.
  • variant controls which reduction technique to use (only epoch, only graph, or combined)

Run a GraSH hyperparameter search

Run the default search on yago3-10 with the following command:

python -m kge start examples/experiments/search_configs/yago3-10/search-complex-yago-combined.yaml

The k-core subgraphs will automatically be generated and saved to data/yago3-10/subsets/k-core/. By default, each experiment will create a new folder in local/experiments/<timestamp>-<config-name> where the results can be found.

Results and Configurations

All results were obtained with the GraSH default settings (num_trials=64, eta=4, search_budget=3, variant=combined)

Yago3-10

Model Variant MRR Hits@1 Hits@10 Hits@100 config
ComplEx Epoch 0.536 0.460 0.672 0.601 config
ComplEx Graph 0.463 0.375 0.634 0.800 config
ComplEx Combined 0.528 0.455 0.660 0.772 config
RotatE Epoch 0.432 0.337 0.619 0.768 config
RotatE Graph 0.432 0.337 0.619 0.768 config
RotatE Combined 0.434 0.342 0.607 0.742 config
TransE Epoch 0.499 0.406 0.661 0.794 config
TransE Graph 0.422 0.311 0.628 0.802 config
TransE Combined 0.499 0.406 0.661 0.794 config

Wikidata5M

Model Variant MRR Hits@1 Hits@10 Hits@100 config
ComplEx Epoch 0.300 0.247 0.390 0.506 config
ComplEx Graph 0.300 0.247 0.390 0.506 config
ComplEx Combined 0.300 0.247 0.390 0.506 config
RotatE Epoch 0.241 0.187 0.331 0.438 config
RotatE Graph 0.232 0.169 0.326 0.432 config
RotatE Combined 0.241 0.187 0.331 0.438 config
TransE Epoch 0.263 0.210 0.358 0.483 config
TransE Graph 0.263 0.210 0.358 0.483 config
TransE Combined 0.268 0.213 0.363 0.480 config

Freebase

Model Variant MRR Hits@1 Hits@10 Hits@100 config
ComplEx Epoch 0.572 0.486 0.714 0.762 config
ComplEx Graph 0.594 0.511 0.726 0.767 config
ComplEx Combined 0.594 0.511 0.726 0.767 config
RotatE Epoch 0.561 0.522 0.625 0.679 config
RotatE Graph 0.613 0.578 0.669 0.719 config
RotatE Combined 0.613 0.578 0.669 0.719 config
TransE Epoch 0.261 0.078 0.518 0.636 config
TransE Graph 0.553 0.520 0.614 0.682 config
TransE Combined 0.553 0.520 0.614 0.682 config

How to cite

@inproceedings{kochsiek2022start,
  title={Start Small, Think Big: On Hyperparameter Optimization for Large-Scale Knowledge Graph Embeddings},
  author={Kochsiek, Adrian and Niesel, Fritz and Gemulla, Rainer},
  booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
  year={2022}
}

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published