when "create_inverse_triples" is set to false for ConvE, the evaluation becomes extremely slow. #1482
Comments
|
Hi @g-yit , this is unfortunately to be expected since ConvE's interaction function is designed to allow fast 1:n tail scoring while being quite inefficient for head scoring. In the original paper, they use inverse relations, so they only need tail scoring 😉 Background: ConvE uses an interaction function of the form |
import pykeen.models
import pykeen.training
import pykeen.optimizers
import pykeen.evaluation
from pykeen.datasets import FB15k237
from pykeen.losses import SoftplusLoss
from pykeen.regularizers import PowerSumRegularizer
from pykeen.sampling import BernoulliNegativeSampler
from tests.ztest.callback.training_callback import ZTrainingCallback
# Load the FB15k-237 dataset
dataset = FB15k237()
regularizer = PowerSumRegularizer(
weight=0.0005,
p=2.0,
apply_only_once=True,
normalize=False
)
# Set up the loss function
loss = SoftplusLoss(reduction='mean')
# Initialize the ConvKB model with the specified parameters
model = pykeen.models.ConvKB(
embedding_dim=100,
num_filters=50,
hidden_dropout_rate=0.0,
entity_initializer='xavier_uniform',
relation_initializer='xavier_uniform',
triples_factory=dataset.training,
regularizer=regularizer,
loss=loss,
).to("cuda")
# Set up the regularizer
# Set up the optimizer
optimizer = pykeen.optimizers.Adam(
params=model.get_grad_params(),
lr=5e-06
)
# Set up the negative sampler
negative_sampler = BernoulliNegativeSampler(
mapped_triples=dataset.training.mapped_triples
)
# Configure the training loop
training_loop = pykeen.training.SLCWATrainingLoop(
model=model,
triples_factory=dataset.training,
optimizer=optimizer,
negative_sampler=negative_sampler,
negative_sampler_kwargs=dict(num_negs_per_pos=1)
)
# Train the model
training_loop.train(
num_epochs=1,
batch_size=256,
triples_factory=dataset.training,
use_tqdm_batch=False,
)
# Evaluate the model
evaluator = pykeen.evaluation.RankBasedEvaluator(filtered=True)
results = evaluator.evaluate(
model=model,
mapped_triples=dataset.testing.mapped_triples,
additional_filter_triples=[
dataset.training.mapped_triples,
dataset.validation.mapped_triples
]
)
print(results.to_dict())When running the above code for ConKB training, the validation process is also very slow. The graphics card is an NVIDIA RTX 4090 with 24GB of memory. |
|
|
|
Thank you. I used the configuration file from the experiment to build the conkb pipeline. There are two issues:
Did I configure something incorrectly? (You can also try the code mentioned above.) Is there any guidance on conkb training code that can yield the correct results? "metrics": { |
|
|
Did you first train a TransE model and use its weights to initialize the ConvKB ones? That one is easy to miss, and unfortunately not easy to set as default initialization for ConvKB (since it requires training another model first). You can find a config for this training on FB15k237 here. Note that there are some best guesses in there because the paper did not report all hyperparameters for this first run. Once you have the weights, you can use Even with those, we were not able to reproduce the paper's results, cf. https://arxiv.org/pdf/2006.13365, Table 9 in the appendix. |
For each triple it needs to score ~15k entities, i.e., it is running 3.31*15k = 49.65k score evaluations per second. |
|
btw, from you method name |
Describe the bug
In PyKEEN, when "create_inverse_triples" is set to false for ConvE, the evaluation becomes extremely slow.
How to reproduce
Environment
1.11.1-dev
Additional information
No response
Issue Template Checks
The text was updated successfully, but these errors were encountered: