Artifacts and Allegro Trains

[sdesrozis]

❓ Questions/Help/Support

I have my own local trains server. It works perfectly well to log scalars and log history. However, I don’t catch how to upload models on the server. It seems models of examples on demo server are not uploaded.

I use the following code from documentation

trains_logger = TrainsLogger(
                project_name="pytorch-ignite-integration",
                task_name="cnn-mnist"
            )
to_save = {"model": model}
handler = Checkpoint(
                to_save,
                TrainsSaver(),
                n_saved=1,
                score_function=lambda e: 123,
                score_name="acc",
                filename_prefix="best",
                global_step_transform=global_step_from_engine(trainer)
            )
validation_evaluator.add_event_handler(Events.EVENT_COMPLETED, handler)

Models are referenced as local files on computing server... I surely missed something, some help is needed.

I don’t see any task.upload_artifact(...) in code. I thought it was the way to upload. If I use official example of the ˋtrains documentation, the call of ˋtorch.save(...) does not upload model 😔

Thanks in advance!

[sdesrozis]

@bmartinn any help is welcome!

[vfdev-5]

@sdesrozis see this example : https://github.com/pytorch/ignite/blob/master/examples/contrib/cifar10/main.py#L345
or how it is done for Pascal VOC ref example

[sdesrozis]

@vfdev-5 Thank you, I think it’s what I did. I give a feedback tomorrow on that.

[sdesrozis]

Here is my test

import torch
import torch.nn.functional as F
from torch import nn

from ignite.contrib.handlers.trains_logger import *
from ignite.engine import Events, Engine
from ignite.handlers import Checkpoint


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=-1)


if __name__ == '__main__':

    model = Net()
    device = "cpu"

    if torch.cuda.is_available():
        device = "cuda"

    model.to(device)  # Move model before creating optimizer

    trainer = Engine(lambda e, b: 123)

    trains_logger = TrainsLogger(project_name="examples", task_name="artifact_test")

    trains_logger.attach_output_handler(
        trainer,
        event_name=Events.ITERATION_COMPLETED,
        tag="training",
        output_transform=lambda loss: {"batchloss": loss},
    )

    handler = Checkpoint(
        {"model": model},
        TrainsSaver()
    )
    trainer.add_event_handler(Events.EPOCH_COMPLETED, handler)

    trainer.run([0, 1, 2], max_epochs=3)

    trains_logger.close()

Outout curve is ok.

but artifact is not uplaoded. The path is related to host machine.

Above screens are from my local trains server but you can see similar things on allegro demoserver. PyTorch-ignite examples don't have uploaded artifacts but native pytorch examples have.

[sdesrozis]

See here clearml/clearml#171

The good way is to use output_uri in Task.init() with value http://<train-server>:8081.

[sdesrozis]

It works. I close this question.