Training metrics

[louisfd]

In burn-train, several metrics can be used during training. It would be great to have more!

• Accuracy
• Loss (the one in use)
• CUDA utilization (memory&compute)
• Top-k accuracy
• CPU utilization
• CPU memory usage
• General GPU utilization
• General GPU memory usage
• Precision Recall
• AUC - ROC
• BLEU score
• ROUGE score

[Elazrod56]

Hello, can I be assigned ? I'll deal with it

[louisfd]

Of course! 😄

[Elazrod56]

If I understood correctly, all Rust files that get metrics are located in burn-train/src/metric/

Just to check, is my top_k_acc.rs file in the right place ?

[louisfd]

Yes, right there :)

[Elazrod56]

Just a question, by "CPU Memory usage", do you mean the RAM or something else ?

[nathanielsimard]

Just the RAM, nothing fancy :)

[Elazrod56]

I can't get myself to figure out how non-hardware metrics work, what they mean and how to implement them. Is that okay if I only implement the hardware metrics (CPU use, memory use, GPU use, GPU memory) ? I'll add temperature metrics for the CPU and the GPU to compensate my incapability to understand deep learning-specific metrics.

[nathanielsimard]

Yes no problem, you can submit a PR with only one metric if you want!

[oleksii-shyman]

Is this issue still relevant? I am interested to continue unfinished metrics.

I have the following questions:

1. I have AMD Radeon RX 5500 GPU on my machine. Is it good enough?
2. I am not really familiar with deep learning metrics, but I am willing to understand them. Would experienced dev be able to guide me in case I have questions?

[nathanielsimard]

You can add metrics with any machine. To understand them, you can probably read some articles first like https://neptune.ai/blog/performance-metrics-in-machine-learning-complete-guide and then maybe use wikipedia for each individual metric you want to add.

[duydl]

Hi, I’m just starting with Burn (and Rust) and I’m interested in this issue. I have a few questions I’d like to ask:

• There’s often a significant overlap between metrics and losses in ML. I checked the losses implemented in burn/crates/burn-core/src/nn/loss and noticed that some commonly used loss functions might be missing, at least compared to the variety available in PyTorch's loss functions. Is there an existing issue or plan to track and add these additional losses also?

• Just out of curiosity, how comprehensive is the Burn implementation aiming to be with respect to metrics and losses? Would something as extensive as PyTorch-Metrics (lightning.ai) be too broad for the core library? Or people would prefer to implement their own custom metric/loss.

• In classification tasks, there’s often need for some metrics to support both single-neuron (binary) and multi-neuron (multi-class) setups. It seems that the current metrics (e.g., AccuracyMetric) are for multi-class classification. Would there be interest in expanding this to cover binary classification as well? Additionally, metrics like precision and AUC would require this too.

[nathanielsimard]

@duydl Since Rust offers feature flags, if we may create one metrics-extra when the amount of metrics becomes too large, but for now I wound't worry about it and contribute metrics to Burn :)

Providing more losses could follow the same pattern: more is better and we may hide some behind feature flags when we have too many.

[sebffischer]

The list should be updated as ROC AUC is now implemented.