Skip to content
This repository has been archived by the owner on Nov 22, 2022. It is now read-only.

Segmentation fault when using PyText 0.3.3 on Google Colab #1415

Open
orestxherija opened this issue Aug 3, 2020 · 0 comments
Open

Segmentation fault when using PyText 0.3.3 on Google Colab #1415

orestxherija opened this issue Aug 3, 2020 · 0 comments

Comments

@orestxherija
Copy link

Steps to reproduce

  1. Open a new Google Colab notebook and set the runtime type to GPU
  2. Install latest version of PyText via !pip install --quiet git+https://github.com/facebookresearch/pytext
  3. Import PyText with import pytext
  4. Clone PyText repository via !git clone https://github.com/facebookresearch/pytext.git
  5. Move pytext/tests/ directory to Google Colab's home directory /content/
  6. Train demo CNN model via !pytext train < pytext/demo/configs/docnn.json

Observed Results

Below I am including the entire log.

Install apex from https://github.com/NVIDIA/apex/.
No config file specified, reading from stdin
Current Version: 8
WARNING - Applying old config adapter for version=8. Please consider migrating your old configs to the latest version.
Current Version: 9
WARNING - Applying old config adapter for version=9. Please consider migrating your old configs to the latest version.
Current Version: 10
WARNING - Applying old config adapter for version=10. Please consider migrating your old configs to the latest version.
Current Version: 11
WARNING - Applying old config adapter for version=11. Please consider migrating your old configs to the latest version.
Current Version: 12
WARNING - Applying old config adapter for version=12. Please consider migrating your old configs to the latest version.
Current Version: 13
WARNING - Applying old config adapter for version=13. Please consider migrating your old configs to the latest version.
Current Version: 14
WARNING - Applying old config adapter for version=14. Please consider migrating your old configs to the latest version.
Current Version: 15
WARNING - Applying old config adapter for version=15. Please consider migrating your old configs to the latest version.
Current Version: 16
WARNING - Applying old config adapter for version=16. Please consider migrating your old configs to the latest version.
Current Version: 17
WARNING - Applying old config adapter for version=17. Please consider migrating your old configs to the latest version.
Current Version: 18
WARNING - Applying old config adapter for version=18. Please consider migrating your old configs to the latest version.
Current Version: 19
WARNING - Applying old config adapter for version=19. Please consider migrating your old configs to the latest version.

===Starting training...
2020-08-03 19:39:44.136867: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1

Parameters: PyTextConfig:
    auto_resume_from_snapshot: False
    debug_path: /tmp/model.debug
    distributed_world_size: 1
    export_caffe2_path: /tmp/model.caffe2.predictor
    export_onnx_path: /tmp/model.onnx
    export_torchscript_path: /tmp/new_docnn.pt1
    gpu_streams_for_distributed_training: 1
    include_dirs: None
    load_snapshot_path: 
    modules_save_dir: 
    random_seed: None
    report_eval_results: False
    report_test_results: True
    save_all_checkpoints: False
    save_module_checkpoints: False
    save_snapshot_path: /tmp/model.pt
    task: DocumentClassificationTask.Config:
        data: Data.Config:
            batcher: PoolingBatcher.Config:
                eval_batch_size: 16
                num_shuffled_pools: 1
                pool_num_batches: 1000
                test_batch_size: 16
                train_batch_size: 16
            in_memory: True
            sort_key: None
            source: TSVDataSource.Config:
                column_mapping: {}
                delimiter: 	
                drop_incomplete_rows: False
                eval_filename: tests/data/test_data_tiny.tsv
                field_names: ['label', 'slots', 'text']
                quoted: False
                test_filename: tests/data/test_data_tiny.tsv
                train_filename: tests/data/train_data_tiny.tsv
        metric_reporter: ClassificationMetricReporter.Config:
            additional_column_names: []
            is_memory_efficient: False
            log_gradient: False
            model_select_metric: ComparableClassificationMetric.ACCURACY
            output_path: /tmp/test_out.txt
            pep_format: False
            recall_at_precision_thresholds: [0.2, 0.4, 0.6, 0.8, 0.9]
            student_column_names: []
            target_label: None
            text_column_names: ['text']
        model: DocModel.Config:
            decoder: MLPDecoder.Config:
                activation: Activation.RELU
                bias: True
                dropout: 0.0
                freeze: False
                hidden_dims: []
                layer_norm: False
                load_path: None
                out_dim: None
                save_path: None
                shared_module_key: None
            embedding: WordEmbedding.Config:
                cpu_only: False
                delimiter:  
                embed_dim: 100
                embeddding_init_std: 0.02
                embedding_init_range: None
                embedding_init_strategy: EmbedInitStrategy.RANDOM
                export_input_names: ['tokens_vals']
                freeze: False
                load_path: None
                lowercase_tokens: True
                min_freq: 1
                mlp_layer_dims: []
                padding_idx: None
                pretrained_embeddings_path: 
                save_path: None
                shared_module_key: None
                skip_header: True
                vocab_file: 
                vocab_from_all_data: False
                vocab_from_pretrained_embeddings: False
                vocab_from_train_data: True
                vocab_size: 0
            inputs: ModelInput:
                dense: None
                labels: LabelTensorizer.Config:
                    allow_unknown: False
                    column: label
                    is_input: False
                    label_vocab: None
                    label_vocab_file: None
                    pad_in_vocab: False
                tokens: TokenTensorizer.Config:
                    add_bos_token: False
                    add_eos_token: False
                    column: text
                    is_input: True
                    max_seq_len: None
                    tokenizer: Tokenizer.Config:
                        lowercase: True
                        split_regex: \s+
                    use_eos_token_for_bos: False
                    vocab: VocabConfig:
                        build_from_data: True
                        min_counts: 0
                        size_from_data: 0
                        vocab_files: []
                    vocab_file_delimiter:  
            output_layer: ClassificationOutputLayer.Config:
                freeze: False
                label_weights: None
                load_path: None
                loss: CrossEntropyLoss.Config:
                save_path: None
                shared_module_key: None
            representation: DocNNRepresentation.Config:
                cnn: CNNParams:
                    causal: False
                    dilated: False
                    kernel_num: 100
                    kernel_sizes: [3, 4]
                    weight_norm: False
                dropout: 0.4
                freeze: False
                load_path: None
                pooling: PoolingType.MAX
                save_path: None
                shared_module_key: None
        trainer: TaskTrainer.Config:
            do_eval: True
            early_stop_after: 0
            epochs: 10
            fp16_args: FP16OptimizerFairseq.Config:
                init_loss_scale: 128
                min_loss_scale: 0.0001
                scale_tolerance: 0.0
                scale_window: None
                threshold_loss_scale: None
            load_best_model_after_train: True
            max_clip_norm: None
            num_accumulated_batches: 1
            num_batches_per_epoch: None
            num_samples_to_log_progress: 1000
            optimizer: Adam.Config:
                eps: 1e-08
                lr: 0.001
                weight_decay: 1e-05
            privacy_engine: None
            report_train_metrics: True
            scheduler: None
            sparsifier: None
            target_time_limit_seconds: None
        use_elastic: None
    test_out_path: /tmp/test_out.txt
    torchscript_quantize: False
    use_config_from_snapshot: True
    use_cuda_for_testing: True
    use_cuda_if_available: True
    use_deterministic_cudnn: False
    use_fp16: False
    use_tensorboard: True
    version: 20


        # for debug of GPU
        use_cuda_if_available: True
        device_id: 0
        world_size: 1
        torch.cuda.is_available(): True
        cuda.CUDA_ENABLED: True
        cuda.DISTRIBUTED_WORLD_SIZE: 1
        
# for debug of FP16: fp16_enabled=False
Creating task: DocumentClassificationTask...
PyText data schema: {'text': <class 'str'>, 'label': <class 'str'>}.
Model :DocModel(
  (embedding): WordEmbedding(
    (word_embedding): Embedding(49, 100, padding_idx=1)
    (mlp): Sequential()
  )
  (representation): DocNNRepresentation(
    (convs): ModuleList(
      (0): Conv1d(100, 100, kernel_size=(3,), stride=(1,), padding=(3,))
      (1): Conv1d(100, 100, kernel_size=(4,), stride=(1,), padding=(4,))
    )
    (dropout): Dropout(p=0.4, inplace=False)
  )
  (decoder): MLPDecoder(
    (mlp): Sequential(
      (0): Linear(in_features=200, out_features=7, bias=True)
    )
  )
  (output_layer): MulticlassOutputLayer()
)
Num trainable parameters: 76507

Worker 0 starting epoch 1
Learning rate(s): 0.001
start training epoch 1


Stage.TRAIN
Epoch:1
loss: 1.950055
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             0.125 |   0.222 |
|          alarm/set_alarm |             0.306 |   0.381 |
|       alarm/snooze_alarm |             0.125 |   0.222 |
| alarm/time_left_on_alarm |             0.100 |   0.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.111 |   0.111 |
|             weather/find |             0.171 |   0.125 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| alarm/set_alarm          |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| alarm/snooze_alarm       |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| alarm/time_left_on_alarm |   0.100 |   0.100 |   0.100 |   0.100 |   0.100 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.111 |   0.111 |   0.111 |   0.111 |   0.111 |
| weather/find             |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  60.2ms |  60.2ms |  60.2ms |  60.2ms |  60.2ms |  60.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                | 268.4ms | 268.4ms | 268.4ms | 268.4ms | 268.4ms | 268.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |  12.5ns |  12.5ns |  12.5ns |  12.5ns |  12.5ns |  12.5ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 662.8ns | 662.8ns | 662.8ns | 662.8ns | 662.8ns | 662.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |  51.5ms |  51.5ms |  51.5ms |  51.5ms |  51.5ms |  51.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |  51.2ms |  51.2ms |  51.2ms |  51.2ms |  51.2ms |  51.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           | 207.1ms | 207.1ms | 207.1ms | 207.1ms | 207.1ms | 207.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   8.3ms |   8.3ms |   8.3ms |   8.3ms |   8.3ms |   8.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   8.1ms |   8.1ms |   8.1ms |   8.1ms |   8.1ms |   8.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              |  16.2ns |  16.2ns |  16.2ns |  16.2ns |  16.2ns |  16.2ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 332.6ms |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 1
/usr/local/lib/python3.6/dist-packages/pytext/metrics/__init__.py:576: RuntimeWarning: invalid value encountered in long_scalars
  recall_diff = added_positives / total_positive


Stage.EVAL
Epoch:1
loss: 1.959144
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.500 |   0.667 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.4ms |  20.4ms |  20.4ms |  20.4ms |  20.4ms |  20.4ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 238.1ns | 238.1ns | 238.1ns | 238.1ns | 238.1ns | 238.1ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   4.2ns |   4.2ns |   4.2ns |   4.2ns |   4.2ns |   4.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   7.1ns |   7.1ns |   7.1ns |   7.1ns |   7.1ns |   7.1ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 26.1ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
Found a better model!

Worker 0 starting epoch 2
Learning rate(s): 0.001
start training epoch 2
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:2
loss: 1.928607
Accuracy: 30.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             0.333 |   0.778 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             0.500 |   0.889 |
| alarm/time_left_on_alarm |             0.500 |   0.889 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             0.333 |   0.625 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  54.6ms |  54.6ms |  54.6ms |  54.6ms |  54.6ms |  54.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.7ms |   4.7ms |   4.7ms |   4.7ms |   4.7ms |   4.7ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.5ns |   8.5ns |   8.5ns |   8.5ns |   8.5ns |   8.5ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 253.1ns | 253.1ns | 253.1ns | 253.1ns | 253.1ns | 253.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 144.2ns | 144.2ns | 144.2ns | 144.2ns | 144.2ns | 144.2ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 62.2ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 2
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:2
loss: 1.956181
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.500 |   0.667 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.1ms |  20.1ms |  20.1ms |  20.1ms |  20.1ms |  20.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |   1.9ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.5ns |   5.5ns |   5.5ns |   5.5ns |   5.5ns |   5.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 242.4ns | 242.4ns | 242.4ns | 242.4ns | 242.4ns | 242.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.4ns |   3.4ns |   3.4ns |   3.4ns |   3.4ns |   3.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.6ns |   6.6ns |   6.6ns |   6.6ns |   6.6ns |   6.6ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 24.9ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 3
Learning rate(s): 0.001
start training epoch 3
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:3
loss: 1.935113
Accuracy: 20.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             0.125 |   0.222 |
|          alarm/set_alarm |             0.792 |   0.762 |
|       alarm/snooze_alarm |             0.111 |   0.111 |
| alarm/time_left_on_alarm |             0.200 |   0.556 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.125 |   0.222 |
|             weather/find |             0.367 |   0.688 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   0.667 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.375 |   0.375 |
| alarm/snooze_alarm       |   0.111 |   0.111 |   0.111 |   0.111 |   0.111 |
| alarm/time_left_on_alarm |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.125 |   0.125 |   0.125 |   0.125 |   0.125 |
| weather/find             |   0.400 |   0.400 |   0.400 |   0.400 |   0.400 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  53.0ms |  53.0ms |  53.0ms |  53.0ms |  53.0ms |  53.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.5ms |   4.5ms |   4.5ms |   4.5ms |   4.5ms |   4.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.7ns |   8.7ns |   8.7ns |   8.7ns |   8.7ns |   8.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 229.1ns | 229.1ns | 229.1ns | 229.1ns | 229.1ns | 229.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 134.1ns | 134.1ns | 134.1ns | 134.1ns | 134.1ns | 134.1ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 60.5ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 3
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:3
loss: 1.953246
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.8ms |  20.8ms |  20.8ms |  20.8ms |  20.8ms |  20.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |   5.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 208.2ns | 208.2ns | 208.2ns | 208.2ns | 208.2ns | 208.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.5ns |   3.5ns |   3.5ns |   3.5ns |   3.5ns |   3.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   5.6ns |   5.6ns |   5.6ns |   5.6ns |   5.6ns |   5.6ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 25.3ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 4
Learning rate(s): 0.001
start training epoch 4
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:4
loss: 1.919790
Accuracy: 20.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             0.200 |   0.556 |
| alarm/time_left_on_alarm |             0.500 |   0.889 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.200 |   0.556 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
| alarm/time_left_on_alarm |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.200 |   0.200 |   0.200 |   0.200 |   0.200 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  53.4ms |  53.4ms |  53.4ms |  53.4ms |  53.4ms |  53.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.9ms |   4.9ms |   4.9ms |   4.9ms |   4.9ms |   4.9ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |  22.3ns |  22.3ns |  22.3ns |  22.3ns |  22.3ns |  22.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 255.4ns | 255.4ns | 255.4ns | 255.4ns | 255.4ns | 255.4ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   4.1ns |   4.1ns |   4.1ns |   4.1ns |   4.1ns |   4.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 152.8ns | 152.8ns | 152.8ns | 152.8ns | 152.8ns | 152.8ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 61.3ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 4
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:4
loss: 1.950457
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 229.9ns | 229.9ns | 229.9ns | 229.9ns | 229.9ns | 229.9ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |  20.4ns |  20.4ns |  20.4ns |  20.4ns |  20.4ns |  20.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 23.8ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 5
Learning rate(s): 0.001
start training epoch 5
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:5
loss: 1.918015
Accuracy: 20.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             0.698 |   0.762 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             0.100 |   0.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.333 |   0.778 |
|             weather/find |             0.583 |   0.875 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.333 |   0.333 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   0.667 |   0.667 |   0.429 |   0.429 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   0.100 |   0.100 |   0.100 |   0.100 |   0.100 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| weather/find             |   0.667 |   0.667 |   0.667 |   0.667 |   0.667 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  76.5ms |  76.5ms |  76.5ms |  76.5ms |  76.5ms |  76.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |  23.4ns |  23.4ns |  23.4ns |  23.4ns |  23.4ns |  23.4ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 223.2ns | 223.2ns | 223.2ns | 223.2ns | 223.2ns | 223.2ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 147.9ns | 147.9ns | 147.9ns | 147.9ns | 147.9ns | 147.9ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 84.0ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 5
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:5
loss: 1.947547
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  20.7ms |  20.7ms |  20.7ms |  20.7ms |  20.7ms |  20.7ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |   2.2ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 265.8ns | 265.8ns | 265.8ns | 265.8ns | 265.8ns | 265.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |   6.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   5.0ns |   5.0ns |   5.0ns |   5.0ns |   5.0ns |   5.0ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   7.3ns |   7.3ns |   7.3ns |   7.3ns |   7.3ns |   7.3ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 26.0ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 6
Learning rate(s): 0.001
start training epoch 6
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:6
loss: 1.891813
Accuracy: 40.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             0.500 |   0.889 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.500 |   0.500 |   0.500 |   0.500 |   0.500 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  54.4ms |  54.4ms |  54.4ms |  54.4ms |  54.4ms |  54.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.3ms |   4.3ms |   4.3ms |   4.3ms |   4.3ms |   4.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.8ns |   8.8ns |   8.8ns |   8.8ns |   8.8ns |   8.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 248.9ns | 248.9ns | 248.9ns | 248.9ns | 248.9ns | 248.9ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 131.8ns | 131.8ns | 131.8ns | 131.8ns | 131.8ns | 131.8ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 61.7ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 6
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:6
loss: 1.944635
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |  19.3ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 221.0ns | 221.0ns | 221.0ns | 221.0ns | 221.0ns | 221.0ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.9ns |   5.9ns |   5.9ns |   5.9ns |   5.9ns |   5.9ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch | 997.5ns | 997.5ns | 997.5ns | 997.5ns | 997.5ns | 997.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |   4.5ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |   3.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |   6.5ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 23.9ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 7
Learning rate(s): 0.001
start training epoch 7
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:7
loss: 1.895764
Accuracy: 30.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             0.833 |   0.938 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  53.1ms |  53.1ms |  53.1ms |  53.1ms |  53.1ms |  53.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   6.5ms |   6.5ms |   6.5ms |   6.5ms |   6.5ms |   6.5ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   9.3ns |   9.3ns |   9.3ns |   9.3ns |   9.3ns |   9.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 274.2ns | 274.2ns | 274.2ns | 274.2ns | 274.2ns | 274.2ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   2.4ms |   2.4ms |   2.4ms |   2.4ms |   2.4ms |   2.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.9ns |   3.9ns |   3.9ns |   3.9ns |   3.9ns |   3.9ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 146.6ns | 146.6ns | 146.6ns | 146.6ns | 146.6ns | 146.6ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 62.6ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 7
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:7
loss: 1.941633
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.0ms |   2.0ms |   2.0ms |   2.0ms |   2.0ms |   2.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.0ns |   6.0ns |   6.0ns |   6.0ns |   6.0ns |   6.0ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 209.7ns | 209.7ns | 209.7ns | 209.7ns | 209.7ns | 209.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |   4.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |   6.3ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 24.5ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 8
Learning rate(s): 0.001
start training epoch 8
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:8
loss: 1.885373
Accuracy: 40.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             0.333 |   0.778 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  61.6ms |  61.6ms |  61.6ms |  61.6ms |  61.6ms |  61.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   5.6ms |   5.6ms |   5.6ms |   5.6ms |   5.6ms |   5.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 266.3ns | 266.3ns | 266.3ns | 266.3ns | 266.3ns | 266.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |   1.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |   4.0ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 190.7ns | 190.7ns | 190.7ns | 190.7ns | 190.7ns | 190.7ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 70.1ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 8
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:8
loss: 1.938505
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             0.833 |   0.750 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.500 |   0.500 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.5ms |  19.5ms |  19.5ms |  19.5ms |  19.5ms |  19.5ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   2.5ms |   2.5ms |   2.5ms |   2.5ms |   2.5ms |   2.5ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |   5.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 234.4ns | 234.4ns | 234.4ns | 234.4ns | 234.4ns | 234.4ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |   1.7ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.7ns |   4.7ns |   4.7ns |   4.7ns |   4.7ns |   4.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 25.5ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 9
Learning rate(s): 0.001
start training epoch 9
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:9
loss: 1.872775
Accuracy: 60.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  62.0ms |  62.0ms |  62.0ms |  62.0ms |  62.0ms |  62.0ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |   9.1ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 250.3ns | 250.3ns | 250.3ns | 250.3ns | 250.3ns | 250.3ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |   1.3ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 156.4ns | 156.4ns | 156.4ns | 156.4ns | 156.4ns | 156.4ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 69.5ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 9
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:9
loss: 1.935212
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |  19.6ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 206.6ns | 206.6ns | 206.6ns | 206.6ns | 206.6ns | 206.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |   5.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.6ns |   4.6ns |   4.6ns |   4.6ns |   4.6ns |   4.6ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |   6.1ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 24.1ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

Worker 0 starting epoch 10
Learning rate(s): 0.001
start training epoch 10
Get numberized rows from cache in stage: Stage.TRAIN


Stage.TRAIN
Epoch:10
loss: 1.872801
Accuracy: 60.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |             1.000 |   1.000 |
|          alarm/set_alarm |             0.792 |   0.762 |
|       alarm/snooze_alarm |             1.000 |   1.000 |
| alarm/time_left_on_alarm |             1.000 |   1.000 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |             1.000 |   1.000 |
|             weather/find |             1.000 |   1.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   0.667 |   0.667 |   0.667 |   0.667 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   0.375 |   0.375 |
| alarm/snooze_alarm       |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/time_left_on_alarm |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| weather/find             |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
+--------------------------+---------+---------+---------+---------+---------+
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                              | Total   | Average | Max     | P50     | P90     | P99     | Count |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> train epoch -> report metrics                          |  58.4ms |  58.4ms |  58.4ms |  58.4ms |  58.4ms |  58.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step                                |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |   4.6ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add gradients               |   8.6ns |   8.6ns |   8.6ns |   8.6ns |   8.6ns |   8.6ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> add metrics                 | 246.9ns | 246.9ns | 246.9ns | 246.9ns | 246.9ns | 246.9ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop                    |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |   1.4ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> backprop -> loss.backward   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> model.train_batch           |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer                   |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |   1.2ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> optimizer -> optimizer.step |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |   1.1ms |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> sparsifier                  |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |   3.8ns |     1 |
| Trainer.train_from_state -> train epoch -> run_step -> zero gradients              | 130.3ns | 130.3ns | 130.3ns | 130.3ns | 130.3ns | 130.3ns |     1 |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                                         | 65.9ms  |         |         |         |         |         |       |
+------------------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
start evaluating epoch 10
Get numberized rows from cache in stage: Stage.EVAL


Stage.EVAL
Epoch:10
loss: 1.931939
Accuracy: 0.00

Soft Metrics:
+--------------------------+-------------------+---------+
| Label                    | Average precision | ROC AUC |
+--------------------------+-------------------+---------+
|       alarm/modify_alarm |               nan |   0.000 |
|          alarm/set_alarm |             1.000 |   1.000 |
|       alarm/snooze_alarm |               nan |   0.000 |
| alarm/time_left_on_alarm |             0.333 |   0.333 |
|    reminder/set_reminder |             1.000 |   1.000 |
|  reminder/show_reminders |               nan |   0.000 |
|             weather/find |               nan |   0.000 |
+--------------------------+-------------------+---------+

Recall at Precision
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | R@P 0.2 | R@P 0.4 | R@P 0.6 | R@P 0.8 | R@P 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   1.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+

Precision at Recall
+--------------------------+---------+---------+---------+---------+---------+
| Label                    | P@R 0.2 | P@R 0.4 | P@R 0.6 | P@R 0.8 | P@R 0.9 |
+--------------------------+---------+---------+---------+---------+---------+
| alarm/modify_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/set_alarm          |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| alarm/snooze_alarm       |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| alarm/time_left_on_alarm |   0.333 |   0.333 |   0.333 |   0.333 |   0.333 |
| reminder/set_reminder    |   1.000 |   1.000 |   1.000 |   1.000 |   1.000 |
| reminder/show_reminders  |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
| weather/find             |   0.000 |   0.000 |   0.000 |   0.000 |   0.000 |
+--------------------------+---------+---------+---------+---------+---------+
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Stage                                                                   | Total   | Average | Max     | P50     | P90     | P99     | Count |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Trainer.train_from_state -> eval epoch -> report metrics                |  26.1ms |  26.1ms |  26.1ms |  26.1ms |  26.1ms |  26.1ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step                      |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |   1.8ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add gradients     |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> add metrics       | 217.9ns | 217.9ns | 217.9ns | 217.9ns | 217.9ns | 217.9ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> backprop          |  20.8ns |  20.8ns |  20.8ns |  20.8ns |  20.8ns |  20.8ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> model.train_batch |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |   1.0ms |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> optimizer         |   4.3ns |   4.3ns |   4.3ns |   4.3ns |   4.3ns |   4.3ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> sparsifier        |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |   3.7ns |     1 |
| Trainer.train_from_state -> eval epoch -> run_step -> zero gradients    |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |   6.2ns |     1 |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+
| Total time                                                              | 30.6ms  |         |         |         |         |         |       |
+-------------------------------------------------------------------------+---------+---------+---------+---------+---------+---------+-------+

=== Saving model to: /tmp/model.pt
Saving pytorch model to: /tmp/model.pt
Exporting metrics
WARNING: Unable to visualize embedding space in TensorBoard.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/pytext/metric_reporters/channel.py", line 376, in export
    model.embedding.visualize(self.summary_writer)
  File "/usr/local/lib/python3.6/dist-packages/pytext/models/embeddings/word_embedding.py", line 168, in visualize
    self.word_embedding.weight, metadata=self.vocab
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/tensorboard/writer.py", line 788, in add_embedding
    fs = tf.io.gfile.get_filesystem(save_path)
AttributeError: module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'
Saving caffe2 model to: /tmp/model.caffe2.predictor
Saving onnx model to: /tmp/model.onnx
/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py:243: UserWarning: `add_node_names' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `add_node_names` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
/usr/local/lib/python3.6/dist-packages/torch/onnx/utils.py:243: UserWarning: `do_constant_folding' can be set to True only when 'operator_export_type' is `ONNX`. Since 'operator_export_type' is not set to 'ONNX', `do_constant_folding` argument will be ignored.
  "`{}` argument will be ignored.".format(arg_name, arg_name))
/bin/bash: line 1:   357 Segmentation fault      (core dumped) pytext train < pytext/demo/configs/docnn.json

Expected Results

The model snapshot should have been saved without a segmentation fault.

Relevant Code

Set up environment:

!pip install --quiet git+https://github.com/facebookresearch/pytext
!git clone https://github.com/facebookresearch/pytext.git
!mv pytext/tests/ .

Import modules:

import pytext

Train:

!pytext train < pytext/demo/configs/docnn.json
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@orestxherija