Fine Tune Model for QNA

[ayush714]

Hi, I am using T5 Mesh Transformer for Fine tuning to build QNA Model, I have a dataset which contains question, context, answer_text, I want to fine tune Mesh Transformer on this dataset, Can you tell me how to do this? I have tried but at the end it says

I0902 17:57:53.197856 139964293789056 ops.py:5757] Done writing checkpoint.
2021-09-02 17:58:00.329768: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
[1]    7455 killed     t5_mesh_transformer --tpu="${dev_tfmt_tpu}"  --tpu_zone="${us-central1-b}"    

I don't know if is correct or not.

[craffel]

The log messages you're getting before the process is killed is benign and unrelated. It's impossible to say why your process is getting killed, sorry.

[ayush714]

@craffel Here's the script that I am running

        t5_mesh_transformer  \
            --tpu="${dev_tfmt_tpu}" \
            --gcp_project="${artifact-findings-dev}" \
            --tpu_zone="${us-central1-b}" \
            --model_dir="${t5-data/pretrained_models/large}" \
            --t5_tfds_data_dir="${/home/welcome/ml-experimentation-dev/research/qna_finetuning/code/exp_directory/dev-testing/artifact_train.tsv}" \
            --gin_file="dataset.gin" \
            --gin_param="utils.tpu_mesh_shape.model_parallelism = 1" \
            --gin_param="utils.tpu_mesh_shape.tpu_topology = '${v3-8}'" \
            --gin_param="MIXTURE_NAME = 'squad_v010_allanswers'" \
            --gin_file="gs://t5-data/pretrained_models/small/operative_config.gin"

[craffel]

It is probably an issue with your system - driver issues, out of memory, etc. I don't think it's something one could debug without access to your system.

[ayush714]

@craffel I got you, but I am using TSV file directly, So Here is my fine tuning script, but I am getting an error,
Here is the fine tuning script,

        t5_mesh_transformer  \
            --tpu = "${dev_tfmt_tpu}" \
            --gcp_project = "${artifact-findings-dev}" \
            --tpu_zone = "${us-central1-b}" \
            --model_dir = "${t5-data/pretrained_models/large}" \
            --t5_tfds_data_dir = "${/home/welcome/ml-experimentation-dev/research/qna_finetuning/code/exp_directory/dev-testing/artifact_train.tsv}" \
            --gin_file = "dataset.gin" \
            --gin_param = "utils.tpu_mesh_shape.model_parallelism = 1" \
            --gin_param = "utils.tpu_mesh_shape.tpu_topology = '${v3-8}'" \
            --gin_param = "utils.run.train_dataset_fn = @t5.models.mesh_transformer.tsv_dataset_fn" \
            --gin_param = "tsv_dataset_fn.filename = 'gs:/path/to/tsv'" \
            --gin_file = "learning_rate_schedules/constant_0_001.gin" \
            --gin_file = "gs://t5-data/pretrained_models/small/operative_config.gin"

and here is the error that I am getting:-

Plz let me know what issue it is and how I can fix it?