Skip to content

Commit

Permalink
use spawn method
Browse files Browse the repository at this point in the history
  • Loading branch information
@holgerroth
holgerroth committed Jan 20, 2024
1 parent 024a1b7 commit 3f6fc77
Showing 1 changed file with 46 additions and 38 deletions.
84 changes: 46 additions & 38 deletions integration/nemo/examples/peft/peft.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "ff99e42d",
"id": "5020dd81",
"metadata": {},
"source": [
"# Parameter-Efficient Fine-Tuning (PEFT) with NeMo\n",
Expand All @@ -19,7 +19,7 @@
},
{
"cell_type": "markdown",
"id": "717d32c7",
"id": "dc9769ef",
"metadata": {},
"source": [
"## Dependencies\n",
Expand All @@ -29,7 +29,7 @@
},
{
"cell_type": "markdown",
"id": "558d3480",
"id": "dab4c639",
"metadata": {},
"source": [
"## Download the pre-trained LLM\n",
Expand All @@ -39,7 +39,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "1c5bbaf7",
"id": "20921eea",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -51,7 +51,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "9f067cde",
"id": "aa852d07",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -66,7 +66,7 @@
},
{
"cell_type": "markdown",
"id": "8a31665b",
"id": "fa530a42",
"metadata": {},
"source": [
"## Data preprocessing\n",
Expand All @@ -79,7 +79,7 @@
},
{
"cell_type": "markdown",
"id": "91593e61",
"id": "b5737e50",
"metadata": {},
"source": [
"#### 1. Download the preprocessing scripts\n",
Expand All @@ -89,7 +89,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "959ce879",
"id": "b2c32fa5",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -102,7 +102,7 @@
},
{
"cell_type": "markdown",
"id": "1d00be68",
"id": "13a2f952",
"metadata": {},
"source": [
"#### 2. Download the Financial PhraseBank Dataset\n",
Expand All @@ -114,7 +114,7 @@
},
{
"cell_type": "markdown",
"id": "0813e466",
"id": "40199807",
"metadata": {},
"source": [
"#### 3. Preprocess the dataset"
Expand All @@ -123,7 +123,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "905f75cc",
"id": "80f84586",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -132,7 +132,7 @@
},
{
"cell_type": "markdown",
"id": "69d393fd",
"id": "d9f8fa9a",
"metadata": {},
"source": [
"#### 4. Split the dataset to simulate clients\n",
Expand All @@ -143,7 +143,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "a582628c",
"id": "a6725683",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -160,7 +160,7 @@
},
{
"cell_type": "markdown",
"id": "a15411e3",
"id": "6c506c6b",
"metadata": {},
"source": [
"Below are some examples of how the training data is distributed amount the three clients when using different values of `alpha`.\n",
Expand All @@ -173,7 +173,7 @@
},
{
"cell_type": "markdown",
"id": "331c965c",
"id": "704ff05d",
"metadata": {},
"source": [
"## Federated learning simulations\n",
Expand All @@ -187,7 +187,7 @@
},
{
"cell_type": "markdown",
"id": "eed15958",
"id": "01fce4ae",
"metadata": {},
"source": [
"#### 1. Convert NeMo PEFT script to FL\n",
Expand Down Expand Up @@ -221,7 +221,7 @@
},
{
"cell_type": "markdown",
"id": "211e60fc",
"id": "655a1f0a",
"metadata": {},
"source": [
"#### 1. Local training\n",
Expand All @@ -236,7 +236,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "eb1dd44a",
"id": "51e4fb4d",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -245,7 +245,7 @@
},
{
"cell_type": "markdown",
"id": "4fcf9002",
"id": "2e515dc2",
"metadata": {},
"source": [
"Then, create the job and configure it for simulating local training."
Expand All @@ -254,7 +254,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "692d1249",
"id": "404fe5fe",
"metadata": {
"scrolled": true
},
Expand Down Expand Up @@ -283,7 +283,7 @@
},
{
"cell_type": "markdown",
"id": "6336e19b",
"id": "df9ca0a5",
"metadata": {},
"source": [
"Next, simulate each client training on their local dataset using the FL simulator. To do this, we only run 1 round of FL, with each client running 1000 steps on their local dataset."
Expand All @@ -292,12 +292,16 @@
{
"cell_type": "code",
"execution_count": null,
"id": "5dd4879f",
"id": "8d7f4970",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# required by NeMo models\n",
"import torch.multiprocessing as mp\n",
"mp.set_start_method(\"spawn\", force=True)\n",
"\n",
"from nvflare import SimulatorRunner \n",
"\n",
"simulator = SimulatorRunner(\n",
Expand All @@ -312,7 +316,7 @@
},
{
"cell_type": "markdown",
"id": "fff894a8",
"id": "2e56653f",
"metadata": {},
"source": [
"#### 2. Federated training\n",
Expand All @@ -325,7 +329,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "708d4ebb",
"id": "ad3406a6",
"metadata": {
"scrolled": true
},
Expand All @@ -344,7 +348,7 @@
},
{
"cell_type": "markdown",
"id": "3bc10072",
"id": "5e591653",
"metadata": {},
"source": [
"Next, simulate the federated training using FedAvg. "
Expand All @@ -353,12 +357,16 @@
{
"cell_type": "code",
"execution_count": null,
"id": "c0ad9fde",
"id": "8559b79f",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# required by NeMo models\n",
"import torch.multiprocessing as mp\n",
"mp.set_start_method(\"spawn\", force=True)\n",
"\n",
"from nvflare import SimulatorRunner \n",
"\n",
"simulator = SimulatorRunner(\n",
Expand All @@ -373,15 +381,15 @@
},
{
"cell_type": "markdown",
"id": "43bf418d",
"id": "3e20ca56",
"metadata": {},
"source": [
"You can visualize the training process using TensorBoard by running `tensorboard --logdir /tmp/nvflare/nemo` in a new terminal."
]
},
{
"cell_type": "markdown",
"id": "6af6eef6",
"id": "a8e5b7c0",
"metadata": {},
"source": [
"## Results\n",
Expand All @@ -397,7 +405,7 @@
},
{
"cell_type": "markdown",
"id": "76ebc045",
"id": "65833f4b",
"metadata": {},
"source": [
"## Inference\n",
Expand All @@ -409,7 +417,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "a51503e4",
"id": "dcf08bc6",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -424,7 +432,7 @@
},
{
"cell_type": "markdown",
"id": "bbf79362",
"id": "7b3667c0",
"metadata": {},
"source": [
"First, we need to convert the best global PEFT model into a NeMo ckpt."
Expand All @@ -433,7 +441,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "4b2523d9",
"id": "3d08150a",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -448,7 +456,7 @@
},
{
"cell_type": "markdown",
"id": "13ccb896",
"id": "2963d08e",
"metadata": {},
"source": [
"Next, we will load the global model."
Expand All @@ -457,7 +465,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "9951bcfe",
"id": "9ffe513d",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -491,7 +499,7 @@
},
{
"cell_type": "markdown",
"id": "0c87b2c9",
"id": "59fa62cb",
"metadata": {},
"source": [
"Run the model"
Expand All @@ -500,7 +508,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "bf4d10e4",
"id": "acd89469",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -527,7 +535,7 @@
},
{
"cell_type": "markdown",
"id": "e3fc97f9",
"id": "d14026fc",
"metadata": {},
"source": [
"The expected output of a well-trained model looks something like this. Note, the test sentences do not include ground truth labels.\n",
Expand All @@ -547,7 +555,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "87a28bb6",
"id": "db70a19a",
"metadata": {},
"outputs": [],
"source": []
Expand Down

0 comments on commit 3f6fc77

Please sign in to comment.