notebooks/api/0.8/00-load-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading data into Syft Domain Server as a Data Owner\n",
    "\n",
    "Welcome to Syft! This tutorial consists of 4 Jupyter notebooks that covers the basics of Syft which includes\n",
    "* [Uploading a private dataset as a Data Owner](./00-load-data.ipynb)\n",
    "* [Submitting code to run analysis on the private dataset as a Data Scientist](./01-submit-code.ipynb)\n",
    "* [Reviewing and approving the code as a Data Owner](02-review-code-and-approve.ipynb)\n",
    "* [Downloading/Retrieving the results of the code execution as a Data Scientist](03-data-scientist-download-result.ipynb)\n",
    "\n",
    "In Syft, a **Data Owner** provides datasets which they would like to make available for study by an outside party they may or may not fully trust has good intentions. Meanwhile, **Data Scientists** are end users who desire to perform computations or answer a specific question using one or more Data Owners' datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Syft & Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "SYFT_VERSION = \">=0.8.2.b0,<0.9\"\n",
    "package_string = f'\"syft{SYFT_VERSION}\"'\n",
    "# %pip install {package_string} -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import syft as sy\n",
    "sy.requires(SYFT_VERSION)\n",
    "from syft import autocache\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch a Syft Domain Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Launch a fresh domain server named \"test-domain-1\" in dev mode on the local machine\n",
    "node = sy.orchestra.launch(name=\"test-domain-1\", port=\"auto\", dev_mode=True, reset=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# log into the node with default root credentials\n",
    "domain_client = node.login(email=\"info@openmined.org\", password=\"changethis\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# List the available API\n",
    "domain_client.api"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Subjects\n",
    "\n",
    "Think of Data Subjects as individuals/organizations/institutions owning a dataset that you can pool together privately in Syft.\n",
    "\n",
    "For this notebook, we'll create a sample dataset that includes trade information of various commodities for different countries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Check for existing Data Subjects\n",
    "data_subjects = domain_client.data_subject_registry.get_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_subjects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert len(data_subjects) == 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Data subjects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "country = sy.DataSubject(name=\"Country\", aliases=[\"country_code\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "canada = sy.DataSubject(name=\"Canada\", aliases=[\"country_code:ca\"])\n",
    "germany = sy.DataSubject(name=\"Germany\", aliases=[\"country_code:de\"])\n",
    "spain = sy.DataSubject(name=\"Spain\", aliases=[\"country_code:es\"])\n",
    "france = sy.DataSubject(name=\"France\", aliases=[\"country_code:fr\"])\n",
    "japan = sy.DataSubject(name=\"Japan\", aliases=[\"country_code:jp\"])\n",
    "uk = sy.DataSubject(name=\"United Kingdom\", aliases=[\"country_code:uk\"])\n",
    "usa = sy.DataSubject(name=\"United States of America\", aliases=[\"country_code:us\"])\n",
    "australia = sy.DataSubject(name=\"Australia\", aliases=[\"country_code:au\"])\n",
    "india = sy.DataSubject(name=\"India\", aliases=[\"country_code:in\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "country.add_member(canada)\n",
    "country.add_member(germany)\n",
    "country.add_member(spain)\n",
    "country.add_member(france)\n",
    "country.add_member(japan)\n",
    "country.add_member(uk)\n",
    "country.add_member(usa)\n",
    "country.add_member(australia)\n",
    "country.add_member(india)\n",
    "\n",
    "country.members"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Adds the data subject and all its members to the registry\n",
    "response = domain_client.data_subject_registry.add_data_subject(country)\n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lets look at the data subjects added to the data\n",
    "data_subjects = domain_client.data_subject_registry.get_all()\n",
    "data_subjects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert len(data_subjects) == 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, we'll be working with Canada's trade dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "canada_dataset_url = \"https://github.com/OpenMined/datasets/blob/main/trade_flow/ca%20-%20feb%202021.csv?raw=True\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(autocache(canada_dataset_url))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Syft, every dataset has two variants - **Mock** and **Private**.\n",
    "\n",
    "* **Mock** dataset is a mock/dummy version of the private data that can be accessed & read by the data scientists.\n",
    "* **Private** dataset is the actual data that will never be accessed by the data scientist.\n",
    "\n",
    "To keep things simple, we sample different data points as Mock & Private. But in reality you would want to generate a random dataset for the Mock variant. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# private data samples\n",
    "ca_data = df[0:10]\n",
    "ca_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Mock data samples\n",
    "mock_ca_data = df[10:20]\n",
    "mock_ca_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Syft Dataset\n",
    "\n",
    "In Syft, `Dataset` is a collection of Assets. For example, `Dataset` can be a \"Lung Cancer Dataset\", and `Assets` will be train, test & validation splits for this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset = sy.Dataset(name=\"Canada Trade Value\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset.set_description(\"Canada Trade Data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset.add_citation(\"Person, place or thing\")\n",
    "dataset.add_url(\"https://github.com/OpenMined/datasets/tree/main/trade_flow\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset.add_contributor(name=\"Andrew Trask\", \n",
    "                        email=\"andrew@openmined.org\",\n",
    "                        note=\"Andrew runs this domain and prepared the dataset metadata.\")\n",
    "\n",
    "dataset.add_contributor(name=\"Madhava Jay\", \n",
    "                        email=\"madhava@openmined.org\",\n",
    "                        note=\"Madhava tweaked the description to add the URL because Andrew forgot.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset.contributors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert len(dataset.contributors) == 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add Assets to the Syft Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ctf = sy.Asset(name=\"canada_trade_flow\")\n",
    "ctf.set_description(\"Canada trade flow represents export & import of different commodities to other countries\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ctf.add_contributor(name=\"Andrew Trask\", \n",
    "                    email=\"andrew@openmined.org\",\n",
    "                    note=\"Andrew runs this domain and prepared the asset.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This is where we add the private data (pandas df/numpy array) to the `Asset`\n",
    "ctf.set_obj(ca_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We must set the shape of this private data\n",
    "ctf.set_shape(ca_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We assign the data subject for whom this data belongs to, in this\n",
    "ctf.add_data_subject(canada)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optionally, if we don't want to add any Mock dataset\n",
    "ctf.no_mock()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We must add this Asset to our Dataset\n",
    "dataset.add_asset(ctf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# In case we want to remove a dataset & its associated assets\n",
    "dataset.remove_asset(name=ctf.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Let's assign the Mock data to the Asset by calling `set_mock` method\n",
    "ctf.set_mock(mock_ca_data, mock_is_real=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's add our Asset back into our \"Canada Trade Value\" Dataset\n",
    "dataset.add_asset(ctf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload Syft Dataset to Domain Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "domain_client.upload_dataset(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can list all the datasets on the Domain Server by invoking the following\n",
    "datasets = domain_client.datasets.get_all()\n",
    "datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert len(datasets) == 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the Syft Dataset from Domain Server\n",
    "\n",
    "Following the logical hierarchy of `Dataset`, `Asset`, and its variant, we can read the data as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Reading the mock dataset\n",
    "mock = domain_client.datasets[0].assets[0].mock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert mock_ca_data.equals(mock)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Reading the real dataset\n",
    "# NOTE: Private data can be accessed by the Data Owners, but NOT the Data Scientists\n",
    "real = domain_client.datasets[0].assets[0].data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert ca_data.equals(real)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a new Data Scientist account on the Domain Server\n",
    "\n",
    "Signup is disabled by default.\n",
    "An Admin/DO can enable it by `domain_client.settings.allow_guest_signup(enable=True)`\n",
    "\n",
    "Refer to notebook [07-domain-register-control-flow](./07-domain-register-control-flow.ipynb) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "domain_client.register(name=\"Jane Doe\", email=\"jane@caltech.edu\", password=\"abc123\", password_verify=\"abc123\", institution=\"Caltech\", website=\"https://www.caltech.edu/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Cleanup local domain server\n",
    "if node.node_type.value == \"python\":\n",
    "    node.land()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}