notebooks/Datasets in the cloud - how to pull and push.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datasets in the cloud - how to pull and push"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[osfclient](https://osfclient.readthedocs.io/en/latest/) allows us to store out datasets in OSF and retrieve files using a command-line interface."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the local directory\n",
    "```bash\n",
    "$ echo \"export OSF_PASSWORD='your password'\" > ~/.osfrc\n",
    "$ source ~/.osf.rc\n",
    "```\n",
    "Go to the local directory you want to interface with the OSF project\n",
    "```bash\n",
    "osf init\n",
    "```\n",
    "> give your OSF username\n",
    "\n",
    "> give the OSF project idea"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List OSF project content\n",
    "```bash\n",
    "$ osf ls > osf_ls.txt\n",
    "$ head osf_ls.txt\n",
    "    osfstorage/foo/foo.txt\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload to a project\n",
    "\n",
    "### Upload a single file\n",
    "\n",
    "Say you want to upload the file `/dir1/subdir1/toto.txt` to the root directory on OSF:\n",
    "\n",
    "`$ osf upload /dir1/subdir1/toto.txt /`\n",
    "\n",
    "Now assuming you want to move it to some pre-existing `/dir2/subdir2` directory on OSF:\n",
    "\n",
    "`$ osf upload /dir1/subdir1/toto.txt /dir2/subdir2/toto.txt`\n",
    "\n",
    "### Upload a directory\n",
    "\n",
    "`$ osf upload -r local/directory/ /remote/directory` \n",
    "\n",
    "### Advanced usage\n",
    "\n",
    "To update the content of an existing directory, one first needs to list its current content and upload what is not there yet. We provide here a `bash` script approach to it, assuming proper treatment of the arguments, existence etc. A full working script can be found on our [github](https://github.com/csblab/PDBClean/blob/master/scripts/osftool.sh).\n",
    "\n",
    "```bash\n",
    "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n",
    "osfls='osf_ls.txt'\n",
    "if ! grep -q \"$localdir\" $osfls; then\n",
    "  osf upload -r $localdir /\n",
    "else\n",
    "  for file in $localdir/*; do\n",
    "    if ! grep -q \"$file\" $osfls; then\n",
    "      osf upload \"$file\" \"/$file\"; fi\n",
    "  done\n",
    "fi\n",
    "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download from a project\n",
    "\n",
    "Careful! The following does not seem to work if the project is Private...\n",
    "\n",
    "### Download a single file\n",
    "\n",
    "`$ osf fetch /osfstorage/dir2/subdir2/toto.txt dir1/subdir1/toto.txt`\n",
    "\n",
    "### Download based on keyword search\n",
    "There is now recursive option in `osfclient` yet for download purposes. Here is a workaround, again assuming proper treatment of arguments, existence etc. A full working script can be found on our [github](https://github.com/csblab/PDBClean/blob/master/scripts/osftool.sh).\n",
    "\n",
    "```bash\n",
    "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n",
    "osfls='osf_ls.txt'\n",
    "if ! grep -q \"$remotedir\" $osfls; then\n",
    "  echo \"$remotedir was not found in OSF...\"\n",
    "else\n",
    "  grep \"$remotedir\" $osfls > \"tmp_${remotedir}.txt\"\n",
    "  while read -r line\n",
    "  do\n",
    "    if [ ! -f ${localdir}/${line##*/} ]; then\n",
    "      echo \">>> $line ...\"\n",
    "      osf fetch $line ${localdir}/${line##*/}\n",
    "    fi\n",
    "  done < tmp_${remotedir}.txt\n",
    "fi\n",
    "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n",
    "```\n",
    "\n",
    "### Download whole project\n",
    "\n",
    "This one is easy: `$ osf clone`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}