{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Datasets in the cloud - how to pull and push" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[osfclient](https://osfclient.readthedocs.io/en/latest/) allows us to store out datasets in OSF and retrieve files using a command-line interface." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setting up the local directory\n", "```bash\n", "$ echo \"export OSF_PASSWORD='your password'\" > ~/.osfrc\n", "$ source ~/.osf.rc\n", "```\n", "Go to the local directory you want to interface with the OSF project\n", "```bash\n", "osf init\n", "```\n", "> give your OSF username\n", "\n", "> give the OSF project idea" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## List OSF project content\n", "```bash\n", "$ osf ls > osf_ls.txt\n", "$ head osf_ls.txt\n", " osfstorage/foo/foo.txt\n", " ...\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Upload to a project\n", "\n", "### Upload a single file\n", "\n", "Say you want to upload the file `/dir1/subdir1/toto.txt` to the root directory on OSF:\n", "\n", "`$ osf upload /dir1/subdir1/toto.txt /`\n", "\n", "Now assuming you want to move it to some pre-existing `/dir2/subdir2` directory on OSF:\n", "\n", "`$ osf upload /dir1/subdir1/toto.txt /dir2/subdir2/toto.txt`\n", "\n", "### Upload a directory\n", "\n", "`$ osf upload -r local/directory/ /remote/directory` \n", "\n", "### Advanced usage\n", "\n", "To update the content of an existing directory, one first needs to list its current content and upload what is not there yet. We provide here a `bash` script approach to it, assuming proper treatment of the arguments, existence etc. A full working script can be found on our [github](https://github.com/csblab/PDBClean/blob/master/scripts/osftool.sh).\n", "\n", "```bash\n", "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n", "osfls='osf_ls.txt'\n", "if ! grep -q \"$localdir\" $osfls; then\n", " osf upload -r $localdir /\n", "else\n", " for file in $localdir/*; do\n", " if ! grep -q \"$file\" $osfls; then\n", " osf upload \"$file\" \"/$file\"; fi\n", " done\n", "fi\n", "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Download from a project\n", "\n", "Careful! The following does not seem to work if the project is Private...\n", "\n", "### Download a single file\n", "\n", "`$ osf fetch /osfstorage/dir2/subdir2/toto.txt dir1/subdir1/toto.txt`\n", "\n", "### Download based on keyword search\n", "There is now recursive option in `osfclient` yet for download purposes. Here is a workaround, again assuming proper treatment of arguments, existence etc. A full working script can be found on our [github](https://github.com/csblab/PDBClean/blob/master/scripts/osftool.sh).\n", "\n", "```bash\n", "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n", "osfls='osf_ls.txt'\n", "if ! grep -q \"$remotedir\" $osfls; then\n", " echo \"$remotedir was not found in OSF...\"\n", "else\n", " grep \"$remotedir\" $osfls > \"tmp_${remotedir}.txt\"\n", " while read -r line\n", " do\n", " if [ ! -f ${localdir}/${line##*/} ]; then\n", " echo \">>> $line ...\"\n", " osf fetch $line ${localdir}/${line##*/}\n", " fi\n", " done < tmp_${remotedir}.txt\n", "fi\n", "echo \"----------\"; echo \"OSF CLIENT\"; echo \"----------\"\n", "```\n", "\n", "### Download whole project\n", "\n", "This one is easy: `$ osf clone`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }