Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into photon_classifier
- Loading branch information
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "a888b10c-1f9f-406e-8d14-9648be234d44", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Cloud Computing Tutorial\n", | ||
| "\n", | ||
| "<br />\n", | ||
| "\n", | ||
| "```{image} ./images/cloud.gif\n", | ||
| ":width: 200px\n", | ||
| ":align: center\n", | ||
| "```\n", | ||
| "\n", | ||
| "**Welcome to the Cloud Computing Tutorial!**\n", | ||
| "\n", | ||
| "This tutorial is just the tip of the ice[SAT-2]berg (😬) of cloud computing. It focuses on accessing data stored in the cloud. An understanding of the difference between the \"download to local\" and \"direct from cloud\" methods of data access will explain how and why the cloud facilitates the scaling and reproducibility of your science.\n", | ||
| "\n", | ||
| ":::{admonition} Learning Goals\n", | ||
| "\n", | ||
| "**At the conclusion of this tutorial, you should be able to answer:**\n", | ||
| "1. What is cloud computing?\n", | ||
| "2. What is cloud object storage and the difference between data stored in the cloud, data on a local file system and data stored in \"on-premise\" data centers.\n", | ||
| "3. How to optimize data for reading from cloud object storage.\n", | ||
| "\n", | ||
| ":::\n", | ||
| "\n", | ||
| "## Outline\n", | ||
| "\n", | ||
| "1. [What is cloud computing?](./01-cloud-computing.ipynb)\n", | ||
| " 1. Definition of cloud computing\n", | ||
| " 2. Exercise: Difference between resources on your local machine and resources in the cloud\n", | ||
| " 3. Why you might use cloud computing\n", | ||
| "2. [Accessing data in the cloud](./02-cloud-data-access.ipynb)\n", | ||
| " 1. Definition of cloud object storage\n", | ||
| " 2. Exercise: How many NASA datasets (aka collections) are in the cloud? How many ICESat-2 datasets are in the cloud? Which DAAC manages ICESast-2 data?\n", | ||
| " 3. Difference between data stored in the cloud, data on a local file system and data stored in \"on-premise\" data centers\n", | ||
| " 4. Why you might use cloud object storage\n", | ||
| "3. [Cloud-Optimized Data](./03-cloud-optimized-data-access.ipynb)\n", | ||
| " 1. What are we optimizing for and why?\n", | ||
| " 2. Anatomy of a structured data file\n", | ||
| " 3. Thought Exercise: Garage analogy\n", | ||
| " 4. How do we optimize data for reading from cloud object storage?\n", | ||
| "4. [Cloud-Optimized ICESat-2 Demo](./04-cloud-optimized-icesat2.ipynb)\n", | ||
| " 1. Cloud-Optimized vs Cloud-Native \n", | ||
| " 1. Creating an ICESat-2 GeoParquet\n", | ||
| " 3. Plot the data with lonboard\n", | ||
| "\n", | ||
| "Or simply: Cloud -> Cloud data access -> Optimized cloud data access -> Demo with ICESat-2" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.9" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "attachments": {}, | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# What is cloud computing?\n", | ||
| "\n", | ||
| "<br />\n", | ||
| "\n", | ||
| "**Cloud computing is compute and storage as a service.** The term \"cloud computing\" is typically used to refer to commercial cloud service providers such as Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure (Azure). These cloud service providers all offer a wide range of computing services, only a few of which we will cover today, via a pay-as-you-go payment structure.\n", | ||
| "\n", | ||
| "```{image} ./images/AWS_OurDataCenters_Background.jpg\n", | ||
| ":width: 600px\n", | ||
| ":align: center\n", | ||
| "```\n", | ||
| "\n", | ||
| "<p style=\"font-size: 10px;\">image src: https://aws.amazon.com/compliance/data-center/data-centers/</p>\n", | ||
| "\n", | ||
| ">Cloud computing is the on-demand delivery of IT resources over the Internet with pay-as-you-go pricing. Instead of buying, owning, and maintaining physical data centers and servers, you can access technology services, such as computing power, storage, and databases, on an as-needed basis from a cloud provider like Amazon Web Services (AWS). ([source](https://aws.amazon.com/what-is-cloud-computing/))\n", | ||
| "\n", | ||
| "This tutorial will focus on AWS services and terminology, but Google Cloud and Microsoft Azure offer the same services.\n", | ||
| "\n", | ||
| ":::{dropdown} 🏋️ Exercise: How many CPUs and how much memory does your laptop have? And how does that compare with CryoCloud?</h3>\n", | ||
| ":open:\n", | ||
| "If you have your laptop available, open the terminal app and use the appropriate commands to determine CPU and memory.\n", | ||
| "\n", | ||
| "<div style=\"width:60%; padding: 30px;\">\n", | ||
| "\n", | ||
| "| Operating System (OS) | CPU command | Memory Command |\n", | ||
| "|-----------------------|-----------------------------------------------------------------------------------|----------------------------|\n", | ||
| "| MacOS | `sysctl -a \\| grep hw.ncpu` | `top -l 1 \\| grep PhysMem` |\n", | ||
| "| Linux (cryocloud) | `lscpu \\| grep \"^CPU\\(s\\):\"` | `free -h` | \n", | ||
| "| Windows | https://www.top-password.com/blog/find-number-of-cores-in-your-cpu-on-windows-10/ | |\n", | ||
| "</div>\n", | ||
| "\n", | ||
| "Now do the same but on hub.cryointhecloud.com.\n", | ||
| "\n", | ||
| "Tip: When logged into cryocloud, you can click the  icon on the far-right toolbar.\n", | ||
| ":::\n", | ||
| "\n", | ||
| "**What did you find?** It's possible you found that your machine has **more** CPU and/or memory than cryocloud!\n", | ||
| "\n", | ||
| ":::{dropdown} So why would we want to use the cloud and not our personal computers?\n", | ||
| " 1. Because cryocloud has all the dependencies you need.\n", | ||
| " 2. Because cryocloud is \"close\" to the data (more on this later).\n", | ||
| " 3. Because you can use larger and bigger machines in the cloud (more on this later).\n", | ||
| " 4. **Having the dependencies, data, and runtime environment in the cloud can simplify reproducible science.**\n", | ||
| ":::\n", | ||
| "\n", | ||
| ":::{admonition} Takeaways\n", | ||
| "\n", | ||
| "* The cloud allows you to access many computing and storage services over the internet. Most cloud services are offered via a \"pay as you go\" model.\n", | ||
| "* Hubs like CryoCloud provide a virtual environment which simplifies reproducible science. You should use them whenever you can!\n", | ||
| ":::" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.9" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 4 | ||
| } |
Oops, something went wrong.