examples/s3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "52a0b1b5",
   "metadata": {},
   "source": [
    "# examples for `hostess.aws.s3`\n",
    "\n",
    "## introduction\n",
    "\n",
    "`hostess.aws.s3` is a collection of utilities for working with S3 \n",
    "objects. Its centerpiece is a class called `Bucket`, which offers\n",
    "a straightforward interface to a single S3 bucket.\n",
    "\n",
    "### capabilities\n",
    "\n",
    "`hostess.aws.s3` is designed as a streamlined alternative to `boto3`'s\n",
    "high-level S3 API. It is intended to make it easy to integrate S3 \n",
    "objects into Python workflows without writing a lot of boilerplate. \n",
    "It makes read/write operations extremely simple, including reads and \n",
    "writes from and to in-memory Python objects.\n",
    "\n",
    "In addition to simple I/O operations, it offers special functionality \n",
    "for several types of tasks that are frequently encountered when working \n",
    "with \"big data\" on S3, but are not straightforward to execute with \n",
    "existing tools, specifically:\n",
    "\n",
    "* building searchable indices of buckets containing many objects\n",
    "* modifying object storage classes\n",
    "* writing larger-than-memory or intermittently-streamed data\n",
    "  to S3 objects\n",
    "\n",
    "### limitations\n",
    "\n",
    "`hostess.aws.s3` does not provide interfaces for S3 administrative \n",
    "operations, including creating and deleting buckets or managing\n",
    "object and bucket permissions. If you need to perform automated S3 \n",
    "admin tasks of this type, you will need to supplement `hostess` with \n",
    "something such as `boto3`, `awscli`, the AWS Web Console. (`hostess.aws` \n",
    "does offer some generic utilities to make using `boto3` easier.) `hostess` \n",
    "may expand its S3 administrative capabilities in the future.\n",
    "\n",
    "### requirements\n",
    "\n",
    "1. If you have no valid AWS credentials, `hostess.aws` (like any other AWS \n",
    "interface) will not function. By default, `hostess` uses the 'default' \n",
    "profile from ~/.aws/credentials. This can be modified in \n",
    "`hostess.config.user_config` or by manually constructing a `boto3` session \n",
    "or client with `init_session` or `init_client` from `hostess.aws.utilities`.\n",
    "\n",
    "2. You need appropriate AWS permissions, both for the specific API call you're\n",
    "making and the specific S3 resources you wish to access. You cannot,\n",
    "for instance, use `Bucket.ls()` without the ListObjectsV2 permission, or\n",
    "`Bucket.put()` without the PutObject permission. \n",
    "\n",
    "A complete discussion of AWS credential and permissions management is beyond \n",
    "the scope of this document. For overviews, please refer to AWS documentation on \n",
    "[account creation](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-creating.html),\n",
    "[credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/security-creds.html),\n",
    "and [S3 permissions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html).\n",
    "\n",
    "### note and caveat\n",
    "This tutorial will walk you through the creation of an s3 bucket using your \n",
    "available AWS credentials and then put data into that bucket. This will start to\n",
    "incur costs attached to your AWS credentials. If this tutorial is run\n",
    "without modification, then the total costs should be approximately one fifth of \n",
    "a cent -- less than the electricity you are using to run your computer. However,\n",
    "if this is unacceptable to you or makes you uncomfortable,\n",
    "you should proceed with caution or not at all. We (the authors of `hostess`)\n",
    "disclaim responsibility. (See the appendix for a breakdown of estimated costs.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af9baf32-5327-4bd1-97b4-8bdb80785563",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## make an S3 bucket\n",
    "\n",
    "This isn't a showcase of `hostess`'s capabilities -- it just uses `boto3`\n",
    "and the Python Standard Library -- but it's a necessary \n",
    "precursor to the subsequent examples. If you have an existing empty bucket \n",
    "you'd like to use, you can skip this section, and if you've already run \n",
    "through this section before, you can reuse the bucket you made.\n",
    "\n",
    "AWS bucket names have to be unique within an AWS Partition (grouping of\n",
    "Regions), so we'll make a bucket with a random name. If you'd like to\n",
    "reuse this bucket later, make sure you note down or can easily look up \n",
    "the name (like in the AWS Web Console)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b19598f5-98c4-482b-acba-adf38fbb90f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import randint\n",
    "from hostess.aws.utilities import init_client\n",
    "\n",
    "bucket_name = f\"hostess-cats-{''.join(str(randint(0, 9)) for _ in range(9))}\"\n",
    "response = init_client('s3').create_bucket(Bucket=bucket_name)\n",
    "if response['ResponseMetadata']['HTTPStatusCode'] == 200:\n",
    "    print(\"We're ready to continue.\")\n",
    "    print(f\"The bucket name is {bucket_name}.\")\n",
    "else:\n",
    "    print(\"Something went wrong. Take a look at the response.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d65c5b73",
   "metadata": {},
   "source": [
    "## initialize a `Bucket`\n",
    "\n",
    "Unless you require special configuration, it's extremely straightforward\n",
    "to initialize a `hostess` `Bucket`: just pass the name of the bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0e92201",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize a Bucket object associated with our shiny new S3 bucket\n",
    "from hostess.aws.s3 import Bucket\n",
    "\n",
    "bucket = Bucket(bucket_name)\n",
    "\n",
    "# Verify that it's empty. We'll discuss `Bucket.ls()` more later.\n",
    "len(bucket.ls())==0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1243f79-efcf-435c-a36c-47438d0f5f46",
   "metadata": {},
   "source": [
    "## read/write operations on buckets\n",
    "\n",
    "There's not a lot you can do with an empty bucket, so let's take the \n",
    "only sensible step and turn our bucket into a repository of cat pictures.\n",
    "\n",
    "First, let's add a quick README file to the bucket to clarify our intent \n",
    "and demonstrate `Bucket`'s basic I/O capabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3200623e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compose helpful documentation and write it to a local file\n",
    "\n",
    "readme = \"\"\"# cat picture repository\n",
    "This bucket contains pictures of cats. Cats are\n",
    "small predatory mammals often kept as pets. Users who are unfamiliar\n",
    "with cats may wish to consult some \n",
    "[basic reference material](https://en.wikipedia.org/wiki/Cat) before\n",
    "browsing the bucket in order to avoid confusion.\n",
    "\"\"\"\n",
    "\n",
    "with open(\"DRAFT_README.md\", \"w\") as stream:\n",
    "    stream.write(readme)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbb55e52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# write the README to the bucket. The first two arguments of \n",
    "# `Bucket.put()` are a source and a destination. Here, we're using\n",
    "# a path to a local file (the file we just wrote) as the source.\n",
    "# The second argument is the object key you'd like to write your\n",
    "# data to. \"README.md\" seems fine in this case.\n",
    "bucket.put(\"DRAFT_README.md\", \"README.md\")\n",
    "\n",
    "# Now, load the README into memory and render it to make sure\n",
    "# our documentation made it to the bucket safely.\n",
    "from IPython.display import Markdown\n",
    "\n",
    "# Bucket.read() reads an S3 object directly into a Python object,\n",
    "# by default a string. pass mode=\"rb\" to read it as bytes instead.\n",
    "# If this command prints the text of our README as Markdown, then\n",
    "# the file has been successfully written to and read from the s3\n",
    "# bucket.\n",
    "Markdown(bucket.read(\"README.md\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1da3dd7b",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "## download cat pictures\n",
    "\n",
    "This isn't a special `hostess` feature, either, but we *do* need real content \n",
    "to put in our bucket, so let's grab some cat pictures from the Cat as \n",
    "a Service API. This should take about 20 seconds, depending\n",
    "on internet weather.\n",
    "\n",
    "This will write the cat pictures to an in-memory array object\n",
    "`catbytes`, not to your local filesystem.\n",
    "\n",
    "*Note: We don't recommend displaying the `catbytes`\n",
    "object in Jupyter; it'll be a **lot** of binary gibberish.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "761bc3d7-42f9-4003-96d1-ce8f8c547457",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "from io import BytesIO\n",
    "import requests\n",
    "from hostess.utilities import mb\n",
    "\n",
    "# feline binary array\n",
    "catbytes = []\n",
    "# number of cats per tag\n",
    "cats_per_tag = 3\n",
    "for adj in (\"cute\", \"angry\", \"white\", \"black\", \"tabby\"):\n",
    "    print(f\"fetching pictures of {adj} cats...\")\n",
    "    for i in range(cats_per_tag):\n",
    "        # get data for a cat picture\n",
    "        resp = requests.get(f\"https://cataas.com/cat/{adj}\")\n",
    "        # parse mimetype so we know the image format (jpeg, gif, &c)\n",
    "        ftype = resp.headers['Content-Type'].replace('image/', '')\n",
    "        # save data & metadata in our list\n",
    "        catbytes.append(\n",
    "            {'data': resp.content, \"prefix\": adj, 'fn': f\"cat_{i}.{ftype}\"}\n",
    "        )\n",
    "        print(f\"{i + 1}/{cats_per_tag}...\", end=\"\")\n",
    "    print(\"\")\n",
    "print(\"... done.\")\n",
    "# integrity check\n",
    "assert all(isinstance(c['data'], bytes) for c in catbytes)\n",
    "print(\"All cat pictures are Python bytes objects,\", end=\" \")\n",
    "# summary info\n",
    "print(\n",
    "    f\"{mb(sum(len(c['data']) for c in catbytes))} MB total volume.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be1e58c2-b48c-4c41-a2f1-5c85243711d6",
   "metadata": {},
   "source": [
    "## write cat pictures to the bucket\n",
    "\n",
    "You saw a moment ago that `hostess.aws.s3.Bucket` can write local files to\n",
    "S3 objects. It can do the same with in-memory objects.\n",
    "\n",
    "### serial uploads\n",
    "\n",
    "`Bucket` supports both **serial** and **parallel** forms of most basic I/O\n",
    "operations, including `put()`. Passing a single source and destination, as\n",
    "follows, makes `Bucket` operate in serial mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fae82d5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# put the cat pictures in the bucket one by one\n",
    "for c in catbytes:\n",
    "    bucket.put(c['data'], f\"{c['prefix']}/{c['fn']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1cb2d1c",
   "metadata": {},
   "source": [
    "### parallel uploads\n",
    "\n",
    "Passing sequences of sources and destinations makes `Bucket` operate\n",
    "in parallel mode. It performs the API calls in multiple threads and\n",
    "returns all the results in a `list` (even if those results are `None`,\n",
    "like for `put()`). AWS encourages the use of this type of \n",
    "['horizontal scaling'](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/horizontal-scaling-and-request-parallelization-for-high-throughput.html),\n",
    "so it's typically a good option if you know everything you need to\n",
    "upload/download/etc. from the outset. This should be about twice as\n",
    "fast as the serial operation in the previous cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfd8c4b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# 'flattened' version of the for loop from the earlier cell\n",
    "cat_data = [c['data'] for c in catbytes]\n",
    "cat_keys = [f\"{c['prefix']}/{c['fn']}\" for c in catbytes]\n",
    "results = bucket.put(cat_data, cat_keys)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d323152",
   "metadata": {},
   "source": [
    "#### a note on supported types\n",
    "\n",
    "`Bucket.put()` supports `bytes`, `str`, `StringIO`, and\n",
    "`BytesIO`. If you want to write a `str`, pass `literal_str=True` so that \n",
    "`Bucket` doesn't interpret it as the path to a local file. All our cat\n",
    "pictures were already `bytes`, so we didn't need to do any preprocessing.\n",
    "\n",
    "#### a note on thread count\n",
    "\n",
    "By default, `Bucket` uses 4 threads for auto-threaded operations.\n",
    "You can change this by passing the `n_threads` \n",
    "argument to `Bucket`, or setting `Bucket.n_threads` after initialization.\n",
    "Setting `n_threads` to `None` turns off auto-threading.\n",
    "A different number may be better depending on network speed, latency,\n",
    "object size, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be8a3279",
   "metadata": {},
   "source": [
    "### indexing our cat pictures\n",
    "\n",
    "Now that we've populated our bucket, we can build an index for it\n",
    "to ensure our uploads worked as we expected and help users find the\n",
    "specific cat pics they need.\n",
    "\n",
    "`Bucket` offers a wealth of options for exploring buckets, and one\n",
    "of its most powerful features is its `df()` method.\n",
    "If you access it before populating a `Bucket`'s contents, it \n",
    "immediately indexes the entirety of the bucket's contents, cache the\n",
    "results in `Bucket.index`, and then returns them as a pandas DataFrame.\n",
    "Subsequent calls to `df()` will use these cached contents rather\n",
    "than performing the full indexing process again. (If you _need_ to\n",
    "re-index, call `Bucket.update_contents()` first.)\n",
    "\n",
    "*CAUTION: don't call this casually on really, really, big buckets\n",
    "(containing e.g. millions of objects or more),\n",
    "especially if you don't care about their entire contents -- it will \n",
    "take a long time to run, and in the worst cases, can even create a \n",
    "larger-than-memory index. See below for how to index buckets\n",
    "in a more controlled way.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfc22858",
   "metadata": {},
   "outputs": [],
   "source": [
    "# look at that wealth of content...\n",
    "index = bucket.df()\n",
    "index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1bcce2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for larger buckets, having immediate access to all of pandas's affordances\n",
    "# can be a lifesaver. say you wanted to know the total size of all the JPEGs\n",
    "# in the bucket:\n",
    "jpeg_mb = mb(index.loc[index['Key'].str.endswith('jpeg'), 'Size'].sum())\n",
    "print(f\"There are {jpeg_mb} MB of cat JPEGs.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "658f43b9",
   "metadata": {},
   "source": [
    "## reading cat pictures from the bucket\n",
    "\n",
    "Now that we've got an index of cat pictures, we can easily download them\n",
    "in parallel, either into memory or to disk. Let's try it both ways, \n",
    "first into memory.\n",
    "\n",
    "#### a note on `Bucket.get()`'s signature\n",
    "\n",
    "By default, `Bucket.get()` reads S3 objects into memory as \n",
    "Python BytesIO objects. If you want that behavior, which we\n",
    "do here, you don't have to explicitly pass a list as the \n",
    "second argument of `get()` to use it in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "685bf463",
   "metadata": {},
   "outputs": [],
   "source": [
    "# read to in-memory objects\n",
    "from random import choice\n",
    "from PIL import Image\n",
    "\n",
    "# just grab the cat pictures, not the README\n",
    "pic_keys = index.loc[index['Key'].str.contains('cat'), 'Key']\n",
    "catbuffers = bucket.get(pic_keys)\n",
    "# the moment of truth for our round-trip operation...\n",
    "print(f\"We uploaded {len(catbytes)} files and got {len(catbuffers)} objects back.\")\n",
    "Image.open(choice(catbuffers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac4f2cdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# write the cat pics into a subfolder of your working directory \n",
    "# named 's3_cat_mirror.' `Bucket.get()` will automatically \n",
    "# create the necessary directory structure to mirror the \n",
    "# organization of the s3 bucket, treating object prefixes like\n",
    "# the names of directories. (See notes on terminology in the appendix.)\n",
    "# In this case, there will\n",
    "# end up being directories for each adjectival category of cat.\n",
    "results = bucket.get(pic_keys, \"cat_s3_mirror/\" + pic_keys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17ba8ccb-aeda-4c20-968c-85819f92b4b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's do a completeness check to make sure that we\n",
    "# actually got all of the cat pictures\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "from hostess.directory import index_breadth_first, make_treeframe\n",
    "\n",
    "# we should have the same total set of pictures. let's compare names\n",
    "# and sizes.\n",
    "local_index = pd.DataFrame(index_breadth_first(\"cat_s3_mirror\"))\n",
    "# strip the local subdirectory name:\n",
    "local_index['path'] = local_index['path'].str.replace('cat_s3_mirror/', '')\n",
    "for _, local in local_index.iterrows():\n",
    "    # don't care about directories\n",
    "    if local['directory'] is True:\n",
    "        continue\n",
    "    remote = index.loc[index['Key'].str.endswith(local['path'])]\n",
    "    assert len(remote) == 1, \"missing? duplicates? not great.\"\n",
    "    remote = remote.iloc[0]\n",
    "    assert mb(remote['Size'], 3) == local['size']\n",
    "print(\"everything matches!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98044a78",
   "metadata": {},
   "outputs": [],
   "source": [
    "# and we can do a direct visual comparison to make sure. this cell\n",
    "# and the next cell should display the same picture when you run them,\n",
    "# because they are attempting to open and display the same file\n",
    "# from your local file system and the remote s3 bucket.\n",
    "Image.open(f\"cat_s3_mirror/{local['path']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9fabb2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "Image.open(bucket.get(local['path']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe9a0eab-5323-443b-99f8-96c441633e09",
   "metadata": {},
   "source": [
    "## cleanup\n",
    "\n",
    "Now that we're done, let's go ahead and delete all the objects in \n",
    "the bucket, and the bucket itself (you can't delete a bucket with \n",
    "anything in it)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0375ffef-adb4-4e07-a023-6a61111bc9c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete all objects\n",
    "_ = bucket.rm(bucket.df()['Key'])\n",
    "\n",
    "# is it empty?\n",
    "bucket.update_contents()\n",
    "assert len(bucket.df()) == 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3938950-36f1-465b-9054-899e024d4e9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete the bucket\n",
    "response = init_client('s3').delete_bucket(Bucket=bucket_name)\n",
    "if response['ResponseMetadata']['HTTPStatusCode'] == 204:\n",
    "    print(\"Cleanup complete. Bucket deleted.\")\n",
    "else:\n",
    "    print(\"Something went wrong. Take a look at the response.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca5fe634-a4d4-4c37-a32e-110fb081282c",
   "metadata": {},
   "source": [
    "## appendix\n",
    "\n",
    "### general s3 notes and vocab\n",
    "\n",
    "Here are some useful notes about S3. These notes aren't specific to \n",
    "`hostess`. They're statements about S3 in general.\n",
    "\n",
    "S3 objects are designed to look a lot like files, S3 buckets\n",
    "are designed to look a lot like filesystems, and in many cases, they \n",
    "can be used like files and filesystems. They aren't, though,\n",
    "and we prefer to use precise vocabulary to emphasize the differences.\n",
    "\n",
    "The full name of an S3 object is called its **_object key name_** or just **_key_**. \n",
    "This includes its full 'path' in addition to its 'file name'. The components of the\n",
    "'path' prior to the 'file name' are called **_prefixes_**. Prefixes are loosely\n",
    "analogous to directories in a filesystem, and interfaces often\n",
    "display them as if they were. However, prefixes are not actually directories,\n",
    "and a key is not actually a path. Directories are a type of file, but \n",
    "prefixes are just \n",
    "[\"a string of characters at the beginning of the object key name\"](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html) that can be used to help organize objects and permissions. \n",
    "They cannot be renamed/moved, given AWS resource tags, or otherwise directly manipulated.\n",
    "\n",
    "Similarly, unlike files, S3 objects also cannot be renamed / moved. \"Moving\" an\n",
    "S3 object always means making a complete copy of it with a different key\n",
    "and subsequently deleting the original, even if the new copy is in the \n",
    "same \"filesystem\" (bucket). It is no slower to copy an object\n",
    "from one bucket to another than to copy an object within the same\n",
    "bucket (unless the buckets are in different AWS Regions).\n",
    "\n",
    "Finally, unlike files, S3 objects are immutable. Although it is possible to read\n",
    "just a portion of an S3 object, it is not possible to perform incremental\n",
    "writes to an S3 object. Anything that looks like a tail-write to an S3\n",
    "object actually overwrites the whole object with a slightly modified\n",
    "copy of itself. This tends to be horribly inefficient. Multipart uploads\n",
    "provide the closest approximation to 'real' tail-writes, and they don't\n",
    "actually create accessible objects until they're completed.\n",
    "\n",
    "### a diagnostic tip\n",
    "\n",
    "If you are having trouble accessing an S3 resource via `hostess`, you can\n",
    "quickly rule out some basic possibilities by calling `hostess.aws.utilities.whoami()`. \n",
    "This function performs an API call that is *always* available to any AWS account, \n",
    "even if an administrator specifically tries to deny it \n",
    "([STS GetCallerIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html)).\n",
    "If it fails, either there is something wrong with your account credentials, or some network \n",
    "issue is preventing you from accessing an AWS API endpoint. If it doesn't, the problem\n",
    "is with access to the resource or API action. Permissions might be wrong, the\n",
    "resource might not exist, it might be in a different region than expected, or your\n",
    "network is specifically blocking access to that resource.\n",
    "\n",
    "### `hostess.aws.s3`'s relationship to `boto3` (advanced topic)\n",
    "\n",
    "`hostess.aws.s3` works primarily as a high-level interface to `boto3`'s low-level S3 API. Due to philosophical differences, it makes almost no use of the `boto3` high-level `Bucket` object, and should be considered a reimagining of that object rather than a wrapper for it. `hostess` `Buckets` do not automatically instantiate or grant access to `boto3` `Buckets`. However, they *do* wrap `boto3` client, resource, and session objects. If users require access to portions of the S3 API not included in `hostess`, they may reference these attributes of a `hostess` `Bucket` in order to perform API calls using the same session/client/resource that underlies the methods of their `hostess` `Bucket`. They may even use them to instantiate a similarly-configured `boto3` `Bucket`, if they would like things to get confusing.\n",
    "\n",
    "### `Bucket.df()` column specification\n",
    "\n",
    "* Key: object key (`object`)\n",
    "* LastModified: last modified time (`datetime64[ns, tzutc()]`)\n",
    "* ETag: S3-generated object tag. Usually the md5 hash, but this is not guaranteed. (`object`)\n",
    "* Size: object size in bytes (`int64`)\n",
    "* StorageClass: [storage class, e.g. DEEP_ARCHIVE or STANDARD](https://aws.amazon.com/s3/storage-classes/)\n",
    "    (`object`)\n",
    "\n",
    "\n",
    "### breakdown of estimated costs for this Notebook \n",
    "\n",
    "Running this Notebook as-is should incur approximately\n",
    "a fifth of a cent in AWS costs.\n",
    "\n",
    "* This estimate assumes that you take an extended break in the middle\n",
    "  of running this Notebook and retain the cat pictures in S3 for 4 hours.\n",
    "* This Notebook also makes DELETE requests, but DELETEs are free. The same is true of data transfer IN.\n",
    "* Estimated 6 MB of cat pictures; the actual amount may\n",
    "  vary by 1-3 MB.\n",
    "* All values in USD.\n",
    "\n",
    "#### S3 Standard timed storage\n",
    "\n",
    "* 6 MB @ 0.023 / GB-month * 4 hours  = 0.000000756\n",
    "\n",
    "#### data out\n",
    "\n",
    "* 20 MB @ 0.09 / GB = 0.0018\n",
    "\n",
    "#### API requests\n",
    "\n",
    "**S3 Standard tier 1**\n",
    "\n",
    "*TODO: update if we add additional ls() examples*\n",
    "\n",
    "* 31 PUT\n",
    "* 2 LIST\n",
    "* 33 total @ 0.005 / 1000 calls = 0.000165\n",
    "\n",
    "**S3 Standard tier 2**\n",
    "\n",
    "* 32 GET\n",
    "* 32 total @ 0.0004 / 1000 calls = 0.0000128\n",
    "\n",
    "**TOTAL: ~0.002 USD**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}