Skip to content

Commit

Permalink
update transforms chapter
Browse files Browse the repository at this point in the history
  • Loading branch information
ncullen93 committed May 30, 2024
1 parent a1acf5c commit 78a55ff
Showing 1 changed file with 273 additions and 20 deletions.
293 changes: 273 additions & 20 deletions book/02-04.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@
"source": [
"# 8. Fixed transforms\n",
"\n",
":::{warning}\n",
"This chapter is an unfinished draft. I recommend skipping it for now since it is likely to change.\n",
":::{note}\n",
"You are viewing a working draft with expected completion in early 2025.\n",
":::\n",
"\n",
"This section until this point has been focused on mapping collections of images and meta-data in whatever format is needed and wherever they may be. However, we have always assumed until now that the images returned from the dataset are in perfect condition and harmonized with each other. \n",
"The current section until this point has been focused on mapping collections of images and meta-data in whatever format is needed and wherever they may be. However, we have always assumed until now that the images returned from the dataset are in \"perfect condition\" and harmonized within each other. \n",
"\n",
"Unfortunately, that is rarely the case. We often work with images that are a variety of imperfections that we must correct before reading them in from their raw form. This can include images have different shapes or orientations from one other, having unnormalized intensity values with outliers, or simply not being in the format needed for our AI models.\n",
"Unfortunately, that is rarely the case. We often work with images that have a variety of imperfections which we must correct after reading them in from their raw form. This can include images having different shapes or orientations from one other, having unnormalized intensity values with outliers, or label images not being in the one-hot format needed for our AI models.\n",
"\n",
"You can call these types of preprocessing steps as \"fixed transforms\". They are fixed because they are deterministic and will give the same result regardless of how many times they're applied to the image. "
"You can call these types of preprocessing steps as \"fixed transforms\". They are fixed because they are deterministic and will give the same result regardless of how many times they're applied to the image. As you'll learn here, these fixed transforms should be applied when mapping datasets."
]
},
{
Expand All @@ -23,17 +23,17 @@
"source": [
"## Prerequisites\n",
"\n",
"In this chapter, we will specifically use the readers and transforms modules from nitrain along with ants and numpy to create images. We will only work with images already loaded in memory in this chapter, but keep in mind that there would be no difference if you use other types of readers."
"In this chapter, we will specifically use the transforms modules from nitrain along with ants and numpy to create images. We will only work with images already loaded in memory in this chapter, but keep in mind that there would be no difference if you use other types of readers."
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"import nitrain as nt\n",
"from nitrain import readers, transforms as tx\n",
"from nitrain import transforms as tx\n",
"import ants\n",
"import numpy as np"
]
Expand All @@ -44,25 +44,142 @@
"source": [
"## Individual transforms\n",
"\n",
"We will start by learning how to apply transforms to images individually."
"We will start by learning how to apply transforms to images individually. Let's start with a basic classification example where all of the images in our dataset are of different shape. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Image shapes:\n",
"[(40, 44), (44, 44), (49, 49), (45, 48), (49, 47), (48, 44), (43, 44), (42, 41), (41, 41), (49, 46)]\n"
]
}
],
"source": [
"# create data in memory\n",
"images = [ants.from_numpy(np.ones((np.random.randint(40,50),np.random.randint(40,50)))) * i for i in range(10)]\n",
"labels = [i for i in range(10)]\n",
"\n",
"print('Image shapes:')\n",
"print([image.shape for image in images])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Composing transforms\n",
"In that case, we may want to resample them to a common shape in order to eventually generate batches with them. There is a transform called `Resample` that takes care of this perfectly. Here is how you would apply it in the context of a dataset."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[(40, 40), (40, 40), (40, 40), (40, 40), (40, 40), (40, 40), (40, 40), (40, 40), (40, 40), (40, 40)]\n"
]
}
],
"source": [
"dataset = nt.Dataset(inputs=images, \n",
" outputs=labels,\n",
" transforms={\n",
" 'inputs': tx.Resample((40,40), use_voxels=True)\n",
" })\n",
"\n",
"If you want to apply multiple transforms in a row, then you can compose them together in a list. It is good practice to compose transforms because it can be faster and save memory."
"print([x.shape for x, y in dataset])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Labels\n",
"As we see, the `transforms` argument is a dictionary where the key is a label telling the dataset which value to apply the transform to, and the value is an initialized transform. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Multiple transforms\n",
"\n",
"Because the `transforms` argument is a dictionary, we can apply transforms to both the inputs and the outputs. A clear example of this would be when both your inputs and outputs are images."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(40, 40) | (40, 40)\n"
]
}
],
"source": [
"dataset = nt.Dataset(inputs=images, \n",
" outputs=images,\n",
" transforms={\n",
" 'inputs': tx.Resample((40,40), use_voxels=True),\n",
" 'outputs': tx.Resample((40,40), use_voxels=True)\n",
" })\n",
"\n",
"In the case where you have multiple readers as inputs or outputs, you may want to apply a transform to only one of the readers. This is possible by using (or at least understanding) the way readers are labelled."
"x, y = dataset[0]\n",
"print(x.shape, '|', y.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Composing transforms\n",
"\n",
"Rarely will we apply only one transform to an image. Thankfully, composing transforms in nitrain is as simple as combining them in lists. Here is an example where we resample the image and then take the square root of the image."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Shape: (40, 40) | Mean: 2.0\n"
]
}
],
"source": [
"dataset = nt.Dataset(inputs=images, \n",
" outputs=labels,\n",
" transforms={\n",
" 'inputs': [tx.Resample((40,40), use_voxels=True),\n",
" tx.Sqrt()]\n",
" })\n",
"\n",
"x, y = dataset[4]\n",
"print('Shape:', x.shape, ' | Mean: ', x.mean())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When transforms are composed in lists, they will be applied in the order given. This is useful for when certain transforms require some other preprocessing to occur before working correctly."
]
},
{
Expand All @@ -71,33 +188,169 @@
"source": [
"## Multi-image transforms\n",
"\n",
"With nitrain datasets, it's also possible to apply transforms to multiple readers together. This can be useful if you have transforms that rely on a \"reference\" image, such as registering all images to a target template."
"Until now, you've seen how to apply transforms to individual images. But there are certain transforms that require multiple images or simply work more efficiently. You can apply transforms to multiple images together by having the transforms dictionary key be a tuple of the value labels. \n",
"\n",
"The example with resampling the images as inputs and outputs would therefore more appropriately look like this."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(40, 40) | (40, 40)\n"
]
}
],
"source": [
"dataset = nt.Dataset(inputs=images, \n",
" outputs=images,\n",
" transforms={\n",
" ('inputs', 'outputs'): tx.Resample((40,40), use_voxels=True),\n",
" })\n",
"\n",
"x, y = dataset[0]\n",
"print(x.shape, '|', y.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Custom transforms\n",
"This concept will be particularly important later on in the book when learning about random, augmenting transforms. Such random transforms have to be applied together in many cases to ensure the alignment between, say, an image and its segmentation is maintained."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Adding labels\n",
"\n",
"We've only used the keys `inputs` and `outputs` in our transforms dictionary. But in the case of multiple readers as inputs or outputs, you may want to apply a transform to only one of the readers. This is possible by using (or at least understanding) the way readers are labelled. \n",
"\n",
"For example, if you have two inputs then there are three available labels: \"inputs\", \"inputs-0\", and \"inputs-1\". Supplying a transform with \"inputs\" as key will apply the transform to both inputs, while \"inputs-0\" or \"inputs-1\" as key will apply the transform to only the first or second input, respectively. Here is an example:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input 1 shape: (40, 40) | Input 2 shape: (50, 50)\n"
]
}
],
"source": [
"# resample each input differently\n",
"dataset = nt.Dataset(inputs=[images, images], \n",
" outputs=images,\n",
" transforms={\n",
" 'inputs-0': tx.Resample((40,40), use_voxels=True),\n",
" 'inputs-1': tx.Resample((50,50), use_voxels=True),\n",
" })\n",
"\n",
"If you don't see the transform that you need, you can use the `tx.CustomFunction` transform to pass in a user-defined function to be applied as a transform."
"x, y = dataset[0]\n",
"\n",
"print('Input 1 shape: ', x[0].shape, ' | Input 2 shape:', x[1].shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"However, it is possible to use custom labels by passing in your inputs as a dictionary instead of a list, where the labels of each input are the key."
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"first-image: (40, 40) | second-image: (50, 50)\n"
]
}
],
"source": [
"# resample each input differently\n",
"dataset = nt.Dataset(inputs={'first-image': images, \n",
" 'second-image': images}, \n",
" outputs=images,\n",
" transforms={\n",
" 'first-image': tx.Resample((40,40), use_voxels=True),\n",
" 'second-image': tx.Resample((50,50), use_voxels=True),\n",
" })\n",
"\n",
"In this chapter, you learned about applying fixed transforms to your datasets. With nitrain transforms, it's possible to apply transforms in arbitrary order and to multiple images at once.\n",
"x, y = dataset[0]\n",
"\n",
"It's important to identify deterministic, preprocessing transforms and distinguish them from random, augmentation transforms (which you'll learn about later) because we can cache the preprocessed images to speed things up. "
"print('first-image: ', x[0].shape, ' | second-image:', x[1].shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": []
"source": [
"This allows you to supply custom labels to your inputs and outputs that better represent what the values actually are. Nonetheless, the authomatic labels are always available."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Custom functions\n",
"\n",
"The nitrain library has a large collection of transforms and we are happy to consider adding more. But if you don't see the transform that you need, or if you want to do something unique to your images, you can use the `tx.CustomFunction` transform to pass in a user-defined function to be applied as a transform."
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"class-0\n"
]
}
],
"source": [
"def make_string(value):\n",
" return f'class-{value}'\n",
"\n",
"# resample each input differently\n",
"dataset = nt.Dataset(inputs=images,\n",
" outputs=labels,\n",
" transforms={\n",
" 'outputs': tx.CustomFunction(make_string)\n",
" })\n",
"\n",
"x, y = dataset[0]\n",
"print(y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"In this chapter, you learned about applying fixed transforms to your datasets. With nitrain transforms, it's possible to apply transforms in arbitrary order and to multiple images at once. It's important to identify deterministic, preprocessing transforms and distinguish them from random, augmentation transforms (which you'll learn about later) because we can cache the preprocessed images to speed things up. \n",
"\n",
"This chapter focused on how to apply transforms and did not give a full rundown of all the available transforms. To understand which transforms are available in nitrain, you should check out the [transforms](www.github.com/nitrain/nitrain/transforms) folder on the GitHub page."
]
}
],
"metadata": {
Expand Down

0 comments on commit 78a55ff

Please sign in to comment.