-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Two or more crops for a single image #77
Comments
Hello, Can you clarify exactly what you are trying to do ? I'm sure we can find a way to work it out. The reason they are currently fused is because we are working with batches so all images have to have the same size as they exit the first operation of the pipeline |
For self-supervised learning you usually need 2 or more crops of the same image. With torchvision this is pretty easy as you can do something like wrapping the transformations with a custom class that calls the underlying transformations multiple times. For example
where base_transform is something like this
Can the same be accomplished with ffcv? |
I see, that's a very good point. We actually got a similar request. The
idea would be to have multiple pipelines use the same field name as source.
That would work for this use case right ?
…On Fri, Jan 21, 2022, 7:24 AM Victor Turrisi ***@***.***> wrote:
For self-supervised learning you usually need 2 or more crops of the same
image. With torchvision this is pretty easy as you can do something like
wrapping the transformations with a custom class that calls the underlying
transformations multiple times. For example
class TwoCropsTransform:
def __init__(self, base_transform):
self.base_transform = base_transform
def __call__(self, x):
q = self.base_transform(x)
k = self.base_transform(x)
return [q, k]
where base_transform is something like this
augmentation = transforms.Compose(
[
transforms.RandomResizedCrop(224, scale=(0.2, 1.0)),
transforms.RandomApply(
[transforms.ColorJitter(0.4, 0.4, 0.4, 0.1)], p=0.8 # not strengthened
),
transforms.RandomGrayscale(p=0.2),
transforms.RandomApply([moco.loader.GaussianBlur([0.1, 2.0])], p=0.5),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
normalize,
]
)
Can the same be accomplished with ffcv?
—
Reply to this email directly, view it on GitHub
<#77 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAPMOG36TWKQE4MDEED5PADUXFF6NANCNFSM5MPK6CLQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you commented.Message ID:
***@***.***>
|
Exactly, I think so. |
As a temporary solution you could extend the original decoder and have it
output images with double the height and have the two crop stiched
together. In your loop you could split them.
On Fri, Jan 21, 2022, 7:29 AM Guillaume Leclerc <
***@***.***> wrote:
… I see, that's a very good point. We actually got a similar request. The
idea would be to have multiple pipelines use the same field name as source.
That would work for this use case right ?
On Fri, Jan 21, 2022, 7:24 AM Victor Turrisi ***@***.***>
wrote:
> For self-supervised learning you usually need 2 or more crops of the same
> image. With torchvision this is pretty easy as you can do something like
> wrapping the transformations with a custom class that calls the underlying
> transformations multiple times. For example
>
> class TwoCropsTransform:
> def __init__(self, base_transform):
> self.base_transform = base_transform
> def __call__(self, x):
> q = self.base_transform(x)
> k = self.base_transform(x)
> return [q, k]
>
> where base_transform is something like this
>
> augmentation = transforms.Compose(
> [
> transforms.RandomResizedCrop(224, scale=(0.2, 1.0)),
> transforms.RandomApply(
> [transforms.ColorJitter(0.4, 0.4, 0.4, 0.1)], p=0.8 # not strengthened
> ),
> transforms.RandomGrayscale(p=0.2),
> transforms.RandomApply([moco.loader.GaussianBlur([0.1, 2.0])], p=0.5),
> transforms.RandomHorizontalFlip(),
> transforms.ToTensor(),
> normalize,
> ]
> )
>
> Can the same be accomplished with ffcv?
>
> —
> Reply to this email directly, view it on GitHub
> <#77 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAPMOG36TWKQE4MDEED5PADUXFF6NANCNFSM5MPK6CLQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
|
I see, but I still think this wouldn't solve it since the random parameters for the other augmentations are generated twice (one for each crop). If there's anything that I could help let me know. Looking forward to using ffcv :) |
Hi @vturrisi ! One thing that might work for now is to just create two |
Hey @andrewilyas, yeah, what you suggested will for sure work. However, wouldn't this be too slow since the images are loaded once for each crop? EDIT: I'll try it tomorrow and see how it scales with the number of crops just to be sure. |
Yeah this will be a little slower but I think it could still be fast enough to saturate most GPUs, especially if it's a simple 2-crop algorithm. Data on how it scales with # crops would be very much appreciated!! We'll keep you updated as we figure out how to support feeding the same field through multiple pipelines! |
There will also be an impact on ram usage right? I'll get to you tomorrow with some data regarding how it scales in time and memory. |
Definitely no impact with os caching |
I'm going to close this issue in favor of #82 which is about implementing the actual clean solution to what is faced by this issue. If you want to discuss the temporary solution we can reopen this one |
Had the same issue this afternoon; this is how I solved it: Make CIFAR10 return the same image twice: class DoubleCifar10Dataset(Dataset):
def __init__(self, root='data', train=True, download=True):
self.cifar10_dataset = torchvision.datasets.CIFAR10(root=root, train=train, download=download)
def __len__(self):
return len(self.cifar10_dataset)
def __getitem__(self, idx):
return self.cifar10_dataset[idx][0], self.cifar10_dataset[idx][0] Create the dataset train_data = utils.DoubleCifar10Dataset(root='data', train=True, download=True)
train_writer = DatasetWriter(f'tmp/cifar10_train_data.beton', {
'image1': RGBImageField(),
'image2': RGBImageField(),
})
train_writer.from_indexed_dataset(train_data) Create train loader:
Now you get two augmentations of the same image! |
@davidrzs thanks for sharing. I'm going the route of creating multiple loaders and wrapping them. I'll share once it's working 100%. |
Hello, I am trying to have 4 images out of pipeline. I saved them like :-
Then I am reading that like this :- ` image_pipeline: List[Operation] = [
But the output is coming as :- cuda:0 tensor(174., device='cuda:0', dtype=torch.float16) torch.Size([80, 3, 224, 224]) It seems the pipeline is not able to go beyond 3rd image for augmentation. Can you please let me know how to fix this? It is a bit urgent as I need to get my experiments running. Thanks a lot! |
@saarthak02 Can you create new issue ? |
Hi @davidrzs
But I found that the data loading is much slower than using a single image with one pipeline. Even worse. The loading is much slower than using PyTorch standard loader with more complicated transforms:
Do you have the same issue? |
Yeah it's completely expected. You are using pytorch augmentations. They are written in python, which means a single one can run at a given time per process. Pytorch spawns sub-processes which means they can run them in parallel, but it's also very inefficient because processes can't share memory and have to communicate with IPC instead of shared FFCV supports Pytorch augmentations just to experiment and see if an augmentation actually helps your model. You should never use them in situations where you care about performance. If you actually want speed you have to get rid of them. If you look at our examples we never use any Pytorch augmentation on the CPU. Two solutions:
|
(I created an issue #127 to issue a warning in this situation so that users don't expect good performance in this case) |
Thanks for your reply. train_loader = Loader(ffcv_cache_path,
batch_size=train_batch_size,
num_workers=num_workers,
order=OrderOption.RANDOM,
drop_last=True,
pipelines={'image0': image_pipeline,
'image1': image_pipeline1,
'attribute': label_pipeline})
|
Can I see your How is your CPU usage ? How are you storing your two images in your dataset ? Are you using RAW or JPEG ? How big are your images ? (I haven't yet sen a situation where FFCV is not at least 10x faster than Pytorch, so I still strongly suspect there is a misconfiguration somewhere in your code) |
(also @andrewilyas was suggesting |
Also: |
Also you seem to be CPU bottleneck. Why not having the |
I store the images by using the default fields. The images are from CelebA dataset of size (178, 218), stored in 'raw' format. writer = DatasetWriter(ffcv_cache_path, {
'image0': RGBImageField(),
'image1': RGBImageField(),
'attribute': NDArrayField(full_train_set.attr.dtype, (full_train_set.attr.shape[1],)),
}) I used 8 workers. Following your advice, here is some methods that works
Some methods that do not work
|
I you have an hyper-threaded processor. I would recommend using as many workers as you have physical cores on your processor (usually what htop shows / 2). And divide by 2 again when you use zip since you have two loaders. I don't think the attribute is causing any slowdown. I think the main bottleneck right now is image resizing and right now with your pipeline you are doing it twice. I'm currently working hard on the next release that will make the pipelines much more flexible this is how it will look like in your case: decoder = ffcv_trns.RandomResizedCrop(scale=(0.08, 1.0), ratio=np.array((3. / 4., 4. / 3.)), size=size)
pipeline_1 = [
ffcv_trns.RandomHorizontalFlip(),
ffcv_trns.ToTensor(),
ffcv_trns.Convert(torch.float32),
ffcv_trns.ToTorchImage(),
ffcv_trns.ToDevice(device, non_blocking=True),
]
pipeline_2 = [
ffcv_trns.RandomHorizontalFlip(),
ffcv_trns.ToTensor(),
ffcv_trns.Convert(torch.float32),
ffcv_trns.ToTorchImage(),
ffcv_trns.ToDevice(device, non_blocking=True),
]
train_loader = Loader(ffcv_cache_path,
batch_size=train_batch_size,
num_workers=num_workers,
order=OrderOption.RANDOM,
drop_last=True,
pipelines={'image': [decoder],
'image1': PipelineSpec(source=decoder, transforms=pipeline_1),
'image2': PipelineSpec(source=decoder, transforms=pipeline_1),
'attribute': label_pipeline}) And you would have a single |
Yeah, I had to do this trick to get performance parity with the old pipeline #64 (comment) . Though it is hacky! I have reverted to the old standard PyTorch pipeline as the speed was already decent enough, so I cannot give you more information. |
Hi there, I would also like to use ffcv dataloaders with SimCLR training. @GuillaumeLeclerc what is the current state of #82? I saw others commenting on going back to standard Pytorch dataloading due to speed issues. Have these been resolved, i.e. is SimCLR training with ffcv dataloading viable? |
Thanks for the suggestions in the thread. I was trying out another approach to generate two different augmentations of a single image in order to do Self-supervised learning, e.g. SimCLR. Although I used the PyTorch transforms in the image pipeline, I was able to see speedup using ffcv comapred to using the standard PyTorch dataloader. However, the final model accuracy was inferior compared to the one trained using the standard PyTorch dataloader. Additional details: The PyTorch dataloader uses the PyTorch files to load the images, as opposed to the ffcv dataloader that uses the beton files. Both pipelines use the same transforms (a rescaling was added in the ffcv pipeline because the Has anyone else faced this issue? |
@GuillaumeLeclerc it seems that the |
Hey! Thank you for the great work.
Is it possible to apply the same pipeline multiple times to the same image? From what I checked, this is currently not possible as it seems that the images are loaded and cropped within a single operation. Is there any way to go around this, loading the image only once and applying the augmentations pipelines multiple times?
The text was updated successfully, but these errors were encountered: