Fine tuning with newer dataset

[NisaHameed]

Hi

I am a student trying to learn new things on SAM on medical images. Currently, I am working on using the MesSAM model over Kvasir polyp dataset. It does not have image embeddings. How do I generate? May I use my own code to extract the bounding box from the images? Also kindly help me with some tips to proceed with the work.

Thank you.

[palonza]

Hi @NisaHameed:

To generate image embeddings for the Kvasir polyp dataset and use them with the MedSAM model, you would typically follow these steps (very summarized): 1) Pretrained Models 2) Feature Extraction 3) Bounding Box Extraction 4) Embedding Generation.

Here an example of image embedding generation of how you can use a pre-trained CNN model (e.g., VGG16) to extract image embeddings using PyTorch:

import torch
import torchvision.models as models
import torchvision.transforms as transforms
from PIL import Image

# Load pre-trained VGG16 model
model = models.vgg16(pretrained=True)
model.eval()

# Preprocess image and extract features
def extract_image_features(image_path):
    image = Image.open(image_path)
    preprocess = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])
    image = preprocess(image).unsqueeze(0)
    with torch.no_grad():
        features = model(image)
    return features

# Usage
image_path = 'path/to/your/image.jpg'
image_features = extract_image_features(image_path)

Here an example of Bounding Box Extraction of how you can use pre-trained object detection model (e.g., Faster R-CNN) to extract bounding boxes from the images using the torchvision library:

import torch
import torchvision.models as models
import torchvision.transforms as transforms
from PIL import Image

# Load pre-trained Faster R-CNN model
model = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()

# Preprocess image and extract bounding boxes
def extract_bounding_boxes(image_path):
    image = Image.open(image_path)
    preprocess = transforms.Compose([
        transforms.ToTensor(),
    ])
    image_tensor = preprocess(image)
    image_tensor = image_tensor.unsqueeze(0)
    
    with torch.no_grad():
        predictions = model(image_tensor)
        
    boxes = predictions[0]['boxes']
    # Process boxes further if needed
    
    return boxes

# Usage
image_path = 'path/to/your/image.jpg'
bounding_boxes = extract_bounding_boxes(image_path)

[NisaHameed]

Great. Thank you very much. Let me certainly try to work on it.

…

On Sun, 28 May 2023, 7:01 am Jоsé P. Alоnza, ***@***.***> wrote: Hi @NisaHameed <https://github.com/NisaHameed>: To generate image embeddings for the Kvasir polyp dataset and use them with the MedSAM model, you would typically follow these steps (very summarized): 1) Pretrained Models 2) Feature Extraction 3) Bounding Box Extraction 4) Embedding Generation. Here an example of image embedding generation of how you can use a pre-trained CNN model (e.g., VGG16) to extract image embeddings using PyTorch: import torchimport torchvision.models as modelsimport torchvision.transforms as transformsfrom PIL import Image # Load pre-trained VGG16 modelmodel = models.vgg16(pretrained=True)model.eval() # Preprocess image and extract featuresdef extract_image_features(image_path): image = Image.open(image_path) preprocess = transforms.Compose([ transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), ]) image = preprocess(image).unsqueeze(0) with torch.no_grad(): features = model(image) return features # Usageimage_path = 'path/to/your/image.jpg'image_features = extract_image_features(image_path) Here an example of Bounding Box Extraction of how you can use pre-trained object detection model (e.g., Faster R-CNN) to extract bounding boxes from the images using the torchvision library: import torchimport torchvision.models as modelsimport torchvision.transforms as transformsfrom PIL import Image # Load pre-trained Faster R-CNN modelmodel = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)model.eval() # Preprocess image and extract bounding boxesdef extract_bounding_boxes(image_path): image = Image.open(image_path) preprocess = transforms.Compose([ transforms.ToTensor(), ]) image_tensor = preprocess(image) image_tensor = image_tensor.unsqueeze(0) with torch.no_grad(): predictions = model(image_tensor) boxes = predictions[0]['boxes'] # Process boxes further if needed return boxes # Usageimage_path = 'path/to/your/image.jpg'bounding_boxes = extract_bounding_boxes(image_path) — Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A27WT6AO5OYIVHIJTZOVGJDXIK5YHANCNFSM6AAAAAAYRRGEI4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[palonza]

@NisaHameed Great! Can you mark the thread as resolved?

[palonza]

hi @NisaHameed , Yes, Please. Thank you

[palonza]

Hello @NisaHameed , it would be nice if you could mark this thread as solved, it would help me. Thanks in advance!

[NisaHameed]

Yes, I will.

…

On Fri, 2 Jun 2023, 10:44 pm Jоsé P. Alоnza, ***@***.***> wrote: Hello @NisaHameed <https://github.com/NisaHameed> , it would be nice if you could mark this thread as solved, it would help me. Thanks in advance! — Reply to this email directly, view it on GitHub <#74 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A27WT6BUTJSF327A6TZY2ODXJIYAJANCNFSM6AAAAAAYRRGEI4> . You are receiving this because you were mentioned.Message ID: ***@***.***>