SAR Image Colorization using Multi-Domain CycleGAN

This project implements a novel approach for colorizing Synthetic Aperture Radar (SAR) images using deep learning techniques. SAR is an active remote sensing technology that provides high-resolution imagery regardless of weather conditions or time of day. However, SAR images are inherently grayscale and can be difficult to interpret. Our system transforms these grayscale SAR images into natural-looking color images that resemble optical satellite imagery across multiple domains.

Project Overview

SAR images are valuable for monitoring Earth's surface under conditions where optical sensors fail, but their grayscale nature makes them challenging for human interpretation. This project addresses this limitation by:

1. Developing a domain-specific generative adversarial network to translate SAR images to realistic color images
2. Supporting multiple target domains (e.g., farmland, desert, urban areas) with a single model
3. Preserving detailed structural information while adding plausible coloration
4. Enabling deployment on resource-constrained environments through lightweight model variants

Technical Approach

Multi-Domain CycleGAN Architecture

The core of our system is a modified CycleGAN architecture with several key enhancements:

• Domain Conditioning: Using adaptive instance normalization (AdaIN) to modulate features based on target domain
• Self-Attention Mechanism: For better spatial consistency and coherence in generated images
• Detail Preservation: Special modules designed to preserve fine structural details from SAR imagery
• Multi-Scale Processing: Feature pyramids and multi-scale discriminators to handle features at different resolutions

Network Architecture Components

1. Enhanced Generator:

   • Domain-specific embedding module
   • Multi-scale input processing
   • Detail-preserving residual blocks
   • Self-attention mechanisms

2. Enhanced Discriminator:

   • Multi-scale feature pyramid extraction
   • Detail-focused attention modules
   • Domain classification branch

3. Lightweight Variants:

   • Memory-efficient alternatives for deployment on resource-constrained devices
   • Simpler architectures with fewer parameters while maintaining reasonable quality

Training Process

The model is trained on paired and unpaired datasets of SAR and optical satellite imagery across multiple domains. The training leverages:

• Cycle consistency loss to ensure structure preservation
• Adversarial loss for realistic coloration
• Domain classification loss for domain-specific features
• Identity loss to maintain content when already in target domain
• Perceptual loss for better visual quality

Directory Structure

/
├── models/                   # Model definitions
│   ├── generators.py         # Generator architectures 
│   ├── discriminators.py     # Discriminator architectures
│   ├── cycle_gan.py          # CycleGAN implementation
│   └── lightweight_generator.py  # Memory-efficient generator variants
├── dataset.py                # Dataset loading and preprocessing
├── train.py                  # Full model training script
├── train_lightweight.py      # Training script for resource-constrained environments
├── train_enhanced.py         # Advanced training with progressive techniques
├── config.py                 # Configuration and hyperparameters
├── utils.py                  # Utility functions
└── SAR_Colorization_Colab_fixed.ipynb  # Interactive Colab notebook for demonstration

Results

The model successfully colorizes SAR images across different domains, producing visually appealing and domain-appropriate colorizations:

• Farmland Domain: Realistic greens and browns matching agricultural landscapes
• Desert Domain: Appropriate sandy colors and terrain features
• Urban Areas: Building structures with appropriate coloration

Quantitatively, the model achieves:

• PSNR: ~20-24 dB (depending on domain)
• SSIM: ~0.70-0.78

Sample Results

Lightweight Model Performance

The lightweight model variants sacrifice some quality for significant reductions in:

• Memory usage: ~75% less memory consumption
• Computation: ~60% faster inference times
• Model size: ~80% smaller

This makes them suitable for deployment on edge devices and in bandwidth-constrained environments.

Usage

Training

To train the full model:

python train.py --sar_root path/to/sar/data --optical_root path/to/optical/data

For resource-constrained environments:

python train_lightweight.py --sar_root path/to/sar/data --optical_root path/to/optical/data --img_size 128

Inference

from models.generators import EnhancedGenerator
import torch
from PIL import Image
import torchvision.transforms as transforms

# Load model
model = EnhancedGenerator(input_nc=1, output_nc=3, n_domains=2)
model.load_state_dict(torch.load('path/to/model_checkpoint.pth'))
model.eval()

# Prepare image
transform = transforms.Compose([
    transforms.Resize((256, 256)),
    transforms.ToTensor(),
    transforms.Normalize([0.5], [0.5])
])

# Load SAR image
sar_img = Image.open('path/to/sar_image.png').convert('L')
sar_tensor = transform(sar_img).unsqueeze(0)

# Create domain vector (e.g., for farmland domain)
domain_vec = torch.zeros(3)  # 2 domains + 1 for SAR
domain_vec[1] = 1  # Select first domain (farmland)
domain_vec = domain_vec.unsqueeze(0)

# Generate colorized image
with torch.no_grad():
    colorized = model(sar_tensor, domain_vec)

# Convert to image
colorized_img = transforms.ToPILImage()(colorized.squeeze(0) * 0.5 + 0.5)
colorized_img.save('colorized_output.png')

Future Work

• Integration of transformer-based attention mechanisms
• Support for additional domains (e.g., snow-covered, urban, coastal)
• User-guided colorization with interactive controls
• Temporal consistency for SAR image sequences

References

1. Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. ICCV 2017.
2. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Loy, C. C., Qiao, Y., & Tang, X. (2018). ESRGAN: Enhanced super-resolution generative adversarial networks. ECCVW 2018.
3. Huang, X., & Belongie, S. (2017). Arbitrary style transfer in real-time with adaptive instance normalization. ICCV 2017.
4. Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2019). Self-attention generative adversarial networks. ICML 2019.

License

MIT