An image processing project where we implement style transfer, which is a type of content mixing of two images, via texture synthesis algorithms, we implemented Style Transfer based on Elad's et al. paper "Style-Transfer via Texture-Synthesis", which relies on classical methods. The original paper can be found at original, and our documentation is available at our doc.
- Segmentation
- Building Gaussian Pyramids
- Initialization
- Color Transfer
- Patch Matching
- Robust Aggregation
- Content Fusion
- Denoise
Each step will be discussed in details.
Our exploration encompassed the implementation of different segmentation techniques. Initially, we employed GrabCut, as advocated in Elad's paper. However, GrabCut's reliance on user-defined foreground and background annotations often resulted in suboptimal performance, particularly when these annotations were absent, leading to degraded outcomes. Subsequently, an alternative edge-based segmentation technique was adopted. This technique involved a series of steps, starting with the application of the Canny edge detection algorithm, followed by dilation, contour extraction, and subsequent filling. Notably, this approach exhibited superior performance in multiple scenarios, showcasing improved outcomes compared to GrabCut in diverse contexts.
| Segmentation mask using GrabCut algorithm | Segmentation mask using edge-Based Segmentation |
|---|---|
 |
 |
The process of constructing Gaussian pyramids involves downscaling the resolution of the content, style and segmentation mask images, creating multiple layers. Operating the style transfer algorithm on each of these layers is instrumental in enhancing the content image's capacity to assimilate broader stylistic features from the style image.
Following the completion of pre-processing functions and pyramid construction, the algorithm initiates X, the primary image subjected to algorithmic operations. Initially set as the content image, X undergoes an additional procedural step wherein a significant amount of Gaussian noise is applied. The deliberate injection of Gaussian noise into the content image, serving as the basis for subsequent operations, fosters a more pronounced and authentic style transfer experience, minimizing repetitive patterns and enhancing the overall stylistic coherence within the transformed image.
| Patch matching on uniform white image with no noise | Patch matching on uniform image with added noise |
|---|---|
 |
 |
The process of color transfer involves transposing the color palette from the style image onto the content image. Several methodologies were explored to accomplish this task, commencing with histogram matching, as advocated in Elad’s paper. While histogram matching showcased efficacy in many instances, it occasionally exhibited limitations, leading to instances of extreme color transfer. To mitigate these challenges and ensure a more nuanced and controlled color transformation, an alternative approach was adopted. Subsequently, color transfer in LAB channels was implemented as an alternative methodology.
| Original image | Style |
|---|---|
 |
 |
| Color transfer using histogram matching | Color transfer using LAB channels. |
|---|---|
 |
 |
The crux of patch matching revolves around identifying the nearest patch in the style image that corresponds to the current patch in the content image. This fundamental process ensures that the changes introduced post-style transfer remain coherent with the original content. The minimization task aims to determine the most fitting patch in the style image, corresponding to each patch in the content image, using a Nearest Neighbor approach. However, the exhaustive nature of Nearest Neighbor for each patch incurs substantial computational overheads. To address the computational complexity while retaining fidelity in patch matching, Principal Component Analysis (PCA) was employed as an alternative.
| Patch matching with patch sizes = [40, 30] | Patch matching with patch sizes = [20, 10] |
|---|---|
 |
 |
Robust Aggregation, a crucial procedure within the style transfer algorithm, involves the systematic adjustment of each patch in the content image. This adjustment aims to iteratively align each content patch closer to its matched counterpart in the style image. The methodology employed for this purpose, as elucidated in Kwatra's paper, relies on Iteratively Reweighted Least Squares (IRLS). This technique systematically refines the content patches by iteratively adjusting their attributes to converge towards the corresponding style patches.
Content Fusion serves as the pivotal stage in the style transfer process, focusing on reapplying the content image in alignment with the segmentation outcome. This crucial step aims to ensure the preservation of significant content elements within the final stylized output.
Denoising stands as a crucial step within the style transfer process, pivotal in guaranteeing seamless transitions within the resultant image. It plays a pivotal role in refining the overall visual quality by minimizing unwanted artifacts or noise. Throughout our experimentation, various denoising methodologies were rigorously assessed for their efficacy in achieving smooth transitions within the output image. Among the methodologies scrutinized, the Bilateral Filter emerged as the most proficient performer.
| Content | Style | Result |
|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
We also modified the algorithm to preserve the geometric shapes in the style images.
| Content | Style | Result |
|---|---|---|
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
| Name | |
|---|---|
| Fares Atef | faresatef553@gmail.com |
| Ghaith Mohamed | gaoia123@gmail.com |
| Amr ElSheshtawy | Sheshtawy321@gmail.com |
| Amr Magdy | amr4121999@gmail.com |