Inpaint channel

CHANGELOG.md

@@ -7,9 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 ### Added
 - [PR#57](https://github.com/EmbarkStudios/texture-synthesis/pull/57) CLI: Added the `flip-and-rotate` subcommand which applies flip and rotation transforms to each example input and adds them as additional examples. Thanks [@JD557](https://github.com/JD557)!
+- [PR#60](https://github.com/EmbarkStudios/texture-synthesis/pull/60) added the ability to specify a channel to use as
+the inpaint mask instead of a separate image. Thanks [@khskarl](https://github.com/khskarl)!
+- Added `SessionBuilder::inpaint_example_channel`
+- CLI: Added `--inpaint-channel <r|g|b|a>`
 
 ### Changed
-- Replace [`failure`](https://crates.io/crates/failure) crate for error handling with just std::error::Error
+- Replace [`failure`](https://crates.io/crates/failure) crate for error handling with just `std::error::Error`
 
 ### Fixed
 - Validate that the `--m-rand` / `random_sample_locations` parameter is > 0. [#45](https://github.com/EmbarkStudios/texture-synthesis/issues/45)

README.md

@@ -47,6 +47,7 @@ fn main() -> Result<(), ts::Error> {
 `cargo run --release -- --out out/01.jpg generate imgs/1.jpg`
 
 You should get the following result with the images provided in this repo:
+
 <img src="https://i.imgur.com/8p6nVYl.jpg" width="600" height="364">
 
 ### 2. Multi example generation
@@ -99,6 +100,7 @@ fn main() -> Result<(), ts::Error> {
 `cargo run --release -- --rand-init 10 --seed 211 --in-size 300x300 -o out/02.png --debug-out-dir out generate imgs/multiexample/1.jpg imgs/multiexample/2.jpg imgs/multiexample/3.jpg imgs/multiexample/4.jpg`
 
 You should get the following result with the images provided in this repo:
+
 <img src="https://i.imgur.com/tbz5d57.jpg" width="600" height="364">
 
 ### 3. Guided Synthesis
@@ -137,6 +139,7 @@ fn main() -> Result<(), ts::Error> {
 to the example will be used as another guide path and there won't be any examples.
 
 You should get the following result with the images provided in this repo:
+
 <img src="https://i.imgur.com/arTCi2f.jpg" width="600" height="364">
 
 ### 4. Style Transfer
@@ -174,6 +177,7 @@ fn main() -> Result<(), ts::Error> {
 `cargo run --release -- --alpha 0.8 -o out/04.png transfer-style --style imgs/multiexample/4.jpg --guide imgs/tom.jpg`
 
 You should get the following result with the images provided in this repo:
+
 <img src="https://i.imgur.com/1E7eDAb.jpg" width="600" height="364">
 
 ### 5. Inpaint
@@ -226,15 +230,53 @@ Note that the `--out-size` parameter determines the size for all inputs and outp
 `cargo run --release -- --out-size 400 --inpaint imgs/masks/3_inpaint.jpg -o out/05.png generate imgs/3.jpg`
 
 You should get the following result with the images provided in this repo:
+
 <img src="https://i.imgur.com/WZm2HHL.jpg" width="600" height="364">
 
-### 6. Tiling texture
+### 6. Inpaint Channel
+
+![Imgur](https://i.imgur.com/FqvV651.jpg)
+
+Instead of using a separate image for our inpaint mask, we can instead obtain the information from a specific
+channel. In this example, the alpha channel is a circle directly in the middle of the image.
+
+#### API - [06_inpaint_channel](lib/examples/06_inpaint_channel.rs)
+
+```rust
+use texture_synthesis as ts;
+
+fn main() -> Result<(), ts::Error> {
+    let texsynth = ts::Session::builder()
+        // Let the generator know that it is using 
+        .inpaint_example_channel(
+            ts::ChannelMask::A,
+            &"imgs/bricks.png",
+            ts::Dims::square(400),
+        )
+        .build()?;
+
+    let generated = texsynth.run(None);
+
+    //save the result to the disk
+    generated.save("out/06.jpg")
+}
+```
+
+#### CLI
+
+`cargo run --release -- --inpaint-channel a -o out/06.png generate imgs/bricks.jpg`
+
+You should get the following result with the images provided in this repo:
+
+<img src="https://imgur.com/7IuVN5K.jpg" width="400" height="400">
+
+### 7. Tiling texture
 
 ![](https://i.imgur.com/nFpCFzy.jpg)
 
 We can make the generated image tile (meaning it will not have seams if you put multiple images together side-by-side). By invoking inpaint mode together with tiling, we can make an existing image tile.
 
-#### API - [06_tiling_texture](lib/examples/06_tiling_texture.rs)
+#### API - [07_tiling_texture](lib/examples/07_tiling_texture.rs)
 
 ```rust
 use texture_synthesis as ts;
@@ -258,54 +300,55 @@ fn main() -> Result<(), ts::Error> {
 
     let generated = texsynth.run(None);
 
-    generated.save("out/06.jpg")
+    generated.save("out/07.jpg")
 }
 ```
 
 #### CLI
 
-`cargo run --release -- --inpaint imgs/masks/1_tile.jpg --out-size 400 --tiling -o out/06.bmp generate imgs/1.jpg`
+`cargo run --release -- --inpaint imgs/masks/1_tile.jpg --out-size 400 --tiling -o out/07.bmp generate imgs/1.jpg`
 
 You should get the following result with the images provided in this repo:
+
 <img src="https://i.imgur.com/foSlREz.jpg" width="600" height="364">
 
-### 7. Repeat texture synthesis transform on a new image
+### 8. Repeat texture synthesis transform on a new image
 
 ![](https://i.imgur.com/WEf6iir.jpg)
 
 We can re-apply the coordinate transformation performed by texture synthesis onto a new image.
 
-#### API - [07_repeat_transform](lib/examples/07_repeat_transform.rs)
+#### API - [08_repeat_transform](lib/examples/08_repeat_transform.rs)
 
 ```rust
 use texture_synthesis as ts;
 
 fn main() -> Result<(), ts::Error> {
-    //create a new session
+    // create a new session
     let texsynth = ts::Session::builder()
         //load a single example image
         .add_example(&"imgs/1.jpg")
         .build()?;
 
-    //generate an image
+    // generate an image
     let generated = texsynth.run(None);
 
-    //now we can apply the same transformation of the generated image
-    //onto a new image (which can be used to ensure 1-1 mapping between multiple images)
-    //NOTE: it is important to provide same number and image dimensions as the examples used for synthesis
-    //otherwise, there will be coordinates mismatch
+    // now we can apply the same transformation of the generated image
+    // onto a new image (which can be used to ensure 1-1 mapping between multiple images)
+    // NOTE: it is important to provide same number and image dimensions as the examples used for synthesis
+    // otherwise, there will be coordinates mismatch
     let repeat_transform_img = generated
         .get_coordinate_transform()
         .apply(&["imgs/1_bw.jpg"])?;
 
-    //save the image to the disk
-    //01 and 01_2 images should match perfectly
-    repeat_transform_img.save("out/01_2.jpg").unwrap();
-    generated.save("out/01.jpg")
+    // save the image to the disk
+    // 08 and 08_repeated images should match perfectly
+    repeat_transform_img.save("out/08_repeated.jpg").unwrap();
+    generated.save("out/08.jpg")
 }
 ```
 
-### 8. Combining texture synthesis 'verbs'
+### 9. Combining texture synthesis 'verbs'
 
 We can also combine multiple modes together. For example, multi-example guided synthesis:
 
@@ -317,7 +360,7 @@ Or chaining multiple stages of generation together:
 
 For more use cases and examples, please refer to the presentation ["More Like This, Please! Texture Synthesis and Remixing from a Single Example"](https://youtu.be/fMbK7PYQux4)
 
-### 9. Additional CLI functionality
+### Additional CLI functionality
 
 Some functionality is only exposed through the CLI and not built into the library.
 

cli/src/main.rs

@@ -5,8 +5,8 @@ use structopt::StructOpt;
 
 use std::path::PathBuf;
 use texture_synthesis::{
-    image::ImageOutputFormat as ImgFmt, load_dynamic_image, Dims, Error, Example, ImageSource,
-    SampleMethod, Session,
+    image::ImageOutputFormat as ImgFmt, load_dynamic_image, ChannelMask, Dims, Error, Example,
+    ImageSource, SampleMethod, Session,
 };
 
 fn parse_size(input: &str) -> Result<Dims, std::num::ParseIntError> {
@@ -37,6 +37,23 @@ fn parse_img_fmt(input: &str) -> Result<ImgFmt, String> {
     Ok(fmt)
 }
 
+fn parse_mask(input: &str) -> Result<ChannelMask, String> {
+    let mask = match &input.to_lowercase()[..] {
+        "r" => ChannelMask::R,
+        "g" => ChannelMask::G,
+        "b" => ChannelMask::B,
+        "a" => ChannelMask::A,
+        mask => {
+            return Err(format!(
+                "unknown mask '{}', must be one of 'a', 'r', 'g', 'b'",
+                mask
+            ))
+        }
+    };
+
+    Ok(mask)
+}
+
 #[derive(StructOpt)]
 #[structopt(rename_all = "kebab-case")]
 struct Generate {
@@ -154,6 +171,9 @@ struct Opt {
     /// Path to an inpaint map image, where black pixels are resolved, and white pixels are kept
     #[structopt(long, parse(from_os_str))]
     inpaint: Option<PathBuf>,
+    /// Flag to extract inpaint from one of the example's channels
+    #[structopt(long, parse(try_from_str = parse_mask), conflicts_with = "inpaint")]
+    inpaint_channel: Option<ChannelMask>,
     /// Size of the generated image, in `width x height`, or a single number for both dimensions
     #[structopt(
         long,
@@ -273,7 +293,7 @@ fn real_main() -> Result<(), Error> {
             match mask.as_str() {
                 "ALL" => example.set_sample_method(SampleMethod::All),
                 "IGNORE" => example.set_sample_method(SampleMethod::Ignore),
-                path => example.set_sample_method(SampleMethod::Image(ImageSource::Path(
+                path => example.set_sample_method(SampleMethod::Image(ImageSource::from_path(
                     &std::path::Path::new(path),
                 ))),
             };
@@ -283,16 +303,25 @@ fn real_main() -> Result<(), Error> {
     let mut sb = Session::builder();
 
     // TODO: Make inpaint work with multiple examples
-    if let Some(ref inpaint) = args.inpaint {
-        let mut inpaint_example = examples.remove(0);
+    match (args.inpaint_channel, &args.inpaint) {
+        (Some(channel), None) => {
+            let inpaint_example = examples.remove(0);
 
-        // If the user hasn't explicitly specified sample masks, assume they
-        // want to use the same mask
-        if args.sample_masks.is_empty() {
-            inpaint_example.set_sample_method(inpaint);
+            sb = sb.inpaint_example_channel(channel, inpaint_example, args.out_size);
         }
+        (None, Some(inpaint)) => {
+            let mut inpaint_example = examples.remove(0);
 
-        sb = sb.inpaint_example(inpaint, inpaint_example, args.out_size);
+            // If the user hasn't explicitly specified sample masks, assume they
+            // want to use the same mask
+            if args.sample_masks.is_empty() {
+                inpaint_example.set_sample_method(inpaint);
+            }
+
+            sb = sb.inpaint_example(inpaint, inpaint_example, args.out_size);
+        }
+        (None, None) => {}
+        (Some(_), Some(_)) => unreachable!("we prevent this combination with 'conflicts_with'"),
     }
 
     sb = sb

lib/benches/all-the-things.rs

@@ -190,6 +190,42 @@ fn inpaint(c: &mut Criterion) {
     group.finish();
 }
 
+fn inpaint_channel(c: &mut Criterion) {
+    static DIM: u32 = 25;
+
+    // Load the example image once to reduce variation between runs,
+    // though we still do a memcpy each run
+    let example_img = ts::load_dynamic_image(ts::ImageSource::from(&"../imgs/bricks.png")).unwrap();
+
+    let mut group = c.benchmark_group("inpaint_channel");
+    group.sample_size(10);
+
+    for dim in [DIM, 2 * DIM, 4 * DIM, 8 * DIM, 16 * DIM].iter() {
+        group.bench_with_input(BenchmarkId::from_parameter(dim), dim, |b, &dim| {
+            b.iter_custom(|iters| {
+                let mut total_elapsed = Duration::new(0, 0);
+                for _i in 0..iters {
+                    let sess = ts::Session::builder()
+                        .inpaint_example_channel(
+                            ts::ChannelMask::A,
+                            ts::Example::builder(example_img.clone()),
+                            ts::Dims::square(dim),
+                        )
+                        .build()
+                        .unwrap();
+
+                    let start = Instant::now();
+                    black_box(sess.run(None));
+                    total_elapsed += start.elapsed();
+                }
+
+                total_elapsed
+            });
+        });
+    }
+    group.finish();
+}
+
 fn tiling(c: &mut Criterion) {
     static DIM: u32 = 25;
 
@@ -235,6 +271,7 @@ criterion_group!(
     guided,
     style_transfer,
     inpaint,
+    inpaint_channel,
     tiling,
 );
 criterion_main!(benches);

lib/examples/06_inpaint_channel.rs

@@ -0,0 +1,17 @@
+use texture_synthesis as ts;
+
+fn main() -> Result<(), ts::Error> {
+    let texsynth = ts::Session::builder()
+        // Let the generator know that it is using
+        .inpaint_example_channel(
+            ts::ChannelMask::A,
+            &"imgs/bricks.png",
+            ts::Dims::square(400),
+        )
+        .build()?;
+
+    let generated = texsynth.run(None);
+
+    //save the result to the disk
+    generated.save("out/06.jpg")
+}

lib/examples/06_tiling_texture.rs → lib/examples/07_tiling_texture.rs

@@ -19,5 +19,5 @@ fn main() -> Result<(), ts::Error> {
 
     let generated = texsynth.run(None);
 
-    generated.save("out/06.jpg")
+    generated.save("out/07.jpg")
 }

lib/examples/08_repeat_transform.rs

@@ -0,0 +1,25 @@
+use texture_synthesis as ts;
+
+fn main() -> Result<(), ts::Error> {
+    // create a new session
+    let texsynth = ts::Session::builder()
+        //load a single example image
+        .add_example(&"imgs/1.jpg")
+        .build()?;
+
+    // generate an image
+    let generated = texsynth.run(None);
+
+    // now we can apply the same transformation of the generated image
+    // onto a new image (which can be used to ensure 1-1 mapping between multiple images)
+    // NOTE: it is important to provide same number and image dimensions as the examples used for synthesis
+    // otherwise, there will be coordinates mismatch
+    let repeat_transform_img = generated
+        .get_coordinate_transform()
+        .apply(&["imgs/1_bw.jpg"])?;
+
+    // save the image to the disk
+    // 08 and 08_repeated images should match perfectly
+    repeat_transform_img.save("out/08_repeated.jpg").unwrap();
+    generated.save("out/08.jpg")
+}