Add InkShifter Augmentation to Shift Ink Pixels Randomly and Follow Background Shadows

[jboarman]

Add jitter to ink and follow shadows or other physical deformations in paper layer

Lettering in real-world document scans aren't perfectly oriented on the page. Either the source is handwritten or just a natural artifact of being physically processed, paper stretching, wrinkling, imperfections in printing process, etc.

The sherlockdoyle/Handwriter repo offers some excellent perturbations that make the output look pretty realistic that we should consider.

Noise masks are used with the following techniques:

1. Jitter Shifting nudges letters randomly in different directions
2. Shadow-Driven Shifting compresses or slants regions to follow the shadow of the underlying paper texture

Random Jitter Pixel Shifting

To add variation in every letter, the text is randomly moved just tiny amount using a noise map. The code uses noise maps (one for each axis). The amount to move the text with these maps can scaled from 0 (no movement) to any specified number of pixels (either in the positive or negative direction). The scale of the noise map itself is also controlled to adjust the frequency of the noise.

def displace_image(img: imgRGB, mapx: imgGray, mapy: imgGray, fill: Tuple[int, int, int] = (255, 255, 255)) -> imgRGB:
    """Apply displacement map to an image."""
    gridx, gridy = np.meshgrid(np.arange(img.shape[1], dtype=np.float32),
                               np.arange(img.shape[0], dtype=np.float32))
    if mapx is None:
        mapx = gridx
    else:
        mapx += gridx
    if mapy is None:
        mapy = gridy
    else:
        mapy += gridy

    return cv2.remap(img, mapx, mapy, cv2.INTER_CUBIC, borderMode=cv2.BORDER_CONSTANT, borderValue=fill)


img_dispx = perlin((H, W), (text_shift_scale, text_shift_scale))
img_dispy = perlin((H, W), (text_shift_scale, text_shift_scale))
disp_img = displace_image(orig, -0.363636364*text_shift_factor*img_dispx, text_shift_factor*img_dispy)

Shadow-Driven Pixel Shifting

Shadows in the background can be used to further displace the image to give the effect that the text curves with the page. The code for this step is less clear, but debugging their code should reveal how the distortion mask is being created to validate any assumptions.

faded_img = put_fading(slanted_img, perlin((H, W), (text_shift_scale, text_shift_scale)), text_fade_factor)
norm_back = cv2.normalize(
    cv2.cvtColor(back, cv2.COLOR_BGR2GRAY),
    None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F
)
page_morphed_img = displace_image(faded_img, None, 40-60*norm_back)

Main Handwriter Codeblock

https://github.com/sherlockdoyle/Handwriter/blob/main/writing_artifact.py#L388-L428