Add Wrinkled, Crumpled Paper Augmentation

[jboarman]

Wrinkled or crumpled paper is not an everyday concern. However, the issue does present real-world distortion that affects items such as receipts carried in one's pockets. Therefore, we should aim to implement an augmentation that simulates this effect.

Possible Methods

There are likely a number of available approaches, but the main options we've identified are as follows:

1. 3D Engines + 3D Meshes -- This technique, similar to DocCreator's implementation, involves 2 major obstacles: (a) leveraging a 3D rendering engine dependency with Python bindings and (b) developing a method to create realistic 3D meshes that simulate the effect of wrinkled/crumpled pages. Both of these obstacles are challenging to overcome, introduce risk along with additional dependencies and likely costly to implement, requiring substantial effort to fine-tune parameters and ensure the resulting meshes look realistic. Realistic 3D meshes will preserve the mesh topology according to physics as opposed to randomly displacing vertices, which is likely a very difficult task without a physics engine involved just to generate those meshes.
   This technique, which is similar to DocCreator's approach, involves tackling two major obstacles: (a) incorporating a 3D rendering engine dependency with Python bindings and (b) developing a method to create convincing 3D meshes that realistically simulate wrinkled or crumpled pages. Both of these challenges are difficult to overcome, introduce additional dependencies, and are likely to be expensive to implement. Moreover, considerable effort is required to fine-tune the parameters and ensure that the resulting meshes are realistic. Creating meshes that accurately preserve mesh topology according to physics, rather than simply randomly displacing vertices, is a complex task that may require the involvement of a physics engine.

2. Displacement Blending Against Paper Texture -- Although Augraphy provides a built-in function to apply ink over a background image, there is still a need to simulate crumpled paper using a randomly generated mesh. One potentially effective method would be to create a tessellation mesh based on Perlin noise, using the median shade of gray from the underlying noise and adding a slight blur to smooth the edges. This mesh could then be used to displace the page, creating the appearance of the wrinkled surface being aligned with the generated mesh. To achieve more realistic results, the mesh and shading can be rendered in multiple iterations, starting with a coarse tessellation and gradually increasing the level of detail in successive iterations.

1st Generation Implementation

As our first implementation, we should start our experiment with the simpler approach:

The approach you described has potential, and it can be used to generate meshes that simulate crumpled paper by leveraging Perlin noise and multiple levels of tessellation. Here's a summary of the method:

1. Generate Perlin noise as the basis for creating the crumpled paper effect.
2. Create a tessellation mesh that uses the Perlin noise values to displace the vertices, creating the appearance of a wrinkled surface.
3. Apply the median shade of grey from the underlying noise to the mesh, which can help create a more realistic shading effect.
4. Add a small amount of blur to smooth out the edges and soften the shading.
5. Displace the ink layer using the same Perlin noise to align the wrinkles on the ink layer with the generated mesh.
6. Render the mesh and shading in iterations, starting with coarse tessellation and applying finer levels of tessellation in successive iterations to achieve more realistic effects. Starting with a course mesh offers the benefit of an underlying gradient providing structure to the overall shape of the page prior to adding the smaller and more random crumples.
7. Apply the mesh during the post processing phase by displacing the pixels of rendered page based on the shaded mesh and then blending the shading of the mesh when merging the ink and paper layers. (see MATLAB example)

See tessellation approaches and noise-driven displacement methods discussed in #242 and #243.

2nd Generation Implementation

Truly realistic paper crumples and folds can only be generated using a physics-aware engine. The most performant engines for generating meshes and resultant renders will be written in Rust or C++. Leveraging a Rust-based engine opens up the opportunity to access its capabilities via WASM which is readily accessible via libraires such as Wasmer-Python, a wrapper for WASM sources (e.g., Rust -> WASM -> Python).

Prior Art

DocCreator offers a very nice rendering of wrinkled and crumpled paper, which is far superior to the simple blending used in NoisyOffice. However, DocCreator's implementation involved a full 3D rendering engine along with supplied real-world 3D meshes obtained using a laser to scan actual wrinkled pages. NoisyOffice, on the other hand, simply blended the foreground text over background images of crumpled paper for an unconvincing effect.

DocCreator Example

NoisyOffice Example

MATLAB Community Example (by DGM)

References:

https://www.mathworks.com/matlabcentral/answers/413506-how-to-create-a-crumpled-wrinkled-paper-effect-on-an-image#answer_976330
https://stackoverflow.com/questions/64278790/how-to-augment-scanned-document-image-with-creases-folds-and-wrinkles
https://github.com/panda3d/panda3d
https://github.com/isl-org/Open3D
https://github.com/mikedh/trimesh
https://bevyengine.org/
https://github.com/FyroxEngine/Fyrox
https://github.com/asny/three-d
https://doc-creator.labri.fr/
https://scikit-image.org/docs/dev/auto_examples/transform/plot_piecewise_affine.html#sphx-glr-auto-examples-transform-plot-piecewise-affine-py
https://blender.stackexchange.com/questions/107978/how-do-you-model-something-that-looks-wrinkled-or-crumpled

[kwcckw]

Now this can be done partially with Folding:
#327