Image editor

[pngwn]

user facing

This is the new ImageEditor, which is a replacement for 3.x's Image with tool=.... I have linked the issues that it fixes below (both bugs and enhancements), so i wont go over every detail but the high level is as follows:

This component is a simple + streamlined image editor that handles basic image manipulations. It should be at least equivalent to the old Image with various tools set but less bad and more good. Additional features include:

• Layers

  • The new component had a concept of layers. Certain operations are constrained to a single layer and do not impact other layers. Details on each tool.
  • Each layers is returned individually to the user function.

• Background

  • has a concept of backgrounds.
  • A background is a special type of layer than cannot be drawn on or erased.
  • default is a plain white background which can be drawn on.
  • can also accept an image as the 'backgrounds'
    • upload will upload a file from the users computer
    • webcam will open a webcam capture window which will be set as the bg when it is captured.
    • paste will paste from clipboard

• Draw

  • users can draw on top of any background.
  • drawing is constrained to a single layer and does impact others.
  • The size and color can be selected by the user from a set of swatches and optionally an arbitrary 'picker'
  • The picker is only present if the app author allows it.
  • The size picker is a range input (slider)
  • The color picker is a custom color picker that users will be somewhat familiar with. The color picker also allows users to build their own swatches.
  • I think the color picker is very nice.

• Erase

  • Erase is basically draw but erase.
  • It only affects drawings on the currently selected layer
  • It has only size which operates the same as draw's size

• Crop

  • Crops the entire canvas
  • Crop is non destructive, you can re-enter crop mode and uncrop however you like.
  • i think crop is very nice too.

• Undo

  • as in 3.x we have undo, except this time is actually works

• Redo

  • Because of the architecture (or the fact that it even has one), we can also easily 'redo' any operation.

• Performance

  • The new image editor is much more performant in my experience. It seems to work well with pretty large images and lots of operations. There are definitely opportunities to optimise though, history is not cheap.

technical

The new Image Editor is built on top of pixi.js, a javascript library for creating 2d webgl experiences. It is an exceptional library if a little confusing but it sits at an almost perfect level of abstraction. High level enough so you don't need to think too much about the inner workings, low level enough that you can dive in when you need to. It is not a sketch library in any way but has all of the correct primitives. It is also really fast.

The image editor has a bunch of code but its a lot less than I thought it would need to be. The image editor has been designed to be flexible and maintainable, it should be relatively easy to add new functionality without needing to touch the whole thing. For example, modification to the crop tool, should only require changes to the cropping part of the code.

The editor uses the command pattern which is a little complex but incredibly powerful. Every editing action is a 'command'. A command knows how to perform itself and how to undo itself. We have a command manager that takes in every command, and adds it to its history. When we want a new feature, we create a new command that is dedicated to just that thing. crop is a command, draw is a command, 'add bg image` is a command.

When we want to undo something we ask the command manager to undo it, it will then call the current history node's undo method and move a point back. And repeat.

When we want to redo something it will call the 'next' commands execute method, and move the history point forwards. The history is implement as a doubly linked list so it is simple to traverse and very difficult to skip nodes.

The 'command' interfaces look like this:

interface Command {
  start?: (...args: any[]) => any;
  continue?: (...args: any[]) => any;
  stop?: (...args: any[]) => any;
  execute: () => void;
  undo: () => void
}

Each command can decide how to implement all of these methods, it doesn't matter as long as they 'work'. Only execute and undo are required.

The reason start continue and stop exist is best explained with an example:

Lets take drawing. A single drawing operation is one line. So execute should draw an entire line onto the canvas (for example when we are 'redoing' a line). However, we can only call execute when the line has finished drawing because that will get added to the history as a single entity. Since this is an image editor we need real time updates. With this in mind, i added start, continue, and stop as optional methods that can be used to provide realtime updates to the canvas, while still giving us the option to undo/redo later.

In the case of drawing:

• we create a new command and trigger start when a user presses their mouse down. We set up some initial textures and snapshot the initial state.
• When the user moves their mouse we trigger continue. here we are interpolating between x/y points (to ensure a smooth line), drawing to the texture we set up before and rendering it.
• When the user releases their mouse button, we call stop. In stop we perform some final clean up and reset the textures. We reuse the textures for performance reasons, so this clean up is important.

After realtime drawing execute is basically a no-op (it always get called when we pass it to the command manager). However if no realtime drawing has been completed then it will draw the full line.

The reason it is important that the command manager is able to handle these realtime updates is because all drawing operations then just live in this one ovject which makes maintenance much easier. Since everything is passed into these commands as arguments, unit testing is also straight forward.

The command pattern is actually really nice for this use case and allows us to easily add complex features without drowning in a sea of complexity. It also unlock other possible features in the future (really fancy ones like macros/ batch processing).

jsdoc

I have made extensive use of jsdoc annotations throughout the code in the hope that it offers a little help for us all when navigating this component in the future.

This is a practice that I think we should experiment with across the rest of the code base (frontend). I personally find it very helpful, both when writing and reading code because it makes me think about my interfaces + what they are actually doing.

Closes #5645
Closes #5206
Closes #5152
Closes #5055
Closes #4931
Closes #4907
Closes #4842
Closes #4677
Closes #4653
Closes #4492
Closes #4413
Closes #4290
Closes #4252
Closes #4159
Closes #4120
Closes #4011
Closes #3810
Closes #3623
Closes #3535
Closes #3472
Closes #3305
Closes #3280
Closes #3138
Closes #3110
Closes #2649
Closes #2425
Closes #2314
Closes #1591
Closes #466

🎯 PRs Should Target Issues

Before your create a PR, please check to see if there is an existing issue for this change. If not, please create an issue before you create this PR, unless the fix is very small.

Not adhering to this guideline will result in the PR being closed.

Tests

1. PRs will only be merged if tests pass on CI. To run the tests locally, please set up your Gradio environment locally and run the tests: bash scripts/run_all_tests.sh

2. You may need to run the linters: bash scripts/format_backend.sh and bash scripts/format_frontend.sh

[pngwn]

Okay, made a few changes.

I couldn't add click to upload yet due to how things are put together, we can do that later if it is really needed but feels off for an image editor imo.

• I've fixed up the 'empty' text so it is more appropriate depending what tools are shown
• Empty text now shows correctly when you clear the component
• Clearing works properly whether its done vis the x button or the Clear button
• Crop can now correctly be disabled.
• Eraser is only enabled if brush is enabled
• After uploading an image, crop will automatically be selected if a crop_size is set
• If sources=[] the draw tool will be selected.
• The status tracker now show on the uploading component (the editor) when we are preparing the data (uploading the files). Then after that is complete the normal process kicks off. We can improve the UX for this later but it should address the current lack of feedback (although 5 seconds seems a long time, i couldn't repro that).
• I've also cleaned up the python API a little. Previously it wasn't possible to disable the brush or eraser (because None cases get a default brush), so eraser and brush can now accept a boolean too.

[pngwn]

And we are done!

Thanks @abidlabs @hannahblair @freddyaboulton @aliabid94 for the help with various bits and pieces, the PRs, and the detailed reviews + tested. Very much appreciated!

[wamiq-reyaz]

Hey all,

thanks for the wonderful work. I am still wondering about a few things though

1. The brushes are not well documented. Like how does one create a custom swatch?
2. If one instead uses a colorpicker component, how does one update the brush color for an ImageEditor? The current events do NOT target the brush color.
3. The documentation for the ImageEditor does not actually showcase how the ImageEditor is actually used but instead has an unrelated rotation example