Content aware fill?

[lukestanley]

Something like this could be cool:
https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_photo/py_inpainting/py_inpainting.html

[minimaxir]

A number of comments on Twitter suggested using a GAN for inpainting (since we're in NN territory anyways), although this would require controlling for class within the GAN as well which makes it a little more difficult.