Flexible and convenient texture loading

[emilk]

Intro

Showing images in egui is currently very cumbersome. There is egui_extras::RetainedImage, but it requires you to store the RetainedImage image somewhere and is obviously not very immediate mode.

Ideally we want users to be able to write something like:

ui.image("file://image.png");
ui.image("https://www.example.com/imag.png");

We also want 3rd party crates like egui_commonmark to be able to use the same system, without having to implement their own image loading.

Desired features

egui is designed to have minimal dependencies and never doing any system calls, and I'd like to keep it that way. Therefor the solution needs to be some sort of plugin system with callbacks. This is also the best choice for flexibility.

There is three steps to loading an image:

• Getting the image bytes
  • file://
  • https://
  • From a static list of include_bytes!("my_icon.png")
  • …
• Decoding the image
  • png, jpg, …
  • svg rasterization
  • …
• Uploading it to a texture

In most cases we want the loaded image to be passed to egui::Context::load_texture, which wil hand the image off to whatever rendering backend egui is hooked up to. This will allow the image loaded to work with any egui integration.

We should also allow users to have more exotic ways of loading images. At Rerun we have a data store where we store images, and it would be great if we could just reference images in that store with e.g. ui.image(rerun://data/store/path);.

In some cases however, the user may want to refer to textures that they themselves have uploaded to the GPU, i.e. return a TextureId::User. We do this at Rerun.

Proposal

I propose we introduce three new traits in egui:

• BytesLoader
• ImageLoader
• TextureLoaader

The user can then mix-and-match these as they will.

Users can register them with ctx.add_bytes_loader, ctx.add_image_loader, ctx.add_texture_loader,
and use them with ctx.load_bytes, ctx.load_image, ctx.load_texture.

We will supply good default implementations for these traits in egui_extras. Most users will never need to look at the details of these.

All these traits are designed to be immediate, i.e. callable each frame. It is therefor up to the implementation to cache any results.

They are designed to be able to suppor background loading (e.g. slow downloading of images).
For this they return Pending when something is being loaded. When the loading is done, they are responsible for calling ctx.request_repaint so that the now-loaded image will be shown.

They can also return an error.

Pending will be shown in egui using ui.spinner, and errors with red text.

Common code

enum LoadError {
    /// This loader does not support this protocol or image format.
    ///
    /// Try the next loader instead!
    NotSupported,

    /// A custom error string (e.g. "File not found: foo.png")
    Custom(String),
}

/// Given as a hint. Used mostly for rendering SVG:s to a good size.
///
/// All variants will preserve the original aspect ratio.
///
/// Similar to `usvg::FitTo`.
pub enum SizeHint {
    /// Keep original size.
    Original,

    /// Scale to width.
    Width(u32),

    /// Scale to height.
    Height(u32),

    /// Scale to size.
    Size(u32, u32),
}

BytesLoader

// E.g. from file or from the web
trait BytesLoader {
    /// Try loading the bytes from the given uri.
    ///
    /// When loading is done, call `ctx.request_repaint` to wake up the ui.
    fn load(&self, ctx: &egui::Context, uri: &str) -> Result<BytesPoll, LoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}


enum BytesPoll {
    /// Data is being loaded,
    Pending {
        /// Set if known (e.g. from a HTTP header, or by parsing the image file header).
        size: Option<Vec2u>,
    },

    /// Bytes are loaded.
    Ready {
        /// Set if known (e.g. from a HTTP header, or by parsing the image file header).
        size: Option<Vec2u>,

        /// File contents, e.g. the contents of a `.png`.
        bytes: Arc<u8>,
    },
}

ImageLoader

// Will usually defer to an `Arc<dyn BytesLoader>`, and then decode the result.
trait ImageLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        size_hint: SizeHint,
    ) -> Result<TexturePoll, TextureLoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}

trait ImagePoll {
    /// Image is loading.
    Pending {
        /// Set if known (e.g. from parsing the image header).
        size: Option<Vec2u>,
    },

    Ready {
        image: epaint::ColorImage, // Yes, only color images for now. Keep it simple.
    }
}

TextureLoader

// Will usually defer to an `Arc<dyn ImageLoader>`,
// and then just pipe the results to `egui::Context::laod_texture`.
trait TextureLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        texture_options: TextureOptions,
        size_hint: FitTo,
    ) -> Result<TexturePoll, TextureLoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}

struct SizedTexture {
    pub id: TextureId,
    pub size: Vec2u,
}

enum TexturePoll {
    /// Texture is loading.
    Pending {
        /// Set if known (e.g. from parsing the image header).
        size: Option<Vec2u>,
    },

    /// Texture is ready.
    Ready(SizedTexture),
}

Common usage

egui_extras::install_texture_loader(ctx);

ui.image("file://foo.png");

Advanced usage:

egui_ctx.add_texture_loader(Box::new(MyTextureLoader::new(…)));

let texture: Result<TexturePoll, _> = egui_ctx.load_texture(uri);

ui.add(egui::Image::from_uri("file://example.svg").fit_to_width());

Implementation

For version one, let's ignore cache eviction, and lets parse all images inline (on the main thread). We can always improve this in future PRs.

in egui

We just have the bare traits here, plus:

impl Context {
    pub fn add_bytes_loader(&self, loader: Arc<dyn BytesLoader>) { … }
    pub fn add_image_loader(&self, loader: Arc<dyn ImageLoader>) { … }
    pub fn add_texture_loader(&self, loaded: Arc<dyn TextureLoaader>) { … }

    // Uses the above registered loaders:
    pub fn load_bytes(&self, uri: &str) { 
        for loaders in &self.bytes_loaders {
            let result = loader.load(uri);
            if matches!(result, Err(LoadError::NotSupported)) {
                continue;  // try next loader
            } else {
                return result;
            }
        }
    }
    pub fn load_image(&self, uri: &str, size_hint: FitTo) { … }
    pub fn load_texture(&self, uri: &str, texture_options: TextureOptions, size_hint: FitTo) { … }
}


/// a bytes loader that loads from static sources (`include_bytes`)
struct IncludedBytesLoader {
    HashMap<String, &'static [u8]>,
}

impl DefaultTextureLoader { } 

impl TextureLoaader for DefaultTextureLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        texture_options: TextureOptions,
        size_hint: FitTo,
    ) -> Result<TexturePoll, TextureLoadError>
    {
        let img = ctx.load_image(uri, size_hint)?;
        match img {
            ImagePoll::Pending { size } => Ok(TexturePoll::Pending { size }),
            ImagePoll::Ready { image } => Ok(ctx.load_texture(image, texture_options)),
        }
    }    
}

in egui_extras

fn install_texture_loader(ctx: &egui::Context) {
    #[cfg(not(target_os == "wasm32"))]
    ctx.egui_add_bytes_loader(Arc::new(FileLoader::new()));

    #[cfg(feature = "ehttp")]
    ctx.egui_add_bytes_loader(Arc::new(EhttpLoader::new()));

    #[cfg(feature = "image")]
    ctx.add_texture_loader(Arc::new(ImageLoader::new()));

    #[cfg(feature = "svg")]
    ctx.add_texture_loader(Arc::new(SvgLoader::new()));
}

/// Just uses `std::fs::read` directly
#[cfg(not(target_os == "wasm32"))]
struct FileLoader {
    cached: HashMap<String, Arc<[u8]>,
}

/// Uses `ehttp` to fetch images from the web in background threads/tasks.
#[cfg(feature = "ehttp")]
struct EhttpLoader {
    cached: HashMap<String, poll_promise::Promise<Result<Arc<[u8]>>>>
}

#[cfg(feature = "image")]
struct ImageCrateLoader {
    cached: HashMap<String, ColoredImage>, // loaded parsed with `image` crate
}
impl ImageLoader for ImageCrateLoader { … }

#[cfg(feature = "svg")]
struct SvgLoader {
    cached: HashMap<(String, SizeHint), ColoredImage>,
}
impl ImageLoader for SvgLoader { … }

Implementation notes

• A lot of the necessary code can be found in https://github.com/lampsitter/egui_commonmark/pull/8/files?diff=split&w=0
• For how to handle SVGs, see egui_extras::RetainedImage, or the above url

[YgorSouza]

Pending will be shown in egui using ui.spinner, and errors with red text.

Maybe these could have overrides as well, for added flexibility.

pub fn pending_ui(self, add_contents: impl FnOnce(&mut Ui)) -> Self;
pub fn error_ui(self, add_contents: impl FnOnce(&mut Ui, LoadError)) -> Self;

But these would have to be evaluated before anything is drawn, so it would probably require having to call a show() method after all the builders. I suppose if users want more customization (like displaying specific error icons, downloading images only once and caching them in the data directory, logging errors etc), it's not that hard to write your own, currently. I tried my hand at that in this project.

[jprochazk]

Most of the proposal is implemented, but there's work left to do:

• Address the last comments on Managed texture loading #3297
• Make egui_extras::FileLoader optional
• Make load error messages shorter
• egui::Image should support a number of different sources
  • URI
  • URI + SizedTexture - equivalent to current .image API
  • URI + Bytes (include_image! shorthand)
• Replace old APIs with new ones, and deprecate old APIs
  • ui.image2 -> ui.image
  • Image2 -> Image
  • load_texture -> allocate_texture + deprecate load_texture
• Loading spinner should be positioned nicely within the allocated space
• Loading policy: Images may take a long time to load, but they shouldn't always use background threads.
  • Options:
    • Async (always use a background thread if possible)
    • Sync (always load synchronously if possible)
    • Auto (size heuristic)
  • Default should be set on Context, with override on Image.
• Better image type detection
  • Add mime: Option<String> to BytesPoll::Ready
    • Read from Content-Type in EhttpLoader
    • Read from file extension in FileLoader
  • ImageCrateLoader should fall back to image::guess_format if mime is None
• Add more examples
  • Add egui logo to the demo widget gallery
  • Image sizing options example
  • Combine a few examples into one image example (download_image, svg, retained_image)
• Deprecate RetainedImage and rework examples using it to use ui.image instead
• Try it in practice by making a PR to https://github.com/lampsitter/egui_commonmark and rerun
  • Use new Image API from egui lampsitter/egui_commonmark#11
  • Update to latest egui + use new Image api rerun-io/rerun#3311
• Image sizing API. Images are currently always ImageFit::ShrinkToFit, we want a more flexible API.
  • size_range, width_range, height_range, respect_aspect_ratio(bool)
  • size_fraction, width_fraction, height_fraction
  • size, width, height
  • original_size + scale
• Improve error messages on CORS errors on web Web: Improve opaque network error message ehttp#33
• Improve error message when there are no image loaders installed
• Improve the "error" image UI (perhaps show a red ⚠️ with details on hover?)
• Remove the newly added #![allow(deprecated)]
• Deprecate and replace Button::image_and_text and similar functions

[jprochazk]

• Improve error messages on CORS errors on web (maybe a ehttp PR)

When the server sends invalid CORS headers, then the browser will output only an opaque network error. This typically shows up as TypeError: Failed to fetch. According to the spec, a network error is a Response object with:

• type set to "error"
• status set to 0
• an empty status message
• an empty header list
• a null body

Browsers may output more information to the developer console or through other means, but they may not expose this information to the page. We can't tell the user it's "likely" a CORS error, because there are a whole bunch of other possible causes for a TypeError: Failed to fetch:

• No internet connection
• DNS resolution failed
• Firewall or proxy blocked the request
• Server is not reachable
• The URL is invalid (yes, this is just an opaque "failed to fetch"...)
• Server's SSL cert is invalid
• The initial get which returned HTML contained CSP headers to block access to the resource
• A browser extension blocked the request (e.g. ad blocker)

I don't know how to improve the situation. We could list all of the possible causes, but that's a lot of stuff to throw at a user. I also don't think it really needs to be improved, because the problem is clear once you look at the devtools console, which you do often anyway.

[emilk]

We can improve the situations like so:

• The current error message from ehttp is huge, and contains a callstack. That callstack is unhelpful imho.
• We can direct the users eyes to the development console

So, I suggest in ehttp we detect the TypeError and return a shorter error message, something like: "Failed to fetch. Check the developer console for details"

[lucasmerlin]

This is awesome!
If I understand this correctly, with these changes it should be possible to implement a, say, WgpuWasmTextureLoader, that uses createImageBitmap to decode images, removing the need to include the image crate in the wasm builds, which should reduce the wasm payload size and improve image decoding performance?

In my project I'm loading a lot of images so I implemented this with my own image abstraction, but it'd be awesome if I could switch to using the new loader traits in the future. I'd be happy to implement a WgpuWasmTextureLoader, either as part of the egui_wgpu crate or as a separate external crate.

[emilk]

createImageBitmap could be used by a new WebImageLoader that I think would fit well into egui_extras. It should implement trait ImageLoader, and be responsible for converting raw bytes into an egui::ColorImage. That would indeed remove the need for the image crate.

That's all you would need to implement though, and it would work for any egui running on web.
The default egui texture loader will just pass the loaded ColorImage to whatever the egui backend is, i.e. wgpu, glow, miniquad, … and it will Just Work™️.

[wangxiaochuTHU]

It's great!

Btw, is it also neccessary/possible that directly passing a grayscale image buffer (like &[u16]) to GPU, to avoid expanding it to ColorImage on CPU? In the current state of version 0.22.0, 16bit-per-pixel grayscale images are not supported. Also, 8bit-per-pixel W x H grayscale images need to be expanded to a rgba Vec<u8> with length W x H x 4 on CPU firstly and then be sent to GPU. When you capture 4K/8K images from a camera device at 30 fps and want to show them on App in realtime, too much overhead is paid for this casting.

[emilk]

@wangxiaochuTHU For now, only ColorImage is supported in egui, i.e. 32-bit sRGBA. Adding more image types (Gray8, Gray16, etc) is definitely possible, but I see that as outside the scope of this issue (it is an optimization, after all).