Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flexible and convenient texture loading #3291

Closed
emilk opened this issue Sep 1, 2023 · 8 comments
Closed

Flexible and convenient texture loading #3291

emilk opened this issue Sep 1, 2023 · 8 comments
Assignees
Milestone

Comments

@emilk
Copy link
Owner

emilk commented Sep 1, 2023

Intro

Showing images in egui is currently very cumbersome. There is egui_extras::RetainedImage, but it requires you to store the RetainedImage image somewhere and is obviously not very immediate mode.

Ideally we want users to be able to write something like:

ui.image("file://image.png");
ui.image("https://www.example.com/imag.png");

We also want 3rd party crates like egui_commonmark to be able to use the same system, without having to implement their own image loading.

Desired features

egui is designed to have minimal dependencies and never doing any system calls, and I'd like to keep it that way. Therefor the solution needs to be some sort of plugin system with callbacks. This is also the best choice for flexibility.

There is three steps to loading an image:

  • Getting the image bytes
    • file://
    • https://
    • From a static list of include_bytes!("my_icon.png")
  • Decoding the image
    • png, jpg, …
    • svg rasterization
  • Uploading it to a texture

In most cases we want the loaded image to be passed to egui::Context::load_texture, which wil hand the image off to whatever rendering backend egui is hooked up to. This will allow the image loaded to work with any egui integration.

We should also allow users to have more exotic ways of loading images. At Rerun we have a data store where we store images, and it would be great if we could just reference images in that store with e.g. ui.image(rerun://data/store/path);.

In some cases however, the user may want to refer to textures that they themselves have uploaded to the GPU, i.e. return a TextureId::User. We do this at Rerun.

Proposal

I propose we introduce three new traits in egui:

  • BytesLoader
  • ImageLoader
  • TextureLoaader

The user can then mix-and-match these as they will.

Users can register them with ctx.add_bytes_loader, ctx.add_image_loader, ctx.add_texture_loader,
and use them with ctx.load_bytes, ctx.load_image, ctx.load_texture.

We will supply good default implementations for these traits in egui_extras. Most users will never need to look at the details of these.

All these traits are designed to be immediate, i.e. callable each frame. It is therefor up to the implementation to cache any results.

They are designed to be able to suppor background loading (e.g. slow downloading of images).
For this they return Pending when something is being loaded. When the loading is done, they are responsible for calling ctx.request_repaint so that the now-loaded image will be shown.

They can also return an error.

Pending will be shown in egui using ui.spinner, and errors with red text.

Common code

enum LoadError {
    /// This loader does not support this protocol or image format.
    ///
    /// Try the next loader instead!
    NotSupported,

    /// A custom error string (e.g. "File not found: foo.png")
    Custom(String),
}

/// Given as a hint. Used mostly for rendering SVG:s to a good size.
///
/// All variants will preserve the original aspect ratio.
///
/// Similar to `usvg::FitTo`.
pub enum SizeHint {
    /// Keep original size.
    Original,

    /// Scale to width.
    Width(u32),

    /// Scale to height.
    Height(u32),

    /// Scale to size.
    Size(u32, u32),
}

BytesLoader

// E.g. from file or from the web
trait BytesLoader {
    /// Try loading the bytes from the given uri.
    ///
    /// When loading is done, call `ctx.request_repaint` to wake up the ui.
    fn load(&self, ctx: &egui::Context, uri: &str) -> Result<BytesPoll, LoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}


enum BytesPoll {
    /// Data is being loaded,
    Pending {
        /// Set if known (e.g. from a HTTP header, or by parsing the image file header).
        size: Option<Vec2u>,
    },

    /// Bytes are loaded.
    Ready {
        /// Set if known (e.g. from a HTTP header, or by parsing the image file header).
        size: Option<Vec2u>,

        /// File contents, e.g. the contents of a `.png`.
        bytes: Arc<u8>,
    },
}

ImageLoader

// Will usually defer to an `Arc<dyn BytesLoader>`, and then decode the result.
trait ImageLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        size_hint: SizeHint,
    ) -> Result<TexturePoll, TextureLoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}

trait ImagePoll {
    /// Image is loading.
    Pending {
        /// Set if known (e.g. from parsing the image header).
        size: Option<Vec2u>,
    },

    Ready {
        image: epaint::ColorImage, // Yes, only color images for now. Keep it simple.
    }
}

TextureLoader

// Will usually defer to an `Arc<dyn ImageLoader>`,
// and then just pipe the results to `egui::Context::laod_texture`.
trait TextureLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        texture_options: TextureOptions,
        size_hint: FitTo,
    ) -> Result<TexturePoll, TextureLoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}

struct SizedTexture {
    pub id: TextureId,
    pub size: Vec2u,
}

enum TexturePoll {
    /// Texture is loading.
    Pending {
        /// Set if known (e.g. from parsing the image header).
        size: Option<Vec2u>,
    },

    /// Texture is ready.
    Ready(SizedTexture),
}

Common usage

egui_extras::install_texture_loader(ctx);

ui.image("file://foo.png");

Advanced usage:

egui_ctx.add_texture_loader(Box::new(MyTextureLoader::new()));

let texture: Result<TexturePoll, _> = egui_ctx.load_texture(uri);

ui.add(egui::Image::from_uri("file://example.svg").fit_to_width());

Implementation

For version one, let's ignore cache eviction, and lets parse all images inline (on the main thread). We can always improve this in future PRs.

in egui

We just have the bare traits here, plus:

impl Context {
    pub fn add_bytes_loader(&self, loader: Arc<dyn BytesLoader>) {}
    pub fn add_image_loader(&self, loader: Arc<dyn ImageLoader>) {}
    pub fn add_texture_loader(&self, loaded: Arc<dyn TextureLoaader>) {}

    // Uses the above registered loaders:
    pub fn load_bytes(&self, uri: &str) { 
        for loaders in &self.bytes_loaders {
            let result = loader.load(uri);
            if matches!(result, Err(LoadError::NotSupported)) {
                continue;  // try next loader
            } else {
                return result;
            }
        }
    }
    pub fn load_image(&self, uri: &str, size_hint: FitTo) {}
    pub fn load_texture(&self, uri: &str, texture_options: TextureOptions, size_hint: FitTo) {}
}


/// a bytes loader that loads from static sources (`include_bytes`)
struct IncludedBytesLoader {
    HashMap<String, &'static [u8]>,
}

impl DefaultTextureLoader { } 

impl TextureLoaader for DefaultTextureLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        texture_options: TextureOptions,
        size_hint: FitTo,
    ) -> Result<TexturePoll, TextureLoadError>
    {
        let img = ctx.load_image(uri, size_hint)?;
        match img {
            ImagePoll::Pending { size } => Ok(TexturePoll::Pending { size }),
            ImagePoll::Ready { image } => Ok(ctx.load_texture(image, texture_options)),
        }
    }    
}

in egui_extras

fn install_texture_loader(ctx: &egui::Context) {
    #[cfg(not(target_os == "wasm32"))]
    ctx.egui_add_bytes_loader(Arc::new(FileLoader::new()));

    #[cfg(feature = "ehttp")]
    ctx.egui_add_bytes_loader(Arc::new(EhttpLoader::new()));

    #[cfg(feature = "image")]
    ctx.add_texture_loader(Arc::new(ImageLoader::new()));

    #[cfg(feature = "svg")]
    ctx.add_texture_loader(Arc::new(SvgLoader::new()));
}

/// Just uses `std::fs::read` directly
#[cfg(not(target_os == "wasm32"))]
struct FileLoader {
    cached: HashMap<String, Arc<[u8]>,
}

/// Uses `ehttp` to fetch images from the web in background threads/tasks.
#[cfg(feature = "ehttp")]
struct EhttpLoader {
    cached: HashMap<String, poll_promise::Promise<Result<Arc<[u8]>>>>
}

#[cfg(feature = "image")]
struct ImageCrateLoader {
    cached: HashMap<String, ColoredImage>, // loaded parsed with `image` crate
}
impl ImageLoader for ImageCrateLoader {}

#[cfg(feature = "svg")]
struct SvgLoader {
    cached: HashMap<(String, SizeHint), ColoredImage>,
}
impl ImageLoader for SvgLoader {}

Implementation notes

@YgorSouza
Copy link
Contributor

Pending will be shown in egui using ui.spinner, and errors with red text.

Maybe these could have overrides as well, for added flexibility.

pub fn pending_ui(self, add_contents: impl FnOnce(&mut Ui)) -> Self;
pub fn error_ui(self, add_contents: impl FnOnce(&mut Ui, LoadError)) -> Self;

But these would have to be evaluated before anything is drawn, so it would probably require having to call a show() method after all the builders. I suppose if users want more customization (like displaying specific error icons, downloading images only once and caching them in the data directory, logging errors etc), it's not that hard to write your own, currently. I tried my hand at that in this project.

@jprochazk
Copy link
Collaborator

jprochazk commented Sep 6, 2023

Most of the proposal is implemented, but there's work left to do:

  • Address the last comments on Managed texture loading #3297
  • Make egui_extras::FileLoader optional
  • Make load error messages shorter
  • egui::Image should support a number of different sources
    • URI
    • URI + SizedTexture - equivalent to current .image API
    • URI + Bytes (include_image! shorthand)
  • Replace old APIs with new ones, and deprecate old APIs
    • ui.image2 -> ui.image
    • Image2 -> Image
    • load_texture -> allocate_texture + deprecate load_texture
  • Loading spinner should be positioned nicely within the allocated space
  • Loading policy: Images may take a long time to load, but they shouldn't always use background threads.
    • Options:
      • Async (always use a background thread if possible)
      • Sync (always load synchronously if possible)
      • Auto (size heuristic)
    • Default should be set on Context, with override on Image.
  • Better image type detection
    • Add mime: Option<String> to BytesPoll::Ready
      • Read from Content-Type in EhttpLoader
      • Read from file extension in FileLoader
    • ImageCrateLoader should fall back to image::guess_format if mime is None
  • Add more examples
    • Add egui logo to the demo widget gallery
    • Image sizing options example
    • Combine a few examples into one image example (download_image, svg, retained_image)
  • Deprecate RetainedImage and rework examples using it to use ui.image instead
  • Try it in practice by making a PR to https://github.com/lampsitter/egui_commonmark and rerun
  • Image sizing API. Images are currently always ImageFit::ShrinkToFit, we want a more flexible API.
    • size_range, width_range, height_range, respect_aspect_ratio(bool)
    • size_fraction, width_fraction, height_fraction
    • size, width, height
    • original_size + scale
  • Improve error messages on CORS errors on web Web: Improve opaque network error message ehttp#33
  • Improve error message when there are no image loaders installed
  • Improve the "error" image UI (perhaps show a red ⚠️ with details on hover?)
  • Remove the newly added #![allow(deprecated)]
  • Deprecate and replace Button::image_and_text and similar functions

abey79 added a commit to rerun-io/rerun that referenced this issue Sep 6, 2023
### What

Add Example page to the Welcome Screen.


Fixes #3096 

<img width="1366" alt="image"
src="https://github.com/rerun-io/rerun/assets/49431240/bbe2e84e-9ade-4da8-b095-d7b0f396c26f">

### TODO

- [x] fix layout issues
- [x] display tags
- [x] have dedicated, short copy for the description: #3201

### Not included in this PR

- **WARNING**: here, we bake in a manifest with hard-coded links to RRDs
that were generated within this PR. This will lead to issue down the
line, when the RRD format changes.
  - #3212
  - #3213
- download updated manifest
  -  #3190 
- load thumbnail from the web
  - emilk/egui#3291
- provide feedback while downloading a RRD
  - #3192

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3191) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3191)
- [Docs
preview](https://rerun.io/preview/3be107e4cc6aa6758a3f22c27a79233b33f2ea6b/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/3be107e4cc6aa6758a3f22c27a79233b33f2ea6b/examples)
<!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)

---------

Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com>
@jprochazk
Copy link
Collaborator

jprochazk commented Sep 12, 2023

  • Improve error messages on CORS errors on web (maybe a ehttp PR)

When the server sends invalid CORS headers, then the browser will output only an opaque network error. This typically shows up as TypeError: Failed to fetch. According to the spec, a network error is a Response object with:

  • type set to "error"
  • status set to 0
  • an empty status message
  • an empty header list
  • a null body

Browsers may output more information to the developer console or through other means, but they may not expose this information to the page. We can't tell the user it's "likely" a CORS error, because there are a whole bunch of other possible causes for a TypeError: Failed to fetch:

  • No internet connection
  • DNS resolution failed
  • Firewall or proxy blocked the request
  • Server is not reachable
  • The URL is invalid (yes, this is just an opaque "failed to fetch"...)
  • Server's SSL cert is invalid
  • The initial get which returned HTML contained CSP headers to block access to the resource
  • A browser extension blocked the request (e.g. ad blocker)

I don't know how to improve the situation. We could list all of the possible causes, but that's a lot of stuff to throw at a user. I also don't think it really needs to be improved, because the problem is clear once you look at the devtools console, which you do often anyway.

@emilk
Copy link
Owner Author

emilk commented Sep 12, 2023

We can improve the situations like so:

  • The current error message from ehttp is huge, and contains a callstack. That callstack is unhelpful imho.
  • We can direct the users eyes to the development console

So, I suggest in ehttp we detect the TypeError and return a shorter error message, something like: "Failed to fetch. Check the developer console for details"

@lucasmerlin
Copy link
Collaborator

This is awesome!
If I understand this correctly, with these changes it should be possible to implement a, say, WgpuWasmTextureLoader, that uses createImageBitmap to decode images, removing the need to include the image crate in the wasm builds, which should reduce the wasm payload size and improve image decoding performance?

In my project I'm loading a lot of images so I implemented this with my own image abstraction, but it'd be awesome if I could switch to using the new loader traits in the future. I'd be happy to implement a WgpuWasmTextureLoader, either as part of the egui_wgpu crate or as a separate external crate.

@emilk
Copy link
Owner Author

emilk commented Sep 14, 2023

createImageBitmap could be used by a new WebImageLoader that I think would fit well into egui_extras. It should implement trait ImageLoader, and be responsible for converting raw bytes into an egui::ColorImage. That would indeed remove the need for the image crate.

That's all you would need to implement though, and it would work for any egui running on web.
The default egui texture loader will just pass the loaded ColorImage to whatever the egui backend is, i.e. wgpu, glow, miniquad, … and it will Just Work™️.

@wangxiaochuTHU
Copy link
Contributor

It's great!

Btw, is it also neccessary/possible that directly passing a grayscale image buffer (like &[u16]) to GPU, to avoid expanding it to ColorImage on CPU? In the current state of version 0.22.0, 16bit-per-pixel grayscale images are not supported. Also, 8bit-per-pixel W x H grayscale images need to be expanded to a rgba Vec<u8> with length W x H x 4 on CPU firstly and then be sent to GPU. When you capture 4K/8K images from a camera device at 30 fps and want to show them on App in realtime, too much overhead is paid for this casting.

@emilk
Copy link
Owner Author

emilk commented Sep 18, 2023

@wangxiaochuTHU For now, only ColorImage is supported in egui, i.e. 32-bit sRGBA. Adding more image types (Gray8, Gray16, etc) is definitely possible, but I see that as outside the scope of this issue (it is an optimization, after all).

@emilk emilk closed this as completed Sep 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants