The document provides the vision for the desired state of assets management in Magento. It covers currently existing managed Magento environments, or the ones envisioned in the future:
- Magento Cloud with Fastly CDN
- Magento Cloud + AEM Assets as DAM provider
- Including Dynamic Media with Akamai CDN
Disclaimer: Other types of setup (e.g., on-prem + different CDN/DAM) can be supported in the similar fashion. Current document just covers the above systems in more details to not grow in size exponentially. There is no intention to limit usage of other types of setup.
- Terminology
- Scenarios
- Asset Transformations
- Watermarking
- Placeholders
- Risks
- Questions
- Breaking Changes
- Asset - anything that exists in a binary format and comes with the right to use.
- Image, video are types of assets
- DAM (Digital Asset Management) - a system responsible for asset management (store, create, update, delete, organize)
- CDN (Content delivery network) - a system responsible for content delivery. In scope of this document, for delivery of images and video.
- Asset delivery - providing publicly available URL for the asset. Usually involves CDN. This is different from asset management, which focuses on the admin side of interactions with the assets, while delivery is the cliend side of it.
- Image transformation - resizing, rotation, watermarking and other automated transformations on an original image.
- Image transformation is responsibility of either DAM or CDN. This includes resizing, rotation, watermarking and so on.
- Client should be able to fetch transformed images by its original URL with additional parameters
- Transformation parameters may differ based on the CDN/DAM providing the transformation service
- Magento back office (Magento Admin) - in scope of this document, a system for products and categories management.
- Links assets to products and categories
- Provides basic functionality for images and video transformation. Long-term, should be offloaded to specialized systems (DAM, CDN)
- Catalog Store-Front Application - application providing product and category information suitable for store-front client scenarios.
- Serves URLs of original assets.
- Should not be aware of asset management functionality (no knowledge about underlying integration with external DAM systems).
- Might have Base CDN URL. This is similar to current Base Media URL and serves the same purpose, but might exist in case both sources of assets should be supported (Magento and CDN).
Detailed steps of the asset flow are described below.
- Step 1: Asset is uploaded to DAM
- Step 2: DAM may perform transformations
- Step 3: Admin opens product edit page
- Step 4: Admin selects necessary image using asset picker UI (provided as part of DAM integration) and saves the product
- Image is linked to the product as provided by DAM
- Image path relative to DAM base URL is stored as image path
- Asset is assigned to the product in DAM
- Step 5: Asset relation is synced to Storefront service in a form of full URL to the asset, as part of product data
- URL to the original image is synced
- Each asset may also include specialized type:
thumbnail,small, etc. The type is not related to the image size or quality and only helps the client understand where the image is supposed to be displayed - The asset itself is not synced to the Storefront and is stored in its original location (in the system responsible for its management: either external DAM or Magento Back Admin)
- Step 6: User opens PDP (product details page)
- Step 7: PWA application loads and requests product details from GraphQL application
- Step 8: GraphQL application requests product details (including asset URLs) from the SF service.
- SF service returns full image URL of the original image
- Step 9: PWA application fetches asset from the CDN by the provided URL
- PWA may include transformation parameters
- Step 10 (optional): CDN fetches the asset from the origin, if not cached
- Step 11 (optional): Origin (DAM) may perform necessary transformations
- Step 12 (optional): CDN may perform necessary transformations
Asset transformation is responsibility of either DAM or CDN, depending on the system setup. Both services may provide some level of transformations. CDN usually provides more basic transformations (resize, rotation, crop, etc), while DAM may provide more smart transformations (e.g., smart crop). Client application (PWA) does not care which part is performing transformations, it must only follow supported URL format when including transformation parameters.
In the first phase (and until further necessary) it is assumed that client application (PWA) is responsible for knowing format of transformed image URL, and so client developer should have knowledge of which DAM/CDN it works with. As a more smart step, backend application (via GraphQL) can provide information necessary for URL formatting.
Image transformation can be done by web server on the Magento side (e.g., Store-Front), but this looks less efficient than using a CDN. In the first phase, this is not going to be supported. This may cause performance issues on pages with many assets loaded (such as product listing), but it is assumed that production systems should use CDN with image transformation support. Scenario with no CDN is assumed to be a development workflow and loading unresized images is considered less critical in this situation, especially assuming Images Upload Configuration allows to cap image size.
Note: watermarking is a special case of asset transformation. See next section for its coverage.
Magento supports the following transformations for images:
- resize
- rotate
- set/change quality
- set background
See \Magento\Catalog\Model\Product\Image for details.
Fastly provides image transformation features with Fastly IO:
- Convert format
- Rotation
- Crop
- Set background color
- Set quality
- Montage (Combine up to four images into a single displayed image.)
and others. See https://docs.fastly.com/en/guides/image-optimization-api for detailed supported parameters.
Provided features fully cover Magento capabilities.
AEM Assets work in integration with Dynamic Media (DM) to deliver assets, and DM provides asset transformation capabilities. DM uses Akamai as CDN. Does it provide additional image transformation capabilities? Are those even needed taking into account that DM provides broad range of features?
- DefaultImage - it allows the client to specify default image. Might be useful for placeholder implementation.
- Resize: crop, scale, size, etc
- Rotate and flip
- Quality
- Background color
- Change image format
and many others. See Dynamic Media Image Serving and Rendering API - Command reference for more details.
In addition, Image Presets with image adjustments can be created in advance and then requested by the client.
Watermarking is a special case of image transformation. It is special in a way that the server decides when and which watermark to apply. All other transformations are initiated by the client.
Applied during image transformations.
See \Magento\Catalog\Model\Product\Image for details.
See "Overlay" in https://docs.fastly.com/en/guides/image-optimization-api
Overlay must be specified via x-fastly-imageopto-overlay header rather than via a URL parameter, which allows the server control it.
To make sure UX is acceptable, the workflow should be described in more details, taking into account Magento scopes.
Watermarking
Scoping?
Placeholders are used in the following cases:
- A product has no image assigned to it.
- An image assigned to the product is unavailable (due to delays caused by distributed architecture or accidentally)
Expectations from the placeholder delivery:
- Client application can distinguish a placeholder from a real product image. This gives the client an ability to control what (if anything) should be displayed as placeholder.
- Placeholder image should be possible to cache on the client side and reuse in different places.
- If possible, the system (Magento?) should provide placeholder URL, so the client can use it if desired (not all clients may want to use admin-configured placeholders).
Based on the above, Catalog service should not handle placeholders and try to provide them if product image is absent. Instead, product image data should be empty, and system configuration GraphQL API should provide URL for placeholders by type. To fully support external DAM systems, it should be possible to specify placeholder URLs in Magento system configuration, in addition to existing "upload file" option. Configuration API should return full URL to the placeholder when requested by the client. This can be:
- URL pointing to Magento application for default placeholder image (example:
https://static.base.url/catalog/product/placeholder.jpg) - URL pointing to uploaded placeholder (example:
https://media.base.url/catalog/product/placeholder.jpg) - URL pointing to external DAM/CDN (example:
https://external.dam.com/my-magento/catalog/product/placeholder.jpg)
The client should not assume the placeholder base URL is the same as Media Base URL. It is responsibility of the client application to handle situation where an image assigned to a product is absent in the storage and handle this situation gracefully.
Magento allows to specify a placeholder image for each type of the product image (base, small, thumbnail, swatch) per store view.
Magento validates whether the file is present on the disk at the moment of URL generation, and provides URL to a placeholder image if the file is absent. This covers two different use cases:
- The image has never been specified for the product, so there is no image available and no file is actually validated for presence.
- The imaghe has been assigned to the product, but then has disappeared from the storage (due to a mistake, failure or otherwise).
Existing behavior limits the system to using local storage for the assets as it has a hard dependency on the local file system, and makes it difficult to fetch images from external DAM systems. As mentioned before, placeholder URLs should be supported.
When integrating with AEM Assets or another external DAM system:
- Content author uploads and publishes placeholder asset in the DAM system
- Magento admin references the published placeholder URL in Magento system configuration
- Client application requests the placeholder URL via Magento GraphQL API to place where a product image can't be loaded
In addition, Dynamic Media supports "DefaultImage", which allows the client to specify default image. This can be used as another fallback path in case an image is absent.
This section summarizes potential issues with certain scenarios.
Risk: Changing Base Media URL will require full resync of Catalog to update image URLs. Mitigation: Follow this procedure for changing Base Media URL:
- Setup infrastructure so that new URL redirects to the old URL if asset doesn't exist.
- Wait for the sync to finish.
- Disable/drop old URL.
Alternative: Sync path to the assets and Base Media URL, add logic of composing full URL to the Storefront application.
The decision of selecting full sync approach is based on:
- It is expected that Base Media URL is a rare event.
- Mitigation steps are straightforward and may be necessary anyways.
- Additional logic adds complexity. In this case, potential complexity is in:
- Fallback for websites -> stores -> store views
- In the future, Magento needs to be integrated with DAM. In case of integration in mixed mode (where both Magento-managed assets and DAM-managed assets are supported), the logic of composing the URL becomes even more complex as it requires Storefront to have knowledge of assets relation to products.
Base on the above, it looks more reasonable to have simple logic on Storefront side than trying to avoid full resync.
Risks: Systems with no CDN/DAM will get performance hit on pages with many assets (such as product listing). Mitigation: Possible options:
- Upload pre-resized thumbnail images.
- Utilize "Images Upload Configuration" to ensure huge images are not uploaded.
In case the feature is highly requested, it can be implemented on the level of web server. There was PoC made for resizing imaged by Nginx.
Risks:
- Client application may use incorrect transformed URL.
- Clients may need to be rewritten when switching to a different CDN/DAM.
- Clients may be broken when switching to a different CDN/DAM.
Mitigation: Align client development with infrastructure of the back office.
Alternative: Provide information necessary for forming correct transformation URL by Storefront API.
Then client can rely on this information to build correct transformed URL dynamically and doesn't need to know about CDN/DAM used behind. Example:
transformationParams:
- width: w
- height: h
- quality: q
url = origUrl + '?' + transformationParams[width] + '=100&' + transformationParams[height] + '=100&' + transformationParams[quality] + '=80'
> https://my.dam.com/catalog/product/my/product.jpg?w=100&h=100&q=80
- Do we sync full image URL to SF or provide Base CDN URL as configuration for the store?
- What do wee do with secure/unsecure URLs in case of full URL?
- Current GraphQL returns transformed images instead of originals (❗ validate this. Might be just broken URL, as it's not clear which exact transformation GraphQL provides from the entire list). Problems with this approach:
- Transformation depends on Magento theme, which is beyond GraphQL scope (GraphQL knows nothing about old Magento themes, like Luma).
- GraphQL clients can't perform or request necessary transformation, only predefined (by irrelevant Magento themes) transformations are provided.