SmugMug Source files don't download (RAW/DNG)

[cobetts]

Hi! I was able to download the JPG files super quick! However, anything that's uploaded with SmugMug Source did not pull the raw down with the JPG as well. I don't see anything in the documentation mentioning Source, so I'm not sure if this is supported. I've attached the [store] portion of my config below.

[store]
destination = "/Users/***/Documents/smugmug-full-backup"
write_csv = true
concurrent_albums = 5
concurrent_downloads = 10

[tommyblue]

hi @cobetts , I didn't even know what source is 😅
I don't have that service, thus I don't have access to the APIs. If you want to investigate the APIs and send me accurate documentation, I can try to implement something

[cobetts]

@tommyblue Fair haha. Source is their additional add-on that stores RAW files alongside JPGs. I searched through documentation and didn't find a specific reference, but their support team was able to provide details on how the RAW files can be accessed and documentation. Does this help?

You can actually access RAW files in SmugMug Source via the Components endpoint:
https://api.smugmug.com/api/v2/library/asset/IMAGEKEY!components

For files in SmugMug Source, that should allow you access to the RAW file via the DownloadUrl element and if we created a JPG preview for it, also that JPG.

[tommyblue]

@cobetts it could, in fact. The problem is that my components don't include any raw, I need help from your side:

• sign in to smugmug
• with the same browser go to https://api.smugmug.com/api/v2!authuser
• scroll down and look for Uris > UserAlbum, then click on the link (in my case is https://api.smugmug.com/api/v2/user/tommyblue!albums)
• you'll get a list of albums, click on an album that contains raw images
• scroll down and click on Uris > AlbumImages
• you'll get the list of images, click on an image that is a raw or contains a raw
• scroll down and click on Uris > Components
• verify you get the components, including the raw image
• scroll to the bottom and open the JSON section
• paste here the JSON content, hiding any personal/secret info (e.g. the urls contain hashes and photo "keys", you can replace all of them with XXXXX before pasting here)

[tommyblue]

To clarify the goal, this is the component I see (under Response in the JSON):

"Component": [
            {
                "ComponentType": "i",
                "ComponentKey": "KKKKKKKK",
                "DownloadUrl": "https://photos.smugmug.com/photos/XXXXXXX/0/YYYYYYYYY/D/XXXXXXX-D.jpg",
                "Extension": "JPG",
                "FileName": "photo.jpg",
                "FileSize": 10000,
                "MD5": "1234567890",
                "Url": "https://photos.smugmug.com/photos/XXXXXXX/0/ZZZZZZZ/O/KKKKKKK.jpg",
                "Uri": "/api/v2/library/asset/KKKKKKKK/component/i/KKKKKKKK",
                "UriDescription": "Representation of a single component of an asset",
                "Uris": {
                    "Image": {
                        "Uri": "/api/v2/image/KKKKKKKK-0",
                        "Locator": "Image",
                        "LocatorType": "Object",
                        "UriDescription": "Image by key",
                        "EndpointType": "Image"
                    }
                }
            }
        ],

I need to see the same structure for a RAW file, to understand if I can extract and download the file

[cobetts]

Hi @tommyblue thank you for the clear walkthrough! It looks like for images with RAW files there are two components stored separately. One for the JPG and one for the RAW. I have found the JSON you requested and added it below:

 "Component": [
            {
                "ComponentType": "i",
                "ComponentKey": "XXXXX",
                "DownloadUrl": "https://photos.smugmug.com/photos/XXXXX.jpg",
                "Extension": "JPG",
                "FileName": "XXXXX.jpg",
                "FileSize": 13355583,
                "MD5": "XXXXX",
                "Url": "https://photos.smugmug.com/photos/XXXXX.jpg",
                "Uri": "/api/v2/library/asset/XXXXX/component/i/XXXXX",
                "UriDescription": "Representation of a single component of an asset",
                "Uris": {
                    "Image": {
                        "Uri": "/api/v2/image/XXXXX",
                        "Locator": "Image",
                        "LocatorType": "Object",
                        "UriDescription": "Image by key",
                        "EndpointType": "Image"
                    }
                }
            },
            {
                "ComponentType": "svi",
                "ComponentKey": "XXXXX",
                "DownloadUrl": "https://www.smugmug.com/smugvault/XXXXX",
                "Extension": "RAF",
                "FileName": "XXXXX.raf",
                "FileSize": 40556112,
                "MD5": "XXXXX",
                "Url": "https://www.smugmug.com/smugvault/XXXXX",
                "Uri": "/api/v2/library/asset/MV7ZL7c/component/svi/XXXXX",
                "UriDescription": "Representation of a single component of an asset"
            }
        ],

[tommyblue]

Thanks @cobetts , I asked in the smugmug forum the possible values of the "ComponentType" field. We must understand how to clearly identify raw images

[cobetts]

Would the component type not be “svi”? I checked multiple RAW files and all of them were located in that ComponentType. I’m pretty sure “svi” just stand for where it’s stored, which all of my RAW files I checked are under /smugvault. The RAF extension is the RAW format from a Fujifilm camera.

[tommyblue]

@cobetts I also think "svi" is what I should look for, but please make this test. In this zip file you'll find a Panasonic RAW:
P1055113.RW2.zip
Would you mind trying to upload it and look if the ComponentType JSON is still "svi"?

[cobetts]

Hi @tommyblue I added the supplied file and checked the API, the ComponentType was still "svi", see below. I also checked a .DNG (from iPhone) and .ARW (from Sony) in addition to the .RAF (from FujiFilm) I sent originally and they all are a ComponentType of "svi" with the same JSON structure. Obviously, the "Extension" changes depending on the type of RAW file.

            {
                "ComponentType": "svi",
                "ComponentKey": "XXXXX",
                "DownloadUrl": "https://www.smugmug.com/smugvault/XXXXX",
                "Extension": "RW2",
                "FileName": "P1055113.rw2",
                "FileSize": 20067328,
                "MD5": "XXXXX",
                "Url": "https://www.smugmug.com/smugvault/XXXXX",
                "Uri": "/api/v2/library/asset/J55GsZg/component/svi/XXXXX",
                "UriDescription": "Representation of a single component of an asset"
            }

[tommyblue]

nice, I think we now know what to do to support raw files. I'll try to find some time do implement that

[tommyblue]

@cobetts would you mind trying #64 ?
I cannot test it not having Source. Add download_raw=true under store in the config file to enable it.
Use DEBUG=1 to get additional info, in particular you should check the Component => <url> line to understand if it's working. In my case the url is always empty but I don't understand if there's an error in my code or what...

[cobetts]

It looks like it's missing the "smugmug-backup" file. When running with download_raw=true from the original smugmug-backup file I copied in, nothing changes. Running with DEBUG=1 I'm not seeing it reference the Component.

[tommyblue]

@cobetts not sure we're on the same page: are you building the binary from the PR I linked or something different? The "smugmug-backup" file is not present in the source, you must either build it or download a release binary, not present here. If you need the binary, I can make it for you

[cobetts]

@tommyblue I'm not super knowledgeable with how this works, so if you could compile that for me, that would be great.

[tommyblue]

@cobetts here it is https://github.com/tommyblue/smugmug-backup/releases/tag/v1.6.0-alpha1

[tommyblue]

@cobetts any chance to test it?

[cobetts]

Hi @tommyblue Sorry, I had not gotten to it until tonight. Thank you for compiling it for me. When running DEBUG=1 I get lots of empty component calls, with no RAW images downloading.

[tommyblue]

damn, I need to check the code. Thanks for the test

[tommyblue]

@cobetts I found an error in the SmugMug APIs: the doc page for AlbumImage reports that the response must include the Components object, but the real API request doesn't get such response. I sent a message to Smugmug support, I'll update the issue if I get a reply

[tommyblue]

@cobetts I got a reply from the SmugMug support and they said Components APIs are not available for third party apps, although their doc isn't really clear on that matter (they're going to fix it). So, unfortunately, we cannot support RAW images 😢
The support guy was very kind and forwarded the feature request to the product team, but even if they decide to support the RAW images download, it won't happen soon

[cobetts]

@tommyblue Bummer! Sorry for sending in an impossible task. The response I had gotten from them said it was possible otherwise I wouldn't have asked. Thanks for trying!

[tommyblue]

No problem, we're ready when they'll eventually decide to open them 😄