Zipping ugoira after downloading original frames?

[nateify]

I am running a nightly build after this commit: 9d1e5f3

Is it possible to restore the behavior of keeping only a ZIP file when ugoira is set to original and not all of the original JPG separately? I tried a postprocessor but it didn't seem to generate any ZIP at all/

{
    "name"       : "zip",
    "compression": "store",
    "extension"  : "zip",
    "whitelist"  : ["pixiv"]
},

If it makes a difference, I have a post processor for ugoira with both "whitelist": ["pixiv","danbooru"], and "keep-files": true but when I set "keep-files": false then there is still no ZIP that gets made, and the original bare JPGs get removed - only a converted WEBM gets written.

[thatfuckingbird]

Haven't tried the new feature yet but I'd also like it to work this way. Ideally the frame timing metadata could also be included in the ZIP, which would make it possible to implement correct playback of these ZIPs as-is without any conversion. I have a suspicion this can already be done with a custom postprocessor somehow...

[mikf]

Ideally the frame timing metadata could also be included in the ZIP, which would make it possible to implement correct playback of these ZIPs as-is without any conversion

Converting to .mkv with -c:v copy is lossless, includes frame timings, and can be played back by at least mpv.

I have a suspicion this can already be done with a custom postprocessor somehow...

exec and python post processors allow you to run basically any arbitrary code, so anything is theoretically possible as long as you implement it yourself ...

[mikf]

This here kind of works, but yeah, there should be a builtin option for this and the zip post processor needs some options for where it should write its archive to.

{
    "extractor": {
        "pixiv": {
            "ugoira": "original",
            "postprocessors": [
                {
                    "name"      : "zip",
                    "filter"    : "type == 'ugoira'",
                    "keep-files": true
                },
                {
                    "name"             : "ugoira",
                    "extension"        : "mkv",
                    "ffmpeg-args"      : ["-c:v", "copy"],
                    "repeat-last-frame": false
                }
            ]
        }
    }
}

[mikf]

ff07aef

{
    "name": "ugoira",
    "mode": "archive"
}

or --ugoira zip

[nateify]

ff07aef

{
    "name": "ugoira",
    "mode": "archive"
}

or --ugoira zip

I tried playing around with it but if you could please help me with this config:

minimal_config.json

My desired goal would be;
Pixiv ugoira: Download original frames and put in archive, convert original frames to WEBM, don't keep original JPEGs - end result is only JSON + ZIP + WEBM
Danbooru: Download original ZIP that was stored on danbooru, and convert the frames inside to WEBM using the same postprocessor as pixiv (my VP9 settings)- final result is also JSON + ZIP + WEBM

It works as is with "ugoira" set to true in pixiv, I tried a different post processor order but it doesn't convert to webm for pixiv

[mikf]

You need to set ugoira for pixiv to "original" for original frames
and the code from the latest commit for "mode": "archive" to have an effect.

I've moved the ugoira post processor from pixiv to the global list after the first one and added a pixiv whitelist, and this seems to work. I end up with ZIP + WEBM + one JSON for each frame.

Enable the ugoira metadata option if you want an animation.json file inside the archive itself.

config

{
    "extractor": {
        "base-directory": ".",
        "postprocessors": [
            {
                "name": "metadata",
                "mode": "json",
                "event": "after"
            },
            {
                "name": "ugoira",
                "whitelist": ["pixiv","danbooru"],
                "ffmpeg-args": [
                    "-an",
                    "-c:v",
                    "libvpx-vp9",
                    "-vf",
                    "scale=in_range=jpeg:out_range=jpeg",
                    "-colorspace",
                    "bt470bg",
                    "-color_range",
                    "full",
                    "-color_primaries",
                    "bt470bg",
                    "-color_trc",
                    "gamma28",
                    "-deadline",
                    "good",
                    "-fps_mode",
                    "passthrough",
                    "-pix_fmt",
                    "yuv444p"
                ],
                "keep-files": true
            },
            {
                "name": "ugoira",
                "mode": "archive",
                "whitelist": ["pixiv"],
                "metadata": false,
                "keep-files": false
            },
            {
                "name": "hash",
                "blacklist": ["booru", "danbooru", "gelbooru", "safebooru", "konachan", "sankaku", "yandere", "paheal", "szurubooru", "kemonoparty", "patreon", "gumroad"],
                "hashes": "md5"
            }
        ],
        "url-metadata": "gdl_file_url",
        "version-metadata": "gdl_version",
        "extension-map": {
            "jpeg": "jpg",
            "jpe": "jpg",
            "jfif": "jpg",
            "jif": "jpg",
            "jfi": "jpg",
            "pnj": "jpg"
        },
        "pixiv": {
            "filename": "{id}_p{num}-{category}_{user[id]}.{extension}",
            "archive-format": "{id}_{num}",
            "archive": "%APPDATA%/gallery-dl/pixiv.sqlite3",
            "directory": {
                "is_bookmarked is True" : ["{category}", "{user[id]} {user[account]}", "bookmarks"],
                "" : ["{category}", "{user[id]} {user[account]}"]
            },
            "favorite":
            {
                "directory": ["{category}", "{user[id]} {user[account]}", "bookmarks"]
            },
            "ugoira": "original",
            "refresh-token": "cache",
            "tags": "original",
            "postprocessors": [
                {
                    "name": "metadata",
                    "mode": "modify",
                    "event": "init",
                    "fields": {
                        "extractor_url": "https://www.pixiv.net/en/artworks/{id!s}"
                    }
                }
            ]
        },
        "danbooru": {
            "filename": {
                "len(tags_artist) > 3": "{id}-{category}_multiple.{extension}",
                "": "{id}-{category}_{tag_string_artist}.{extension}"
            },
            "archive": "%APPDATA%/gallery-dl/danbooru.sqlite3",
            "ugoira": true,
            "metadata": true,
            "postprocessors": [
                {
                    "name": "metadata",
                    "mode": "modify",
                    "event": "init",
                    "fields": {
                        "extractor_url": "https://danbooru.donmai.us/posts/{id!s}"
                    }
                }
            ]
        },
        "generic": {
            "enabled": false
        },
        "path-restrict": {
            "[": "",
            "]": "",
            "'": "",
            ",": "_",
            " ": "_",
            ":": "",
            "/": "_",
            "\\": "_",
            "\"": "",
            "*": "",
            "?": "",
            "<": "",
            ">": "",
            "|": ""
        }
    },
    "output": {
        "progress": true,
        "ansi": true
    },
    "downloader": {
        "mtime": false,
        "http": {
            "retry-codes": [408, 430]
        },
        "ytdl": {
            "module": "yt_dlp",
            "forward-cookies": true,
            "format": "bv*+ba/b"
        }
    }
}

[Corrupt-Specturion]

How do I disable WebM creation (or, preferably, make it do -c copy into an MKV container instead), and what units do the delay values in animation.json represent?

"extension": "mkv",
"ffmpeg-args": ["-c:v", "copy"] / "ffmpeg-args": ["-c", "copy"]

doesn't work, as it just creates an MKV with queued JPEG/PNG/GIF images (as if I'd opened multiple video files in MPV) instead of a continuous stream, and still creates a WebM alongside the MKV.

[Hrxn]

Well, how do you get WebM in the first place? Maybe you should check your post-processors, and clean them up?

The post-processor object you probably want is just this:

        {
            "name": "ugoira",
            "extension": "mkv",
            "ffmpeg-args": ["-c", "copy"],
            "libx264-prevent-odd": false,
            "repeat-last-frame": false
        }

Also, as mikf mentioned, you also should have the mkvmerge binary in your PATH, then you should get the correct-est Ugoiras possible...

[Corrupt-Specturion]

Well, how do you get WebM in the first place? Maybe you should check your post-processors, and clean them up?

Beats me.

"extractor":
    {
        "base-directory": "./gallery-dl/",
        "parent-directory": true,
        "postprocessors":
        [
            {
                "name": "metadata",
                "mode": "json",
                "event": "after"
            },
            {
                "name": "ugoira",
                "whitelist": ["pixiv","danbooru"],
                "keep-files": true
            },
            {
                "name": "ugoira",
                "mode": "archive",
                "whitelist": ["pixiv"],
                "metadata": true,
                "keep-files": false,
                "extension": "mkv",
                "ffmpeg-args": ["-c", "copy"]
            }
        ],

"pixiv":
        {
            "refresh-token": "REDACTED",
            "avatar": true,
            "background": true,
            "tags": "original",
            "ugoira": "original",
            "include": "avatar,background,artworks",
            "metadata": true,
            "tags": "original",
            "postprocessors":
            [
                {
                    "name": "metadata",
                    "event": "post,skip",
                    "filename": "{id}.json"
                }
            ]
        },

"ugoira":
        {
            "mode": "archive",
            "metadata":"true"
        },

I've also got MKVToolNix (mkvmerge) installed now (and in PATH), and it still does the same thing.

[mikf]

           {
               "name": "ugoira",
               "whitelist": ["pixiv","danbooru"],
               "keep-files": true
           },

This creates a .webm file.

                "mode": "archive",

"ugoira":
      {
           "mode": "archive",
           "metadata":"true"
       },

... and these tell ugoira post processors to generate ZIP archives, albeit with non-zip filename extrensions.

Remove them and keep the -c copy one and it should generate a proper, playable file.

---

edit: Here are your cleaned up pixiv settings.

Pixiv Config

        "pixiv":
        {
            "refresh-token": "REDACTED",
            "tags": "original",
            "ugoira": "original",
            "include": "avatar,background,artworks",
            "metadata": true,
            "postprocessors":
            [
                {
                    "name": "metadata",
                    "event": "post,skip",
                    "filename": "{id}.json"
                },
                {
                    "name": "ugoira",
                    "metadata": true,
                    "keep-files": false,
                    "extension": "mkv",
                    "ffmpeg-args": ["-c", "copy"]
                }
            ]
        },

[Joebugg]

Too bad that https://i.pximg.net/img-zip-ugoira/img/****/**/**/**/**/**/123456789_ugoira1920x1080.zip doesn't include the content of https://www.pixiv.net/ajax/illust/123456789/ugoira_meta?lang=en&version=39df2eee99db17d6580ec18ad96e6d80ec3d183a or https://www.pixiv.net/ajax/illust/123456789/ugoira_meta?lang=en or we wouldn't even be needing to reZIP it, right?

Without the frame data, Ugoira are a mess to reencode. Without VFR and frame lengths list, they will just play at whatever speed the encoder defaulted to, per second. It's the classic "Ugoira conversion runs too fast" syndrome.

[mikf]

Ugoira .zip archives contain lower-quality frames than when downloading them as individual files. See #6056.

Having mkvmerge available when encoding to WEBM or MKV ensures accurate frame time codes, even for Ugoira with varied frame delays.