Downloader fails with error when download archive is used and video exists in archive

[whargrove]

When using extra_arg("--download-archive") (and next as path to the archive file), the downloader will fail with error if the file already exists in the archive. This is caused by yt-dlp returning null in stdout when -J is specified and the video is already in the download archive.

For example:

% yt-dlp -J --no-simulate --no-progress "https://www.youtube.com/watch?v=Y2pvhyK0sfE" -f mp4 -o "channels/%(channel)s/%(upload_date>%Y-%m-%d)s - %(title)s.mp4" --download-archive ./archives/@BenHorne/archive.txt
null

% echo $?
0

When using the library it fails with a json error: "json error: invalid type: null, expected struct SingleVideo"

Recommend that YoutubeDlOutput be extended with new symbol: ArchivedSingleVideo if stdout contains "null" and --download-archive arg is used.

[onkoe]

Hey, @whargrove! I'm not the maintainer, but out of frustration with these errors, I ended up trying to write my own yt-dlp library. However, yt-dlp emits output as JSON. You can check it with yt-dlp --dump-json! Working with it can be a massive pain - mostly because of the JSON parsing you can see above. In writing my own version of the lib, I inevitably found json_typegen, a tool which automatically creates Rust structs from JSON input.

That sounds annoying to me, but I figured out that you either have to choose between:

1. Writing ALL OF THE STRUCTS YOURSELF (!!!) or
2. Generating them :)

Anyone with the knowledge would want to generate them. You can do so with the json_typegen tool I mentioned earlier. I think, though, that in making the library, the maintainer removed the generated serde_json::Value typed fields to allow easier use for... users! However, this change seems to result in weird JSON errors when values don't fit their expected type. It certainly makes sense that the errors bubble up from serde_json, then! In fact, I found your error here!

Since no one could ever possibly have enough time to maintain each type of value and check every website type, it's probably a good idea to utilize the Value type and just provide some tools to help users work around it. In particular, documenting the usage of TryFrom on the Value type would probably help to make the library keep its correctness without eluding utility to its users.

@GyrosOfWar, would you be interested in taking another look at this? I think it would make your great library much more useful! :)