Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error unexpected EOF and it stop audio record immediately but stream is not end #136

Open
Nekofoxmiu opened this issue Mar 17, 2023 · 10 comments

Comments

@Nekofoxmiu
Copy link

2023/03/17 04:36:51 INFO: Loaded cookie file mypath
2023/03/17 04:36:52 Channel: なつめえりちゃんねる / Natsume Eri ch.
2023/03/17 04:36:52 Video Title: 【雑談 】おさぎょうしよ!【なつめえり】
2023/03/17 04:36:52 DEBUG: Retrieving URLs from web DASH manifest
2023/03/17 04:36:54 TRACE: Setting itag 136 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 137 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 298 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 299 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 139 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 140 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 133 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 135 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 134 from web adaptive formats
2023/03/17 04:36:54 TRACE: Setting itag 160 from web adaptive formats
2023/03/17 04:36:54 DEBUG: Retrieving URLs from web adaptive formats
2023/03/17 04:36:54 Selected quality: 1080p60 (h264)  https://www.youtube.com/watch?v=-qE4e0H4kss
2023/03/17 04:36:54 INFO: Starting download to H:\bot\youtube video\-qE4e0H4kss__1567626896\【雑談 】おさぎょうしよ!【なつめえり】--qE4e0H4kss.f140.ts
2023/03/17 04:36:54 INFO: Starting download to H:\bot\youtube video\-qE4e0H4kss__1567626896\【雑談 】おさぎょうしよ!【なつめえり】--qE4e0H4kss.f299.ts
2023/03/17 05:33:21 DEBUG: audio1: Error with fragment 21706: unexpected EOF
2023/03/17 05:36:52 ERROR: Error retrieving player response: unable to retrieve player response object from watch page
2023/03/17 05:36:53 DEBUG: audio1: Stream is finished and highest sequence reached
2023/03/17 05:36:53 DEBUG: audio1: exiting
2023/03/17 05:36:53 DEBUG: audio-download thread closing
2023/03/17 07:19:52 DEBUG: video1: Stream is finished and highest sequence reached
2023/03/17 07:19:52 DEBUG: video1: exiting
2023/03/17 07:19:52 DEBUG: video-download thread closing
2023/03/17 07:19:52 Download Finished
2023/03/17 07:19:52 WARNING: Mismatched number of video and audio fragments.
2023/03/17 07:19:52 WARNING: The files should still be mergable but data might be missing.

It often happen after 1 hour record.
Also I am not sure it will affect or not, I am running ytarchive in node.js childprocess.
Here is my code.

 let ytdl = child_process.exec(`ytarchive.exe --trace --merge --add-metadata --no-frag-files --thumbnail --retry-frags 100 --cookies "${rootFloder}\\setting\\yt_cookies.txt" -w -o "${rootFloder}\\youtube video\\%(title)s-%(id)s" ${video_ID} best `, {
        cwd: `${rootFloder}\\exe_tool`,
        maxBuffer: 1024 * 1024 * 1024
    })
@Nekofoxmiu Nekofoxmiu changed the title Error unexpected EOF and it stop record immediately but stream is not end Error unexpected EOF and it stop audio record immediately but stream is not end Mar 17, 2023
@Kethsar
Copy link
Owner

Kethsar commented Mar 17, 2023

Hmm, I should add an option to dump the HTML when it cannot find the data we're looking for.
The EOF is not actually part of the issue, that's benign. It's the automatic 1-hour refresh encountering an issue. This causes yta to assume the stream is unavailable and just stop there. I assume video stops before the stream has actually ended as well, but takes longer to catch up since you're only running one thread.

The main thing I can think of that would cause this is it serving you some kind of verification page on that refresh attempt.

@Nekofoxmiu
Copy link
Author

Hmmmm...OK...so should I closed this issue or change the title?

@Kethsar
Copy link
Owner

Kethsar commented Mar 17, 2023

You can leave it as is for now. I might add the option later to dump the HTML on failing to extract a player response from it, and then upload a binary here for you to use. That way if it's something fixable I'll have something to go on, but if it's some kind of bot-check then we'll know you're out of luck for a while.
I have an idea of something I can do instead of hard-stopping on failure too though.

@Narratur
Copy link

After some admittedly unscientific testing, I think this has to do with using a cookies file. I’ve been running into this near 100% of the time around 1 hour in to streams recently, so I was messing around with it and tried several things including changing my IP to no effect, before noticing that dropping --cookies seemed to stop the issue. So I recorded two more streams with the exact same options except one loading the cookies file:

ytarchive 0.3.2
2023/03/21 22:33:12 INFO: Loaded cookie file E:\YouTube\DL\cookies.txt
2023/03/21 22:33:13 Channel: Subaru Ch. 大空スバル
2023/03/21 22:33:13 Video Title: 【#生スバル】バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ!!!!!!! /  biohazard HD REMASTER【ホロライブ/大空スバル】

2023/03/21 22:33:13 Stream starts at 2023-03-22T12:00:00+00:00 in 30407 seconds.
2023/03/21 22:33:13 Waiting for this time to elapse...
2023/03/22 06:59:59 DEBUG: Woke up 1 seconds early. Continuing sleep...
Stream is 15 seconds late...
Stream is 30 seconds late...
Stream is 45 seconds late...
Stream is 60 seconds late...
Stream is 75 seconds late...
Stream is 90 seconds late...
Stream is 105 seconds late...
Stream is 120 seconds late...
Stream is 135 seconds late...
Stream is 150 seconds late...
Stream is 165 seconds late...
Stream is 180 seconds late...
Stream is 195 seconds late...
Stream is 210 seconds late...
Stream is 225 seconds late...
Stream is 240 seconds late...
Stream is 255 seconds late...
Stream is 270 seconds late...
Stream is 285 seconds late...
Stream is 300 seconds late...
Stream is 315 seconds late...
Stream is 330 seconds late...
Stream is 345 seconds late...
Stream is 360 seconds late...
Stream is 375 seconds late...
Stream is 390 seconds late...
Stream is 405 seconds late...
Stream is 420 seconds late...
Stream is 435 seconds late...
Stream is 450 seconds late...
Stream is 465 seconds late...
Stream is 480 seconds late...
Stream is 495 seconds late...
Stream is 510 seconds late...
Stream is 525 seconds late...
Stream is 540 seconds late...
Stream is 555 seconds late...

2023/03/22 07:09:32 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 07:09:33 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 07:09:33 Selected quality: 1080p60 (h264)
2023/03/22 07:09:33 WARNING: Formatted filename is too long. Truncating the title to try and fix.
2023/03/22 07:09:33 INFO: Starting download to E:\YouTube\DL\puNX_86hAM0__2571697921\【#生スバル】バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.f140.ts
2023/03/22 07:09:33 INFO: Starting download to E:\YouTube\DL\puNX_86hAM0__2571697921\【#生スバル】バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.f299.ts
2023/03/22 07:33:53 DEBUG: video3: Error with fragment 1466: unexpected EOF
2023/03/22 07:34:14 DEBUG: audio1: Error with fragment 1487: unexpected EOF
2023/03/22 08:00:15 DEBUG: audio4: Error with fragment 3048: unexpected EOF
Video Fragments: 3608; Audio Fragments: 3607; Max Fragments: 3606; Max Sequence: 3606; Total Downloaded: 1.20GiB
2023/03/22 08:09:33 ERROR: Error retrieving player response: unable to retrieve player response object from watch page
2023/03/22 08:09:33 DEBUG: audio4: Stream is finished and highest sequence reached
2023/03/22 08:09:33 DEBUG: audio4: exiting
2023/03/22 08:09:33 DEBUG: audio2: exiting
2023/03/22 08:09:33 DEBUG: audio1: exiting
2023/03/22 08:09:33 DEBUG: audio3: exiting
2023/03/22 08:09:34 DEBUG: audio-download thread closing
2023/03/22 08:09:34 DEBUG: video4: Stream is finished and highest sequence reached
2023/03/22 08:09:34 DEBUG: video4: exiting
2023/03/22 08:09:34 DEBUG: video3: exiting
2023/03/22 08:09:34 DEBUG: video2: exiting
2023/03/22 08:09:34 DEBUG: video1: exiting
2023/03/22 08:09:34 DEBUG: video-download thread closing
2023/03/22 08:09:34 9; Audio Fragments: 3608; Max Fragments: 3607; Max Sequence: 3607; Total Downloaded: 1.20GiB
Download Finished
2023/03/22 08:09:34 WARNING: Mismatched number of video and audio fragments.
2023/03/22 08:09:34 WARNING: The files should still be mergable but data might be missing.
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【#生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.f140.ts to E:\YouTube\DL\【#生スバル】バイ オハザード HDリマスターするしゅばああああああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.f140.ts
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【#生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.f299.ts to E:\YouTube\DL\【#生スバル】バイ オハザード HDリマスターするしゅばああああああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.f299.ts
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【#生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.jpg to E:\YouTube\DL\【#生スバル】バイオハ ザード HDリマスターするしゅばああああああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.jpg
2023/03/22 08:09:34 INFO: Moving file E:\YouTube\DL\puNX_86hAM0__2571697921\【#生スバル】バイオハザード HDリマスターするしゅばあああ あああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.description to E:\YouTube\DL\【#生スバル】 バイオハザード HDリマスターするしゅばああああああああああああああああああああああああああああ!!!!!!! _  biohazard-puNX_86hAM0.description
2023/03/22 08:09:34 Muxing final file...

and one without it:

ytarchive 0.3.2
2023/03/21 22:31:42 Channel: Matsuri Channel 夏色まつり
2023/03/21 22:31:42 Video Title: 【 APEX 】V最協顔合わせ!【ホロライブ/夏色まつり】

2023/03/21 22:31:42 Stream starts at 2023-03-22T09:00:00+00:00 in 19698 seconds.
2023/03/21 22:31:42 Waiting for this time to elapse...
2023/03/22 03:59:59 DEBUG: Woke up 1 seconds early. Continuing sleep...
Stream is 15 seconds late...
Stream is 30 seconds late...
Stream is 45 seconds late...

2023/03/22 04:00:47 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 04:00:47 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 04:00:47 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 04:00:47 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 04:00:47 Selected quality: 1080p (h264)
2023/03/22 04:00:47 INFO: Starting download to E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f140.ts
2023/03/22 04:00:47 INFO: Starting download to E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f137.ts
2023/03/22 04:14:17 DEBUG: audio4: Error with fragment 831: unexpected EOF
2023/03/22 04:47:50 DEBUG: audio4: Error with fragment 2844: unexpected EOF
2023/03/22 05:00:46 DEBUG: video2: Error with fragment 3617: unexpected EOF
2023/03/22 05:00:48 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 05:00:48 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 05:00:48 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 05:00:48 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 05:26:41 DEBUG: audio3: Error with fragment 5173: unexpected EOF
2023/03/22 05:37:05 DEBUG: audio2: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:05 DEBUG: video4: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:12 DEBUG: audio2: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:14 DEBUG: video4: Error with fragment 5791: unexpected EOF
2023/03/22 05:37:20 DEBUG: audio2: Error with fragment 5791: unexpected EOF
2023/03/22 06:00:49 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 06:00:49 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 06:00:49 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 06:00:49 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 06:08:02 DEBUG: video1: Error with fragment 7632: unexpected EOF
2023/03/22 06:09:10 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:10 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:17 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:18 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:24 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:27 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:32 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:35 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:40 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:44 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:47 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:52 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:09:55 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:01 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:03 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:05 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:06 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:08 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:09 DEBUG: audio1: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:09 DEBUG: audio1: Fragment 7694: 10/10 retries
2023/03/22 06:10:10 DEBUG: Retrieving URLs from Android DASH manifest
2023/03/22 06:10:11 DEBUG: video4: Error with fragment 7694: unexpected EOF
2023/03/22 06:10:12 DEBUG: Retrieving URLs from Android adaptive formats
2023/03/22 06:10:12 DEBUG: Retrieving URLs from web DASH manifest
2023/03/22 06:10:12 DEBUG: Retrieving URLs from web adaptive formats
2023/03/22 06:10:12 DEBUG: audio2: exiting
2023/03/22 06:10:12 DEBUG: audio3: exiting
2023/03/22 06:10:12 DEBUG: audio4: exiting
2023/03/22 06:10:12 DEBUG: audio1: exiting
2023/03/22 06:10:12 DEBUG: video4: Stream has ended and fragment number is within two of the known max, probably not actually created
2023/03/22 06:10:12 DEBUG: audio-download thread closing
2023/03/22 06:10:12 DEBUG: video3: exiting
2023/03/22 06:10:12 DEBUG: video2: exiting
2023/03/22 06:10:12 DEBUG: video1: exiting
2023/03/22 06:10:12 DEBUG: video4: exiting
2023/03/22 06:10:12 DEBUG: video-download thread closing
2023/03/22 06:10:12 4; Audio Fragments: 7694; Max Fragments: 7693; Max Sequence: 7693; Total Downloaded: 3.77GiB
Download Finished
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライ ブ_夏色まつり】-XeRxEkENvzQ.f140.ts to E:\YouTube\DL\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f140.ts
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライ
ブ_夏色まつり】-XeRxEkENvzQ.f137.ts to E:\YouTube\DL\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.f137.ts
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライ ブ_夏色まつり】-XeRxEkENvzQ.jpg to E:\YouTube\DL\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.jpg
2023/03/22 06:10:12 INFO: Moving file E:\YouTube\DL\XeRxEkENvzQ__1951538377\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライ ブ_夏色まつり】-XeRxEkENvzQ.description to E:\YouTube\DL\【 APEX 】V最協顔合わせ!_ wえる、藍沢エマ【ホロライブ_夏色まつり】-XeRxEkENvzQ.description
2023/03/22 06:10:12 Muxing final file...

The former died at an hour despite the stream continuing while the latter was fine. Neither of these streams are members streams or actually need cookies, but I tend to leave the option in for convenience and only update the cookies file on the rare occasion I archive a members stream, so they are usually being loaded while out of date. Up until now this hasn’t been a problem outside of members streams themselves, but now I wonder if whatever refresh is happening at 1 hour is causing youtube to look at the outdated cookies for some reason.

@Kethsar
Copy link
Owner

Kethsar commented May 22, 2023

Someone else I have contact with ran into this too recently. I had him use a quick custom build that dumped the HTML on error. The image of the page is below.
Unfortunately, I have no idea what to do about this. Apparently it's not even consistent.
05-22_10-32-02

@CatsCodes
Copy link

Removing cookies so far does seem avoid the error.
I found this error again here: https://arca.live/b/vtubers/51239046 and they talk about environment variable 1024. I hope this helps.

@NeloBlivion
Copy link

NeloBlivion commented Sep 17, 2023

Also been getting this issue with cookies, but it's not the most consistent... some streams break at 1 hour (and can break later after starting another download), while others run perfectly fine for hours. And sometimes when a stream ends and the archive's finished downloading, it will show the player response error even though nothing's broken.

now I wonder if whatever refresh is happening at 1 hour is causing youtube to look at the outdated cookies for some reason.

I noticed that yt-dlp actively updates the cookies.txt file whenever you download through it; I haven't paid attention to what it changes, but perhaps this needs to be implemented here?

@NeloBlivion
Copy link

I think that's exactly what we need to implement, i've tried exporting a new cookies.txt before the hour mark and even though the error still popped up, it continued to download after (presumably by reading the file again)
image

@Kethsar
Copy link
Owner

Kethsar commented Sep 18, 2023

That is interesting, but also separate from the issue at hand. That one does appear to be cookies no longer working mid-download, but it still retrieved the actual watch page, as opposed to some random page about cookie settings. Also it does not re-read the cookies file in the middle of downloading, only once at the start of loading yta (or at the start of each loop when monitoring a channel).

According to the Golang docs: The [Cookie]Jar is used to insert relevant cookies into every outbound Request and is updated with the cookie values of every inbound Response. This means that any updated cookies are already in use in the running instance, so writing them to file won't actually help. If the cookies in the file itself were stale, it would immediately cause issues for a membership stream. I'm not sure what Youtube has been doing to break cookies on occasion like this.

I could potentially write updated cookies to the file for the hell of it, but if you are running multiple instances of yta at once, that could be messy. I'm not sure how useful that actually is either since they are already desynced from your browser cookies the moment you export and load another Youtube page.

@lbmaian
Copy link
Contributor

lbmaian commented Nov 13, 2023

Related issue for yt-dlp has a user workaround: yt-dlp/yt-dlp#8227 (comment)

A simple workaround would be to generate the cookies file in an incognito window, where the cookies are never used again in browser and therefore will never be rotated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants