Skip to content
This repository has been archived by the owner on Oct 18, 2024. It is now read-only.

Automatic extraction of subtitles not working #711

Closed
kevindd992002 opened this issue Jan 17, 2020 · 38 comments
Closed

Automatic extraction of subtitles not working #711

kevindd992002 opened this issue Jan 17, 2020 · 38 comments

Comments

@kevindd992002
Copy link

I have automatic extraction of embedded subtitles enabled. I refreshed all metadata of my tv show library. So I'm expecting everything with an embedded subtitle to at least have an accompanying srt file if SZ cannot search for a better one from the subtitle providers, right? One of the first shows that was refreshed was Arrow but looking at the media files with embedded PGS subtitles, I still don't see any SRT files for them.

Also, when embedded subtitles extraction happens, it creates an SRT file and removes the embedded subs from the media files themselves, right?

What logs do you need to look at?

@kevindd992002
Copy link
Author

Here are the PMS Plugin Logs if you need them:

https://www.dropbox.com/s/rd8cela7dii9160/PMS%20Plugin%20Logs.zip?dl=0

@kevindd992002
Copy link
Author

One example media filename that doesn't have an SRT:

Arrow (2012) - S08E01 - Starling City - [WEBDL-1080p] [EAC3 5.1ch] [h264 8bit] - Sonarr.mkv

Even though the release group was not parsed correctly by Sonarr, I have the Sonarr API in the SZ settings so SZ should be able to query for the original filename in Sonarr.

In the logs I posted above, I had the "Don't search for subtitles of a language if there are embedded subtitles inside the media file (MKV/MP4)?" setting checked so I thought that was the one causing it. I unchecked that and refreshed the metadata again but the result is still the same.

@pannal
Copy link
Owner

pannal commented Jan 18, 2020

Also, when embedded subtitles extraction happens, it creates an SRT file and removes the embedded subs from the media files themselves, right?

  • SZ doesn't modify the media files (e.g. it doesn't remove any embedded subtitle files; that might change or be a feature in the future)

  • PGS/image based subtitles might be converted to SRT, but it's highly unlikely

@pannal
Copy link
Owner

pannal commented Jan 18, 2020

You seem to be using mediainfo, which results in an error with a couple of mp4 files in your logs.

Can you post the output of mediainfo "Arrow (2012) - S08E01 - Starling City - [WEBDL-1080p] [EAC3 5.1ch] [h264 8bit] - Sonarr.mkv" as well as mediainfo --Inform="Text;-%ID%_%Title%" "Arrow (2012) - S08E01 - Starling City - [WEBDL-1080p] [EAC3 5.1ch] [h264 8bit] - Sonarr.mkv"?

Thanks!

@pannal
Copy link
Owner

pannal commented Jan 18, 2020

And do the same for Arrow (2012) - S08E03 - Leap of Faith - [WEBDL-1080p] [AAC 2.0ch] [h264 8bit] - Sonarr.mp4 please.

@kevindd992002
Copy link
Author

Also, when embedded subtitles extraction happens, it creates an SRT file and removes the embedded subs from the media files themselves, right?

* SZ doesn't modify the media files (e.g. it doesn't remove any embedded subtitle files; that might change or be a feature in the future)

* PGS/image based subtitles might be converted to SRT, but it's highly unlikely

Oh ok. So no way of extracting PGS subtitles and saved to SRT then?

Here's the mediainfo output:

root@ebc2133f420e:/data/tvshows/Arrow/Season 08# mediainfo "Arrow (2012) - S08E0                                                                                                                  1 - Starling City - [WEBDL-1080p] [EAC3 5.1ch] [h264 8bit] - Sonarr.mkv"
General
Unique ID                                : 3281137500970528958009618764800053512                                                                                                                  54 (0xF6D868DEAC1F420FDC7AF54A7A65A756)
Complete name                            : Arrow (2012) - S08E01 - Starling City                                                                                                                   - [WEBDL-1080p] [EAC3 5.1ch] [h264 8bit] - Sonarr.mkv
Format                                   : Matroska
Format version                           : Version 4 / Version 2
File size                                : 2.97 GiB
Duration                                 : 42 min 25 s
Overall bit rate                         : 10.0 Mb/s
Encoded date                             : UTC 2019-10-16 07:16:37
Writing application                      : mkvmerge v36.0.0 ('Is That Jazz?') 64                                                                                                                  -bit
Writing library                          : libebml v1.3.9 + libmatroska v1.5.2

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 42 min 25 s
Bit rate mode                            : Constant
Bit rate                                 : 9 374 kb/s
Nominal bit rate                         : 10 000 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.189
Stream size                              : 2.78 GiB (94%)
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : E-AC-3
Format/Info                              : Enhanced Audio Coding 3
Codec ID                                 : A_EAC3
Duration                                 : 42 min 25 s
Bit rate mode                            : Constant
Bit rate                                 : 640 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 187.500 FPS (256 SPF)
Compression mode                         : Lossy
Stream size                              : 194 MiB (6%)
Language                                 : English
Service kind                             : Complete Main
Default                                  : Yes
Forced                                   : No

Text
ID                                       : 3
Format                                   : UTF-8
Codec ID                                 : S_TEXT/UTF8
Codec ID/Info                            : UTF-8 Plain Text
Duration                                 : 42 min 18 s
Bit rate                                 : 70 b/s
Count of elements                        : 689
Stream size                              : 22.0 KiB (0%)
Title                                    : SDH
Language                                 : English
Default                                  : No
Forced                                   : No
root@ebc2133f420e:/data/tvshows/Arrow/Season 08# mediainfo "Arrow (2012) - S08E03 - Leap of Faith - [WEBDL-1080p] [AAC 2.0ch] [h264 8bit] - Sonarr.mp4"
General
Complete name                            : Arrow (2012) - S08E03 - Leap of Faith - [WEBDL-1080p] [AAC 2.0ch] [h264 8bit] - Sonarr.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 3.03 GiB
Duration                                 : 42 min 15 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 10.3 Mb/s
Writing application                      : Lavf58.33.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 42 min 15 s
Bit rate mode                            : Constant
Bit rate                                 : 9 375 kb/s
Nominal bit rate                         : 10 000 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.976 (24000/1001) FPS
Minimum frame rate                       : 23.810 FPS
Maximum frame rate                       : 24.390 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.189
Stream size                              : 2.77 GiB (91%)
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio #1
ID                                       : 2
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format profile                           : LC
Codec ID                                 : mp4a-40-2
Duration                                 : 42 min 15 s
Bit rate mode                            : Variable
Bit rate                                 : 256 kb/s
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 76.6 MiB (2%)
Language                                 : English
Default                                  : Yes
Alternate group                          : 1

Audio #2
ID                                       : 3
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Codec ID                                 : ac-3
Duration                                 : 42 min 15 s
Bit rate mode                            : Constant
Bit rate                                 : 640 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 193 MiB (6%)
Language                                 : English
Service kind                             : Complete Main
Default                                  : No
Alternate group                          : 1

Text
ID                                       : 4
Format                                   : Timed Text
Muxing mode                              : sbtl
Codec ID                                 : tx3g
Duration                                 : 41 min 37 s
Bit rate mode                            : Variable
Bit rate                                 : 83 b/s
Stream size                              : 25.3 KiB (0%)
Title                                    : SDH
Language                                 : English
Default                                  : Yes
Forced                                   : No
Alternate group                          : 3

@kevindd992002
Copy link
Author

I'm doing all these because the Plex seems to be transcoding the video when the subtitles are PGS.

@pannal
Copy link
Owner

pannal commented Jan 18, 2020

Please also do mediainfo --Inform="Text;-%ID%_%Title%" "FILENAME" for both of them.

Edit: I can't see any PGS subtitle inside the Starling City one. Is that not the one you're referring to?
Edit: Corrected --Inform parameter

@pannal
Copy link
Owner

pannal commented Jan 18, 2020

I'm doing all these because the Plex seems to be transcoding the video when the subtitles are PGS.

It has to. It can only direct play videos with text based subtitles. Otherwise it has to overlay the bitmap based subtitle over the video stream, which can only be done by transcoding.

@pannal
Copy link
Owner

pannal commented Jan 18, 2020

Was Arrow (2012) - S08E01 - Starling City - [WEBDL-1080p] [EAC3 5.1ch] [h264 8bit] - Sonarr.mkv maybe an upgrade to a previously available version of that episode?
An upgrade that doesn't change the filename will not necessarily trigger an update for all metadata agents, and thus won't necessarily trigger SZ. (It has no PGS)

@kevindd992002
Copy link
Author

I get this for S08E01: -3_SDH
And this for S08E03: -4_SDH

Oops, you can disregard S08E01 if it doesn't have a PGS sub. Sorry about that. This is the one I'm sure about that has a PGS:

Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv

And yes, almost all of these were recently updated by Sonarr. So if an update happens, how can SZ update the sub as well?

@mouzzampk2014-zz
Copy link

mouzzampk2014-zz commented Jan 18, 2020

I get this for S08E01: -3_SDH
And this for S08E03: -4_SDH

Oops, you can disregard S08E01 if it doesn't have a PGS sub. Sorry about that. This is the one I'm sure about that has a PGS:

Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv

And yes, almost all of these were recently updated by Sonarr. So if an update happens, how can SZ update the sub as well?

Yes SZ will upgrade the subtitles accordingly and remove the previous one

@kevindd992002
Copy link
Author

kevindd992002 commented Jan 18, 2020

I get this for S08E01: -3_SDH
And this for S08E03: -4_SDH
Oops, you can disregard S08E01 if it doesn't have a PGS sub. Sorry about that. This is the one I'm sure about that has a PGS:
Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv
And yes, almost all of these were recently updated by Sonarr. So if an update happens, how can SZ update the sub as well?

Yes SZ will upgrade the subtitles accordingly and remove the previous one

That's what I thought. Also manually refreshing the metadata will always trigger the agents (including SZ), correct? I did refresh all metadata so it should've extracted the subs.

@pannal

Also, the reason why I brought up S08E01 as one episode that has an issue is because Plex detects the episode with an SRT sub (and plays it successfully with a sub) even though I don't exactly see an SRT in the same folder as the media file. I tried playing it just now and it did play with a sub and then after a few seconds, a sub was downloaded correctly. I'm assuming this is because of the "hybrid-plus" setting I have set. But still, why would Plex show that it has an SRT sub if it doesn't have the actual physical file? And would it really play with a sub right away even if it still in the process of downloading it when you play a file? I was assuming that it kinda take a minute to download and display the sub with the hybrid-plus setting.

Or are there embedded SRT subs?

Here's a concrete example of what I'm saying:

image

image

Look how S08E06 doesn't have an SRT file in that same folder but Plex recognizes that it does. And when I play this file, it "right away" displays the subs. And then after a few seconds the subs for S08E06 and S08E07 get downloaded.

This behavior DOES NOT happen for Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv (the one with the PGS subs). Here's the mediainfo for that episode:

root@ebc2133f420e:/data/tvshows/Arrow/Season 06# mediainfo "Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv"
General
Unique ID                                : 193206045323935109292794337985694682287 (0x915A217CEF40AC0634BDB464341E4CAF)
Complete name                            : Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv
Format                                   : Matroska
Format version                           : Version 4 / Version 2
File size                                : 4.31 GiB
Duration                                 : 42 min 29 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 14.5 Mb/s
Encoded date                             : UTC 2018-08-16 11:30:39
Writing application                      : mkvmerge v25.0.0 ('Prog Noir') 64-bit
Writing library                          : libebml v1.3.6 + libmatroska v1.4.9

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 4 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 42 min 29 s
Bit rate mode                            : Variable
Bit rate                                 : 11.8 Mb/s
Maximum bit rate                         : 30.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.238
Stream size                              : 3.51 GiB (81%)
Title                                    : MPEG-4 AVC Video / 11999 kbps / 1080p / 23.976 fps / 16:9 / High Profile 4.1
Language                                 : English
Default                                  : Yes
Forced                                   : No

Audio
ID                                       : 2
Format                                   : DTS
Format/Info                              : Digital Theater Systems
Format profile                           : MA / Core
Codec ID                                 : A_DTS
Duration                                 : 42 min 29 s
Bit rate mode                            : Variable / Constant
Bit rate                                 : 2 161 kb/s / 1 509 kb/s
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 kHz
Frame rate                               : 93.750 FPS (512 SPF)
Bit depth                                : 16 bits
Compression mode                         : Lossless / Lossy
Stream size                              : 657 MiB (15%)
Title                                    : DTS-HD Master Audio / 5.1 / 48 kHz / 2161 kbps / 16-bit
Language                                 : English
Default                                  : Yes
Forced                                   : No

Text #1
ID                                       : 3
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 25 s
Bit rate                                 : 56.1 kb/s
Count of elements                        : 1444
Stream size                              : 17.0 MiB (0%)
Title                                    : SDH
Language                                 : English
Default                                  : No
Forced                                   : No

Text #2
ID                                       : 4
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 20 s
Bit rate                                 : 52.7 kb/s
Count of elements                        : 1442
Stream size                              : 16.0 MiB (0%)
Language                                 : Danish
Default                                  : No
Forced                                   : No

Text #3
ID                                       : 5
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 22 s
Bit rate                                 : 50.6 kb/s
Count of elements                        : 1420
Stream size                              : 15.3 MiB (0%)
Language                                 : Dutch
Default                                  : No
Forced                                   : No

Text #4
ID                                       : 6
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 20 s
Bit rate                                 : 53.6 kb/s
Count of elements                        : 1440
Stream size                              : 16.2 MiB (0%)
Language                                 : Finnish
Default                                  : No
Forced                                   : No

Text #5
ID                                       : 7
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 22 s
Bit rate                                 : 54.8 kb/s
Count of elements                        : 1440
Stream size                              : 16.6 MiB (0%)
Title                                    : Parisian
Language                                 : French
Default                                  : No
Forced                                   : No

Text #6
ID                                       : 8
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 20 s
Bit rate                                 : 50.8 kb/s
Count of elements                        : 1442
Stream size                              : 15.4 MiB (0%)
Language                                 : Norwegian
Default                                  : No
Forced                                   : No

Text #7
ID                                       : 9
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 25 s
Bit rate                                 : 55.2 kb/s
Count of elements                        : 1632
Stream size                              : 16.7 MiB (0%)
Title                                    : Brazilian
Language                                 : Portuguese
Default                                  : No
Forced                                   : No

Text #8
ID                                       : 10
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 20 s
Bit rate                                 : 50.4 kb/s
Count of elements                        : 1442
Stream size                              : 15.3 MiB (0%)
Title                                    : Castilian
Language                                 : Spanish
Default                                  : No
Forced                                   : No

Text #9
ID                                       : 11
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 22 s
Bit rate                                 : 54.2 kb/s
Count of elements                        : 1636
Stream size                              : 16.4 MiB (0%)
Title                                    : Latin American
Language                                 : Spanish
Default                                  : No
Forced                                   : No

Text #10
ID                                       : 12
Format                                   : PGS
Codec ID                                 : S_HDMV/PGS
Codec ID/Info                            : Picture based subtitle format used on BDs/HD-DVDs
Duration                                 : 42 min 20 s
Bit rate                                 : 52.9 kb/s
Count of elements                        : 1442
Stream size                              : 16.0 MiB (0%)
Language                                 : Swedish
Default                                  : No
Forced                                   : No

Menu
00:00:00.000                             : en:Chapter 1
00:00:35.118                             : en:Chapter 2
00:03:32.295                             : en:Chapter 3
00:10:02.977                             : en:Chapter 4
00:17:07.485                             : en:Chapter 5
00:25:22.813                             : en:Chapter 6
00:32:41.376                             : en:Chapter 7
00:35:32.589                             : en:Chapter 8
00:42:00.893                             : en:Chapter 9

root@ebc2133f420e:/data/tvshows/Arrow/Season 06# mediainfo --Inform="Text;-%ID%_%Title%" "Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv"
-3_SDH-4_-5_-6_-7_Parisian-8_-9_Brazilian-10_Castilian-11_Latin American-12_

Also, the upgrades that happened for these Arrow episodes did change the filenames, that's for sure. And like I said above, regardless if the filenames were changed or not, I was assuming that the "refresh all metadata" option in Plex, will retrigger SZ to do its thing. Or am I assuming incorrectly?

@pannal
Copy link
Owner

pannal commented Jan 18, 2020

You're mistaking what Plex UI tells you.

This is the difference between Language (SRT) and Language (SRT External)
image

Non-External are embedded.

pannal added a commit that referenced this issue Jan 18, 2020
@pannal
Copy link
Owner

pannal commented Jan 18, 2020

Please try the latest develop-2.6 branch; I've tried fixing your mp4 stream detection issue. Automatic extraction for S08E03 might now work.

pannal added a commit that referenced this issue Jan 18, 2020
pannal added a commit that referenced this issue Jan 19, 2020
@kevindd992002
Copy link
Author

kevindd992002 commented Jan 19, 2020

You're mistaking what Plex UI tells you.

This is the difference between Language (SRT) and Language (SRT External)
image

Non-External are embedded.

Aha! I knew there was something off with how I understood English (SRT). Thanks for the clarification!

Please try the latest develop-2.6 branch; I've tried fixing your mp4 stream detection issue. Automatic extraction for S08E03 might now work.

Ok, great. I'll try this out. However, some questions:

  1. Will SZ consider a "refresh all metadata" event like a "new addition"? I'm asking because the automatic extraction setting specifically says "upon media addition".

  2. Will your fix also apply the PGS example in S06E16?

  3. If the After automatic extraction of embedded subtitles, also immediately search for available subtitles? setting is unchecked (and the automatic extraction of embedded subs is checked), will SZ still update the an extracted sub when the scheduled updater triggers? I guess my question is why would you want to enable this specific setting if you know that any extracted sub will have a 0 score anyway? Is it because if there's no available sub in the Internet, at least you have an extracted sub for the media file?

  4. Is a downloaded sub from the Internet almost always better than the default embedded subs you get with the file when you download it? Or is this a question that has really no answer?

  5. For the Don't search for subtitles of a language if they already exist on the filesystem (metadata/filesystem)? setting, when does SZ consider a sub file as pre-existing on the fileystem? Are automatically extracted embedded subs considered as such?

@kevindd992002
Copy link
Author

kevindd992002 commented Jan 19, 2020

Btw, are these the correct instructions for updating to the develop branch? Is there a way to have it configured so that SZ is automatically updated when a new version of the develop branch comes out? If you remember, I'm using LSIO's plex docker container.

Would a startup script for the container similar to this work?

apt update
apt install git -y
cd /config/Library/Application\ Support/Plex\ Media\ Server/Plug-ins
rm -r Sub-Zero.bundle
git clone https://github.com/pannal/Sub-Zero.bundle.git
cd Sub-Zero.bundle
git checkout develop-2.6

?

@pannal
Copy link
Owner

pannal commented Jan 19, 2020

Will SZ consider a "refresh all metadata" event like a "new addition"? I'm asking because the automatic extraction setting specifically says "upon media addition".

Yes.

Will your fix also apply the PGS example in S06E16?

It shouldn't, SZ can't do anything with PGS, as it's image based.

If the After automatic extraction of embedded subtitles, also immediately search for available subtitles? setting is unchecked (and the automatic extraction of embedded subs is checked), will SZ still update the an extracted sub when the scheduled updater triggers? I guess my question is why would you want to enable this specific setting if you know that any extracted sub will have a 0 score anyway? Is it because if there's no available sub in the Internet, at least you have an extracted sub for the media file?

Exactly. I want to have a subtitle right away. Also this way it doesn't need to trigger the providers on every new addition.

Is a downloaded sub from the Internet almost always better than the default embedded subs you get with the file when you download it? Or is this a question that has really no answer?

That depends. For WEB-DLs it doesn't really matter as most of the ones available are at least derived from if not complete copies of the embedded ones.

It's more interesting for BluRay and DVD, as the subtitles shipped might not have been correctly OCR'd and often contain errors - SZ will fix most of them, but community-created subtitles are almost always better in this case.

For the Don't search for subtitles of a language if they already exist on the filesystem (metadata/filesystem)? setting, when does SZ consider a sub file as pre-existing on the fileystem? Are automatically extracted embedded subs considered as such?

Yes.

Btw, are these the correct instructions for updating to the develop branch?

Yep.

Would a startup script for the container similar to this work?

Yes, but it might be easier/faster to always just fetch the latest zip of that branch: https://github.com/pannal/Sub-Zero.bundle/archive/develop-2.6.zip

@kevindd992002
Copy link
Author

kevindd992002 commented Jan 19, 2020

Will your fix also apply the PGS example in S06E16?

It shouldn't, SZ can't do anything with PGS, as it's image based.

But should it still download a new subtitle from the providers for the episode that has only PGS subs in it?

If the After automatic extraction of embedded subtitles, also immediately search for available subtitles? setting is unchecked (and the automatic extraction of embedded subs is checked), will SZ still update the an extracted sub when the scheduled updater triggers? I guess my question is why would you want to enable this specific setting if you know that any extracted sub will have a 0 score anyway? Is it because if there's no available sub in the Internet, at least you have an extracted sub for the media file?

Exactly. I want to have a subtitle right away. Also this way it doesn't need to trigger the providers on every new addition.

What do you mean this way it doesn't need to trigger the providers on every new addition? Are you pertaining to when the "After...." setting is checked or unchecked?

For the Don't search for subtitles of a language if they already exist on the filesystem (metadata/filesystem)? setting, when does SZ consider a sub file as pre-existing on the fileystem? Are automatically extracted embedded subs considered as such?

So if this setting is checked and the "After..." setting is checked, which has a higher priority when the media file has an embedded sub? Because what will happen is that SZ extracts the embedded subs which, according to you, will be considered as a "pre-existing" sub file. After extraction, will SZ still download a sub for that media file?

Yes.

Btw, are these the correct instructions for updating to the develop branch?

Yep.

Would a startup script for the container similar to this work?

Yes, but it might be easier/faster to always just fetch the latest zip of that branch: https://github.com/pannal/Sub-Zero.bundle/archive/develop-2.6.zip

Ok, got it. Is there a way to have the develop branch always automatically be updated just like the stable branch? Also, if I remove the SZ folder under Plug-ins, will my settings be retained after git cloning the dev repo?

@kevindd992002
Copy link
Author

kevindd992002 commented Jan 19, 2020

One more example for media files with embedded PGS subs. If I have the Don't search for subtitles of a language if there are embedded subtitles inside the media file (MKV/MP4)? setting checked and the subs are PGS, then I would end up with no initial subtitle, correct? Because it cannot extract the PGS sub and SZ detects the file has an embedded sub so it does not even attempt to download it.

For the subtitle update schedule, will SZ treat all media as fair game? My head is exploding as to which setting takes precedence over which setting :( A flowchart would be very very nice to explain all these, just a thought.

@kevindd992002
Copy link
Author

Any updates @pannal ?

@pannal
Copy link
Owner

pannal commented Jan 25, 2020

But should it still download a new subtitle from the providers for the episode that has only PGS subs in it?

Yes, as long as you don't have Include non-text subtitle formats (anything else than .srt/.ssa/.ass/.vtt; embedded or external) in the above? enabled.

What do you mean this way it doesn't need to trigger the providers on every new addition? Are you pertaining to when the "After...." setting is checked or unchecked?

When that option is checked, an additional search is explicitly done upon addition of a new media file, when an embedded subtitle has been extracted. Leaving this disabled lessens the impact on the providers obviously, as the next check would be a scheduled one.

So if this setting is checked and the "After..." setting is checked, which has a higher priority when the media file has an embedded sub? Because what will happen is that SZ extracts the embedded subs which, according to you, will be considered as a "pre-existing" sub file. After extraction, will SZ still download a sub for that media file?

The pre-existing check is done before the embedded detection/extraction is done. If you have the "After..." checked, it will still search the providers.

The order is:

  • check the current state of affairs
  • extract embedded if wanted
  • search the providers if no embedded was extracted or searching after extracted embedded was explicitly enabled

Ok, got it. Is there a way to have the develop branch always automatically be updated just like the stable branch?

No, that's impossible due to the way the Plex plugin updater handles things.

Also, if I remove the SZ folder under Plug-ins, will my settings be retained after git cloning the dev repo?

Pretty much always. The only way to effectively kill your settings is to actually delete the Plugin XML or move so weirdly between versions that some migration process breaks things. Never happened before.

One more example for media files with embedded PGS subs. If I have the Don't search for subtitles of a language if there are embedded subtitles inside the media file (MKV/MP4)? setting checked and the subs are PGS, then I would end up with no initial subtitle, correct? Because it cannot extract the PGS sub and SZ detects the file has an embedded sub so it does not even attempt to download it.

Only if you have Include non-text subtitle formats (anything else than .srt/.ssa/.ass/.vtt; embedded or external) in the above? enabled.

@kevindd992002
Copy link
Author

kevindd992002 commented Jan 26, 2020

But should it still download a new subtitle from the providers for the episode that has only PGS subs in it?

Yes, as long as you don't have Include non-text subtitle formats (anything else than .srt/.ssa/.ass/.vtt; embedded or external) in the above? enabled.

I don't have that setting enabled but when I search subs for that episode, I still don't get any. That specific episode (the one with PGS) was recently upgraded by Sonarr so SZ should know the original filename by querying Sonarr.

One concrete example is: Arrow (2012) - S06E16 - The Thanatos Guild - [Bluray-1080p Remux] [DTS-HD MA 5.1ch] [AVC 8bit] - EPSiLON.mkv

I've refreshed the metadata for the Arrow series several times now but I still don't get any subs for that episode which has embedded PGS subs. Here are the latest set of logs with dev 2.6: https://www.dropbox.com/s/rd8cela7dii9160/PMS%20Plugin%20Logs.zip?dl=0

What do you mean this way it doesn't need to trigger the providers on every new addition? Are you pertaining to when the "After...." setting is checked or unchecked?

When that option is checked, an additional search is explicitly done upon addition of a new media file, when an embedded subtitle has been extracted. Leaving this disabled lessens the impact on the providers obviously, as the next check would be a scheduled one.

So if this setting is checked and the "After..." setting is checked, which has a higher priority when the media file has an embedded sub? Because what will happen is that SZ extracts the embedded subs which, according to you, will be considered as a "pre-existing" sub file. After extraction, will SZ still download a sub for that media file?

The pre-existing check is done before the embedded detection/extraction is done. If you have the "After..." checked, it will still search the providers.

The order is:

* check the current state of affairs

* extract embedded if wanted

* search the providers if no embedded was extracted or searching after extracted embedded was explicitly enabled

So then the extracted sub will not really be considered "pre-existing" during the first ever processing for a specific episode, correct? Because SZ checks the current state of affairs before extracting the embedded subs, it sees no pre-existing file, does it?

And if there is already a pre-existing SRT for a media file, will SZ still try to extract the embedded sub? I'm assuming not anymore.

Also, what is your take on my reply here?

@pannal
Copy link
Owner

pannal commented Jan 27, 2020

I've refreshed the metadata for the Arrow series several times now but I still don't get any subs for that episode which has embedded PGS subs. Here are the latest set of logs with dev 2.6: https://www.dropbox.com/s/rd8cela7dii9160/PMS%20Plugin%20Logs.zip?dl=0

The log is incomplete, it cuts off at the end where it gets interesting regarding S06E16. Can you reupload?

So then the extracted sub will not really be considered "pre-existing" during the first ever processing for a specific episode, correct? Because SZ checks the current state of affairs before extracting the embedded subs, it sees no pre-existing file, does it?

Yes.

Also, what is your take on my reply here?

As long as you don't have Include non-text subtitle formats (anything else than .srt/.ssa/.ass/.vtt; embedded or external) in the above? enabled, it won't treat an embedded PGS subtitle as existing.

@kevindd992002
Copy link
Author

Ok, that's weird. I checked again and now all my episodes in Arrow have subtitles already! So I guess the refresh all metadata didn't get everything but the scheduled task did? What could be the cause of that?

Here's a re-upload of the logs: https://www.dropbox.com/s/rd8cela7dii9160/PMS%20Plugin%20Logs.zip?dl=0

@kevindd992002
Copy link
Author

Also, what caused the stable release not successfully get most subs? Is it because I'm using mediainfo and your code for the stable released lacked support for that or something?

@pannal
Copy link
Owner

pannal commented Jan 27, 2020

Here's a re-upload of the logs: https://www.dropbox.com/s/rd8cela7dii9160/PMS%20Plugin%20Logs.zip?dl=0

The Arrow run seems to have taken over most of the logs, so I can't really see what was going on with S06E16.

Also, what caused the stable release not successfully get most subs? Is it because I'm using mediainfo and your code for the stable released lacked support for that or something?

The develop-2.6 branch has the fixes for the issues I've been able to find in your original logs.

@kevindd992002
Copy link
Author

If I delete the subs for S06E16 and retrigger a refresh metadaya for arrow again, would that accurately recreate the same scenario?
Any remote ideas as to why SZ would only get the sub during a scheduled scan and not during a refresh metadata?

@pannal
Copy link
Owner

pannal commented Jan 29, 2020

There are a couple of instances where this could happen. For example if one of the providers isn't reachable or throttled at that specific moment.

You should be able to retrigger using the method you described, yes.

@kevindd992002
Copy link
Author

Is it generally safe to stay with the develop branch? If I understand correctly, the dev branch will always be ahead of the stable branch, correct?

@pannal
Copy link
Owner

pannal commented Feb 9, 2020

Yeah, it's usually safe. You're missing out on automatic updates, that you only get with stable releases, though.

@kevindd992002
Copy link
Author

Oh ok, I forgot about that. For this specific issue though, will the fix be in the stable release anytime soon?

@pannal
Copy link
Owner

pannal commented Feb 10, 2020

I'm preparing for a new stable release next weekend.

@pannal pannal closed this as completed Feb 15, 2020
@kevindd992002
Copy link
Author

Now how do I revert back to the stable release without destroying anything?

@kevindd992002
Copy link
Author

So then the extracted sub will not really be considered "pre-existing" during the first ever processing for a specific episode, correct? Because SZ checks the current state of affairs before extracting the embedded subs, it sees no pre-existing file, does it?

Yes.

@pannal I'm still confused with this statement. Let' say a new file has an embedded subtitle and I have these settings:

image

So when the media gets added, SZ will:

  1. Check current state of affairs -> no pre-existing file in the filesystem.
  2. Extract embedded sub to srt -> yes
  3. Search for sub because there is no pre-existing file in the filesystem -> yes

So with these events, the After... setting doesn't really have any effect whether it is disabled or enabled because SZ will still search for a sub even if the embedded sub is extracted "during the first run". So can you give a use case where that After... setting is useful?

Also, are all the settings on those pic only applicable for new additions? For example, since during the next scheduled run SZ already sees a "pre-existing" srt file for that sample above, will it still update the sub if a better one is found?

@kevindd992002
Copy link
Author

Also, I'm looking at my subzero plugin logs and all I can see are queries to subscene. I have addic7ed and anti-captcha enabled but I don't see any attempts to query addic7ed. I'm using the latest stable version (switch back from the develop-2.6 branch).

@kevindd992002
Copy link
Author

kevindd992002 commented Apr 2, 2020

@pannal Any updates on this? Do you need me to create a new issue?

@gitthangbaby
Copy link

gitthangbaby commented May 10, 2020

nice head exploding topic:)
if i uncheck
"Don't search for subtitles of a language if they already exist on the filesystem (metadata/filesystem)?"
log reports ".... has missing languages:" all languages for all items. i can refresh three times and three times all languages will be reported as missing.

but it won't download another copy to internal storage., luckily. but i'm worried about endless processing of same media.

i was hoping to have SZ download subtitles in its own storage no matter if external subtitles exist. and by external i mean files. (i've checked XML generates names for each subtitle, but it's not present in Plex files..so it must be generated realtime.. tried to trick SZ to use distinguishable rare VTT format to save but it doesn't work on internally saved files)

-> if i check this box, SZ won't download anything if files exist
-> if i uncheck it, SZ will forever report missing languages, possibly keep running nonstop.

i think i'd need metadata only checkbox:
"Don't search for subtitles of a language if they already exist on the filesystem (metadata)?"
as i want to have 1 copy of SZ subtitle per languages stored internally and 1 copy per language of external file stored on filesystem:

English (SRT) <- embedded
English (SRT External) <- from existing file
English (SRT External) <- from metadata

and ideally called "SRT Internal" but i guess Plex won't allow it. so have to stick to this confusing duplicit naming.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants