[tiktok:user+TTUser] Unable to extract sigi state #5

jacktou · 2023-11-06T09:54:20Z

WARNING: The program functionality for this site has been marked as broken, and will probably not work.
ERROR: [tiktok:user+TTUser] hugostunes: Unable to extract sigi state; please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U

bashonly · 2023-11-07T20:50:12Z

~~Seems like it was a temporary issue, I can no longer reproduce the error~~

EDIT:

If you are experiencing this issue, try installing/reinstalling the newest version of the plugin

S-Kantor · 2023-11-09T03:38:43Z

It's happening on every profile I'm using it on @bashonly. (even @therock)

bashonly · 2023-11-09T04:20:19Z

[tiktok:user+TTUser] Extracting URL: https://www.tiktok.com/@therock
[tiktok:user+TTUser] therock: Downloading user webpage
[download] Downloading playlist: therock
[tiktok:user+TTUser] therock: Downloading page 1
[tiktok:user+TTUser] therock: Downloading page 2
[tiktok:user+TTUser] therock: Downloading page 3
[tiktok:user+TTUser] therock: Downloading page 4
[tiktok:user+TTUser] therock: Downloading page 5

sorry, if I can't reproduce the error, then I can't fix it either

provide more info, e.g. logs, --write-pages dumps, etc

bashonly · 2023-11-09T04:21:21Z

what happens when you open the user's pages in a private browsing tab? is tiktok asking you to log in?

nijippoi · 2023-11-09T14:24:47Z

Same error here.
Opening TT user page on private browsing tab shows a captcha.
Going through proxies is a hit and 90% miss 😓

yt-dlp.exe -v --write-pages --extractor-args "tiktok:web_fallback" https://www.tiktok.com/@megadeth
[debug] Command-line config: ['-v', '--write-pages', '--extractor-args', 'tiktok:web_fallback', 'https://www.tiktok.com/@megadeth']
[debug] Portable config "C:\apps\utils\misc\yt-dlp.conf": ['--ffmpeg-location', 'C:\\apps\\media\\ffmpeg\\bin', '-o', '%(title).200s [%(id)s].%(ext)s', '--add-header', 'Accept-Language:ja-JP', '--embed-metadata', '--embed-chapters', '--sleep-interval', '3', '--retry-sleep', 'http:linear=3::2', '--retry-sleep', 'fragment:exp=3:30']
[debug] Encodings: locale cp932, fs utf-8, pref cp932, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.10.13 [b634ba742] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg 6.0-full_build-www.gyan.dev (setts), ffprobe 6.0-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, sqlite3-3.35.5, websockets-11.0.3
[debug] Proxy map: {}
[debug] Extractor Plugins: TTUser (TikTokUserIE)
[debug] Plugin directories: ['C:\\apps\\utils\\misc\\yt-dlp-plugins\\yt-dlp-TTUser\\yt_dlp_plugins']
[debug] Loaded 1890 extractors
WARNING: The program functionality for this site has been marked as broken, and will probably not work.
[tiktok:user+TTUser] Extracting URL: https://www.tiktok.com/@megadeth
[tiktok:user+TTUser] megadeth: Downloading user webpage
[tiktok:user+TTUser] Saving request to megadeth_https_-_www.tiktok.com_@megadeth.dump
ERROR: [tiktok:user+TTUser] megadeth: Unable to extract sigi state; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 715, in extract
  File "C:\apps\utils\misc\yt-dlp-plugins\yt-dlp-TTUser\yt_dlp_plugins\extractor\tt_user.py", line 115, in _real_extract
    sec_uid = self._get_sec_uid(user_url, user_name, msg)
  File "C:\apps\utils\misc\yt-dlp-plugins\yt-dlp-TTUser\yt_dlp_plugins\extractor\tt_user.py", line 101, in _get_sec_uid
    data = try_call(lambda: self._get_sigi_state(webpage, user_name))
  File "yt_dlp\utils\_utils.py", line 2604, in try_call
  File "C:\apps\utils\misc\yt-dlp-plugins\yt-dlp-TTUser\yt_dlp_plugins\extractor\tt_user.py", line 101, in <lambda>
    data = try_call(lambda: self._get_sigi_state(webpage, user_name))
  File "yt_dlp\extractor\tiktok.py", line 52, in _get_sigi_state
  File "yt_dlp\extractor\common.py", line 1277, in _search_json
  File "yt_dlp\extractor\common.py", line 1263, in _search_regex

megadeth_https_-www.tiktok.com@megadeth.dump.txt

bashonly · 2023-11-09T15:02:58Z

Opening TT user page on private browsing tab shows a captcha.

That would be the issue then. Something is causing you to get a captcha page instead of the user page (IP flagged? fingerprinting? who knows)

Possible workaround would be to solve the captcha in browser and then pass --cookies-from-brower

bashonly · 2023-11-09T15:09:51Z

or you could use the extractor arg if you know the secUid of the user

Pass --extractor-args "tiktok:sec_uid=ID" to specify a secondary user id with the value of ID

nijippoi · 2023-11-09T15:40:23Z

@bashonly Thanks for the tip!
Since the page returned with the captcha still has the actual page, I got the secUid by changing as follows:

    def _get_sec_uid(self, user_url, user_name, msg):
        webpage = self._download_webpage(
            user_url, user_name, fatal=False, headers={'User-Agent': 'Mozilla/5.0'},
            note=f'Downloading {msg} webpage', errnote=f'Unable to download {msg} webpage')
        udata_json = clean_html(get_element_by_id('__UNIVERSAL_DATA_FOR_REHYDRATION__', webpage))
        udata = json.loads(udata_json)
        u = udata['__DEFAULT_SCOPE__']['webapp.user-detail']['userInfo']['user']
        return u['secUid']

wireman21 · 2023-11-13T05:20:55Z

Hi sorry for asking but how does one get the "secUid"? I tried googling and it let me here but those links don't seem to work anymore. (link: carcabot/tiktok-signature#167 )

bashonly · 2023-11-13T13:11:45Z

Hi sorry for asking but how does one get the "secUid"?

If you are able to download a single video from the user with yt-dlp, you can get the sec_uid with this command:

yt-dlp --print channel_id "URL"

where URL is the link to the single video webpage

wireman21 · 2023-11-13T14:14:57Z

If you are able to download a single video from the user with yt-dlp, you can get the sec_uid with this command:
yt-dlp --print channel_id "URL"
where URL is the link to the single video webpage

Thank you, was getting the "Unable to extract sigi state" error before and now it works again after passing --extractor-args "tiktok:sec_uid=ID"

wireman21 · 2023-11-13T15:26:08Z

Just an update (not complaining and still better than nothing), I did a test run with an account with 1800++ videos and the issue still persist every now and then in the middle of downloading.

[download] Downloading item 1186 of 1874 [TikTok] Extracting URL: https://www.tiktok.com/@ [TikTok] : Downloading video feed WARNING: [TikTok] : Unable to find video in feed; please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U; trying with webpage

Vladimir-russian · 2023-11-14T00:54:21Z

I am having this same issue. This "yt-dlp --print channel_id "URL"" do solve the problem however it is time consuming if you got multiple urls to download

Any solution please ?

bashonly · 2023-11-14T07:37:20Z

@wireman21

the issue still persist every now and then in the middle of downloading
WARNING: [TikTok] : Unable to find video in feed

That's a different issue, and it's not an issue specific to this plugin. The same thing would happen if you tried to download that video individually with its video URL via yt-dlp.

And what's that this is for:

Pass --extractor-args "tiktok:web_fallback" to extract videos from web API when unavailable from mobile feed (user extraction will take longer)

To use both extractor-args simultaneously, you would pass them as:
--extractor-args "tiktok:web_fallback;sec_uid=SECUID"

bashonly · 2023-11-14T07:40:51Z

@Vladimir-russian

I am having this same issue. This "yt-dlp --print channel_id "URL"" do solve the problem however it is time consuming if you got multiple urls to download
Any solution please ?

You could try applying the patch posted above by nijippoi and seeing if that helps.

Otherwise, no, there is no solution for now. I'll need to investigate. Could maybe try making sigi state extraction non-fatal to see if the other two user pages are captcha-walled as well

bashonly · 2023-11-14T12:07:13Z

33da60a (thanks @nijippoi)

If you are experiencing this issue, try installing/reinstalling the newest version of the plugin

Vladimir-russian · 2023-11-15T00:00:50Z

@bashonly Thanks for the tip! Since the page returned with the captcha still has the actual page, I got the secUid by changing as follows:

    def _get_sec_uid(self, user_url, user_name, msg):
        webpage = self._download_webpage(
            user_url, user_name, fatal=False, headers={'User-Agent': 'Mozilla/5.0'},
            note=f'Downloading {msg} webpage', errnote=f'Unable to download {msg} webpage')
        udata_json = clean_html(get_element_by_id('__UNIVERSAL_DATA_FOR_REHYDRATION__', webpage))
        udata = json.loads(udata_json)
        u = udata['__DEFAULT_SCOPE__']['webapp.user-detail']['userInfo']['user']
        return u['secUid']

Hi, how do I make this as part of my config file ? (totally newbie to coding)

bashonly · 2023-11-15T00:02:58Z

@Vladimir-russian just install the newest version of the plugin. It incorporates that patch

wireman21 · 2023-11-15T04:18:13Z

@bashonly

To use both extractor-args simultaneously, you would pass them as: --extractor-args "tiktok:web_fallback;sec_uid=SECUID"

Thank you for taking the time to explain things like this. I actually use the command --extractor-args "tiktok:sec_uid=ID" --extractor-args "tiktok:web_fallback" together in the same line, figured it didn't work so didn't use it, but you explaining the correct syntax here I think helps a lot of us who are not as tech literate.

Also the plug-in now works again like normal after re-installation for me.

Thank you again.

Zachary247 · 2023-11-15T20:16:52Z

I noticed when testing mass a big channel If a video doesnt have a sound you will get
WARNING: Unable to find video in feed;
ERROR: Unable to extract sigi state;
I haven't had this happen before.
( I know this worked before (last month) becuase I remember it being really cool how the download would restore the no sound of the video you viewed on the app or browser if you were to use the url )
.\yt-dlp --extractor-args "tiktok:sec_uid=MS4wLjABAAAAZN-W1gC7Paw30Oc1eRlHctiPYBn-Z2OG8TtK7ZTaCOOwxa6qAu_rgMd9aG9FV2iV" -vU https://www.tiktok.com/@adriannaeves/video/6862481367289040133 [debug] Command-line config: ['--extractor-args', 'tiktok:sec_uid=MS4wLjABAAAAZN-W1gC7Paw30Oc1eRlHctiPYBn-Z2OG8TtK7ZTaCOOwxa6qAu_rgMd9aG9FV2iV', '-vU', 'https://www.tiktok.com/@adriannaeves/video/6862481367289040133'] [debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8 [debug] yt-dlp version stable@2023.11.14 from yt-dlp/yt-dlp [a9d3f4b20] (win_exe) [debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.22000-SP0 (OpenSSL 1.1.1k 25 Mar 2021) [debug] exe versions: ffmpeg 6.0-full_build-www.gyan.dev (setts), ffprobe 6.0-full_build-www.gyan.dev [debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, requests-2.31.0, sqlite3-3.35.5, urllib3-2.1.0, websockets-12.0 [debug] Proxy map: {} [debug] Request Handlers: urllib, requests [debug] Extractor Plugins: TTUser (TikTokUserIE) [debug] Plugin directories: ['C:\\Users\\zacha\\AppData\\Roaming\\yt-dlp\\plugins\\yt-dlp-TTUser\\yt_dlp_plugins'] [debug] Loaded 1899 extractors [debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest Latest version: stable@2023.11.14 from yt-dlp/yt-dlp yt-dlp is up to date (stable@2023.11.14 from yt-dlp/yt-dlp) [TikTok] Extracting URL: https://www.tiktok.com/@adriannaeves/video/6862481367289040133 [TikTok] 6862481367289040133: Downloading video feed WARNING: [TikTok] 6862481367289040133: Unable to find video in feed; please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U; trying with webpage [TikTok] 6862481367289040133: Downloading webpage ERROR: [TikTok] 6862481367289040133: Unable to extract sigi state; please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U File "yt_dlp\extractor\common.py", line 715, in extract File "yt_dlp\extractor\tiktok.py", line 700, in _real_extract File "yt_dlp\extractor\tiktok.py", line 52, in _get_sigi_state File "yt_dlp\extractor\common.py", line 1277, in _search_json File "yt_dlp\extractor\common.py", line 1263, in _search_regex
Let me know if you guys are getting the same.
Also Thank you bashonly sir for all the good work and comments! :)

bashonly · 2023-11-15T20:23:10Z

I know this worked before (last month)

If you're referring to yt-dlp's built-in tiktok user extractor, that's because it got the video info from a different API endpoint where those videos were still available from. We no longer can access that endpoint (which is why the plugin is necessary). Your only hope of downloading those videos now is by using the web_fallback extractor-arg; see the post above for how to properly pass both sec_uid and web_fallback extractor args

bashonly · 2023-11-16T10:37:04Z

@modelfe that is a video URL, not a user URL. The plugin is not being invoked in that log at all. That is a yt-dlp issue

bashonly · 2023-11-17T10:21:47Z

How did it fail to download with the plugin and web_fallback? That's the log I would be interested in, and something I may be able to fix in the plugin. Anything else is native yt-dlp and outside the scope of this issue tracker

bashonly · 2023-11-19T18:14:26Z

@modelfe Are you able to view those videos on the website (not the app) when not logged in?

wireman21 · 2023-11-19T21:48:43Z

You probably need to authenticate with "--cookies" or "--cookies-from-browser"

If the video is friends only, make sure you're following the profile.

bashonly · 2023-11-19T22:55:25Z

@modelfe @wireman21
Essentially this is a yt-dlp issue, not an issue with the plugin.

[TikTok] Unable to extract sigi state yt-dlp/yt-dlp#8620

As far as the plugin is concerned, all this really tells me is that web_fallback may be entirely useless and should be removed / not included in the final yt-dlp extractor

bashonly · 2023-11-19T23:50:23Z

@wireman21 OK but that's not related to this issue. You could open a new issue for that, with a full verbose log

bashonly mentioned this issue Nov 13, 2023

Error downloading channel #7

Closed

bashonly added the bug Something isn't working label Nov 13, 2023

bashonly mentioned this issue Nov 14, 2023

Config file and saving files by user/channel name yt-dlp/yt-dlp#8573

Closed

9 tasks

bashonly changed the title ~~[tiktok:user+TTUser] hugostunes: Unable to extract sigi state~~ [tiktok:user+TTUser] Unable to extract sigi state Nov 14, 2023

bashonly closed this as completed Nov 15, 2023

bashonly pinned this issue Nov 15, 2023

This comment was marked as off-topic.

Sign in to view

Repository owner locked as resolved and limited conversation to collaborators Nov 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tiktok:user+TTUser] Unable to extract sigi state #5

[tiktok:user+TTUser] Unable to extract sigi state #5

jacktou commented Nov 6, 2023

bashonly commented Nov 7, 2023 •

edited

Loading

S-Kantor commented Nov 9, 2023

bashonly commented Nov 9, 2023

bashonly commented Nov 9, 2023

nijippoi commented Nov 9, 2023 •

edited

Loading

bashonly commented Nov 9, 2023

bashonly commented Nov 9, 2023

nijippoi commented Nov 9, 2023

wireman21 commented Nov 13, 2023

bashonly commented Nov 13, 2023

wireman21 commented Nov 13, 2023

wireman21 commented Nov 13, 2023

Vladimir-russian commented Nov 14, 2023

bashonly commented Nov 14, 2023 •

edited

Loading

bashonly commented Nov 14, 2023

bashonly commented Nov 14, 2023 •

edited

Loading

Vladimir-russian commented Nov 15, 2023

bashonly commented Nov 15, 2023

wireman21 commented Nov 15, 2023 •

edited

Loading

Zachary247 commented Nov 15, 2023 •

edited

Loading

bashonly commented Nov 15, 2023 •

edited

Loading

This comment was marked as off-topic.

bashonly commented Nov 16, 2023

This comment was marked as off-topic.

bashonly commented Nov 17, 2023

This comment was marked as off-topic.

bashonly commented Nov 19, 2023

wireman21 commented Nov 19, 2023 •

edited

Loading

This comment was marked as off-topic.

bashonly commented Nov 19, 2023 •

edited

Loading

This comment was marked as off-topic.

bashonly commented Nov 19, 2023

[tiktok:user+TTUser] Unable to extract sigi state #5

[tiktok:user+TTUser] Unable to extract sigi state #5

Comments

jacktou commented Nov 6, 2023

bashonly commented Nov 7, 2023 • edited Loading

If you are experiencing this issue, try installing/reinstalling the newest version of the plugin

S-Kantor commented Nov 9, 2023

bashonly commented Nov 9, 2023

bashonly commented Nov 9, 2023

nijippoi commented Nov 9, 2023 • edited Loading

bashonly commented Nov 9, 2023

bashonly commented Nov 9, 2023

nijippoi commented Nov 9, 2023

wireman21 commented Nov 13, 2023

bashonly commented Nov 13, 2023

wireman21 commented Nov 13, 2023

wireman21 commented Nov 13, 2023

Vladimir-russian commented Nov 14, 2023

bashonly commented Nov 14, 2023 • edited Loading

bashonly commented Nov 14, 2023

bashonly commented Nov 14, 2023 • edited Loading

Vladimir-russian commented Nov 15, 2023

bashonly commented Nov 15, 2023

wireman21 commented Nov 15, 2023 • edited Loading

Zachary247 commented Nov 15, 2023 • edited Loading

bashonly commented Nov 15, 2023 • edited Loading

This comment was marked as off-topic.

bashonly commented Nov 16, 2023

This comment was marked as off-topic.

bashonly commented Nov 17, 2023

This comment was marked as off-topic.

bashonly commented Nov 19, 2023

wireman21 commented Nov 19, 2023 • edited Loading

This comment was marked as off-topic.

bashonly commented Nov 19, 2023 • edited Loading

This comment was marked as off-topic.

bashonly commented Nov 19, 2023

bashonly commented Nov 7, 2023 •

edited

Loading

nijippoi commented Nov 9, 2023 •

edited

Loading

bashonly commented Nov 14, 2023 •

edited

Loading

bashonly commented Nov 14, 2023 •

edited

Loading

wireman21 commented Nov 15, 2023 •

edited

Loading

Zachary247 commented Nov 15, 2023 •

edited

Loading

bashonly commented Nov 15, 2023 •

edited

Loading

wireman21 commented Nov 19, 2023 •

edited

Loading

bashonly commented Nov 19, 2023 •

edited

Loading