Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: OPML Import Failed #3118

Closed
hardwareadictos opened this issue Jul 5, 2024 · 13 comments
Closed

[Bug]: OPML Import Failed #3118

hardwareadictos opened this issue Jul 5, 2024 · 13 comments
Labels
bug Something isn't working

Comments

@hardwareadictos
Copy link

What happened?

WWhen i import an opml file from AntennaPod/Pocketcast podcastUtils crashes and doesn't import any subscription

What did you expect to happen?

Podcasts being imported from OPML file, even if any suscription doesnt exists anymore or returns a 404 error

Steps to reproduce the issue

Just importing an opml file on the web UI

Audiobookshelf version

v2.10.1

How are you running audiobookshelf?

Docker

What OS is your Audiobookshelf server hosted from?

Windows

If the issue is being seen in the UI, what browsers are you seeing the problem on?

None

Logs

2024-07-05 10:48:32.005

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:48:32.605

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:49:06.776

ERROR

[podcastUtils] getPodcastFeed Error Error: getaddrinfo ENOTFOUND compilando.audio

2024-07-05 10:49:18.809

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:49:32.252

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "http:" not supported. Expected "https:"

2024-07-05 10:49:33.842

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:49:41.946

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:49:50.536

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:50:02.407

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:50:41.092

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:50:51.982

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:51:00.969

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:51:07.391

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:51:08.585

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:52:04.777

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:52:09.011

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:52:14.265

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 403

2024-07-05 10:52:16.661

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:52:24.899

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:52:25.881

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:52:30.336

ERROR

[podcastUtils] Invalid XML or RSS feed

2024-07-05 10:52:41.841

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:52:42.391

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:52:42.705

ERROR

[podcastUtils] getPodcastFeed Error Error: getaddrinfo ENOTFOUND podcastelardelgeek.com

2024-07-05 10:52:46.365

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:52:46.741

ERROR

[podcastUtils] getPodcastFeed Error Error: getaddrinfo ENOTFOUND compilando.audio

2024-07-05 10:52:51.651

ERROR

[podcastUtils] getPodcastFeed Error Error: getaddrinfo ENOTFOUND podcasteltelardelgeek.com

2024-07-05 10:52:51.744

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:52:54.408

ERROR

[podcastUtils] getPodcastFeed Error Error: getaddrinfo ENOTFOUND www.geekjoze.es

2024-07-05 10:52:58.705

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:52:59.432

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:53:03.108

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:53:06.196

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:53:09.735

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:53:10.552

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:53:11.958

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:53:12.026

ERROR

[podcastUtils] getPodcastFeed Error Error: getaddrinfo ENOTFOUND www.hablandodehardware.com

2024-07-05 10:53:18.209

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:53:40.055

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: timeout of 12000ms exceeded

2024-07-05 10:53:40.174

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:53:40.272

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:53:48.904

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "http:" not supported. Expected "https:"

2024-07-05 10:53:53.868

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:53:56.760

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:53:56.952

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:53:57.281

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:54:00.669

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:54:05.690

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:54:11.847

ERROR

[xmlToJSON] Error Error: Unexpected close tag Line: 274 Column: 7 Char: >

2024-07-05 10:54:11.848

ERROR

[podcastUtils] Invalid XML or RSS feed

2024-07-05 10:54:27.815

ERROR

[podcastUtils] getPodcastFeed Error Error: getaddrinfo ENOTFOUND compilando.audio

2024-07-05 10:54:27.821

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:54:29.257

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:54:29.324

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:54:30.531

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:54:34.929

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:54:35.039

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:54:36.499

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 500

2024-07-05 10:54:36.896

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:54:38.472

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:55:09.162

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "http:" not supported. Expected "https:"

2024-07-05 10:55:10.476

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:55:18.428

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:55:26.018

ERROR

[podcastUtils] getPodcastFeed Error Error [ERR_FR_REDIRECTION_FAILURE]: Redirected request failed: Protocol "https:" not supported. Expected "http:"

2024-07-05 10:55:33.485

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:56:10.887

ERROR

[podcastUtil] Invalid podcast no episodes

2024-07-05 10:56:18.809

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:56:25.769

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:56:32.079

ERROR

[podcastUtils] getPodcastFeed Error AxiosError: Request failed with status code 404

2024-07-05 10:56:32.865

ERROR

[podcastUtil] Invalid podcast no episodes

Additional Notes

OPML file: https://pastecode.io/s/j10cuenc
@hardwareadictos hardwareadictos added the bug Something isn't working label Jul 5, 2024
@mikiher
Copy link
Contributor

mikiher commented Jul 10, 2024 •
edited
Loading

I tried importing your OPML, and it successfully imported all the podcasts that it did find.

There were many podcasts in the list for which the podcast feed link is broken or empty, and therefore you get errors for those - that is normal. You said: "podcastUtils crashes and doesn't import any subscription". Is the server actually crashing? when I tried it, I saw no crush, but the import took quite a bit of time (maybe 5-10 minutes), due to the large number of feeds.

What would you want Audiobookshelf to do with a podcast that does not have a working feed? If it doesn't have a feed it's not actionable (you cannot search and download episodes, you cannot display its cover image - you can't do anything with it).

@hardwareadictos
Copy link
Author

I tried importing your OPML, and it successfully imported all the podcasts that it did find.

There were many podcasts in the list for which the podcast feed link is broken or empty, and therefore you get errors for those - that is normal. You said: "podcastUtils crashes and doesn't import any subscription". Is the server actually crashing? when I tried it, I saw no crush, but the import took quite a bit of time (maybe 5-10 minutes), due to the large number of feeds.

What would you want Audiobookshelf to do with a podcast that does not have a working feed? If it doesn't have a feed it's not actionable (you cannot search and download episodes, you cannot display its cover image - you can't do anything with it).

Nice! I didn't wait those five minutes as i saw lots of 404 errors on the logs, glad it worked for you.

I'm going to reproduce it and see if it works.

@hardwareadictos
Copy link
Author

Well, still the same. I just load the OPML file and it crashes 2/3 minutes later:

image

image

image

Which browser are you using? I tried Edge on Windows Server 2019, and Edge/Firefox on OpenSUSE.

@mikiher
Copy link
Contributor

mikiher commented Jul 10, 2024

I doubt the browser is the issue here (this is all happening on the server).

So, just to be clear, what I see in your screenshots is a message saying "Failed to parse OPML file". I don't see a crash - so the server is still running, but the OPML import fails, correct?

@hardwareadictos
Copy link
Author

I doubt the browser is the issue here (this is all happening on the server).

So, just to be clear, what I see in your screenshots is a message saying "Failed to parse OPML file". I don't see a crash - so the server is still running, but the OPML import fails, correct?

yes. that's it!

@mikiher
Copy link
Contributor

mikiher commented Jul 10, 2024 •
edited
Loading

Can I ask you to please set Audiobookshelf Server Log Level (in Settings -> Logs) to Debug, and try to import the OPML again?
After that can you please attach the latest actual log file from the server? It can be found at /metadata/logs/daily
Also if there's any crash log at /metadata/logs/crash_logs.txt, please attach that as well.

@hardwareadictos
Copy link
Author

hardwareadictos commented Jul 10, 2024 •
edited
Loading

Can I ask you to please set Audiobookshelf Server Log Level (in Settings -> Logs) to Debug, and try to import the OPML again? After that can you please attach the latest actual log file from the server? It can be found at /metadata/logs/daily Also if there's any crash log at /metadata/logs/crash_logs.txt, please attach that as well.

Done! Uploading the logs, just search on todays log from 12:00 and beyond, on crash logs you will see something about SQLITE DB crashing, i think (but im not sure) that's realted to another issue i opened: #3123

crash_logs.txt
2024-07-10.txt

@mikiher
Copy link
Contributor

mikiher commented Jul 10, 2024

OK, thanks for sending the logs. The logs seem to indicate that the OPML upload operation on the server finished successfuly, so I'm suspecting maybe some timeout or another connection-related error between the browser and the server.

Can I ask you to open the developer tools on the browser (f12 on Chrome and Edge), and go to the Console tab. Then try to import the opml file again, and wait for the "Failed to parse OPML file" message. Please check if some error message is printed to the console at same time the "Failed to parse OPML file" message appears, and paste it here.

@hardwareadictos
Copy link
Author

hardwareadictos commented Jul 10, 2024 •
edited
Loading

And you were right:

image

Error 504. I'm runing under an nginx proxy manager with websockets enabled.

BUT: If i attack de service via the internal ip:port i am able to get all the podcasts correctly and i can add them to my library!!:

image

So something in the proxy is giving those timeouts. Let me check the proxy and the documentation.

EDIT: as of the documentation, only websockets needs to be enabled so let me investigate how to make those timeouts bigger to avoid the 502 error on Nginx Proxy Manager.

@hardwareadictos
Copy link
Author

Okay seems that adding proxy_read_timeout 600s (10 minutes in my case because the opml was taking 5 mins to import through the proxy); to Advanced tab on the proxy host helps:

image

Let me test this fore some days and i will report back :)

@mikiher
Copy link
Contributor

mikiher commented Jul 10, 2024

@advplyr that's an interesting type of failure. We should probably make the OPML import a non-blocking task like metadata embedding. What do you think?

@advplyr
Copy link
Owner

advplyr commented Jul 11, 2024

Yeah we definitely should. We may have to skip the confirmation step then and just add the podcasts. Showing a task in the appbar.

@mikiher mikiher mentioned this issue Jul 14, 2024
Closed
advplyr added a commit that referenced this issue Jul 16, 2024
@advplyr
Fix:Large OPML import timeouts #3118
37ad1cc 
- Added OPML Api endpoints for /parse and /create, removed old
- Show task for OPML import and create failed tasks for failed feeds
@advplyr advplyr added the awaiting release Issue is resolved and will be in the next release label Jul 16, 2024
glorenzen pushed a commit to glorenzen/audiobookshelf that referenced this issue Jul 24, 2024
@advplyr @glorenzen
Fix:Large OPML import timeouts advplyr#3118
01678cf 
- Added OPML Api endpoints for /parse and /create, removed old
- Show task for OPML import and create failed tasks for failed feeds
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 4, 2024

Fixed in v2.12.0.

@github-actions github-actions bot closed this as completed Aug 4, 2024
@github-actions github-actions bot removed the awaiting release Issue is resolved and will be in the next release label Aug 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make OPML import non-blocking mikiher/audiobookshelf
3 participants
@hardwareadictos @mikiher @advplyr