r/gonewildaudio sub support (pitch)

[PenitentMonke]

What do people think of adding support for r/gonewildaudio submissions?
This is somewhat embarrassing, but I noticed there's pornhub support so...

Hopefully I'm not posting this in the wrong place, apologies in advance if that's the case.

Put simply, they're selfposts containing one or more links to a specific website where audio files are hosted. I've got it near working here and was wondering whether I could/should open an issue and/or PR soon. I also want to get the "tags" in the posts and save them in a convenient manner, but I think that's out of bdfr's scope.

It's also possible to just use the normal selfpost handler and throw the output into another script that gets the audio(s) and tags, but bdfr is just so handy I decided to give this a shot.

Thoughts?

[Serene-Arc]

I'm hesitant to include this based on security, reliability, and economic concerns . If I'm understanding right, this is a text post with hyperlinks in the text that link out to a site with audio. To do this with the current setup of the BDFR, it would involve a module that searches the text for links and then perhaps matches them with another module if possible. So my concerns:

• Security: I don't think it's a good idea to just try and download any link that a user posts. Normally the BDFR gets around this by using the same method that Reddit uses: community policing of the links, subreddit rules, etc. We also use our own regex to try and keep it to a limited number of sites. Admittedly this isn't my biggest concern but it still is one. Automatically trying to scrape any website that someone links to in their text isn't a good idea, even for something simple like the BDFR.
• Reliability: Again, this is a smaller concern, but when we open up the BDFR to try and handle unknown websites in different ways, the chances of something going wrong shoot straight up. Maybe we could catch most of them but probably not all of them.
• Economics: This is the biggest one for me. If this scheme of downloading any link in a text post that matches a downloader module, that is potentially a lot of data that isn't prompted, filtered, or known in advance. Bear in mind that the BDFR does not do per-subreddit logic, so this would be applied to every post that the BDFR comes across.

@aliparlakci can weigh in if he wants but I'm extremely wary of implementing this. It would be fine if you implemented it for yourself of course, since you know what you're downloading and how much there's likely to be. The BDFR was made to be to easily extensible and modifiable but implementing this change in the wider repository might have some unintended consequences.

[PenitentMonke]

Thank you very much for your input, I think those are extremely valid points. To be honest, I've considered the idea some more since posting this and came to similar concerns regarding security and reliability. I'm trusting my regex to reliably find all resources in both the post and the website, taking advantage of the standard the sub's submissions must follow and the destination website always being the same, which is still quite a bit of trust. This code'll likely remain on my machine, but from the start I only wished to know if anyone else was interested in the idea. If there are concerns, it's probably best to err on the side of caution.

I'm a little curious about the per-subreddit logic part though. For Gwa, I just added another condition on the download factory/pull_lever() to detect when the sub's base url is matched (before SelfPost) and added a site downloader for it. Do you think that should be avoided for some reason? Like security/reliability or because it's a bit of a kludge, maybe. That was simplest way I found to not have that logic apply on every post.

[Serene-Arc]

You're more than welcome to host the code yourself, that's what GitHub is good for.

The main reason is that per-subreddit processing is completely different from how the BDFR processing works for everything else. It's based on the type of link present in the post, not so much the subreddit the post is in. Sure it's easy to do in code but that's not sustainable or feasible to do for every single subreddit and having two radically different approaches to processing submissions seems avoidable.

The archiver section of the BDFR downloads the text of the post, as well as the self-post module. It would be pretty easy to create another tool that reads those saved posts and extracts the links to download it. Considering the points above, I think a more specialised tool would be appropriate.