Limit in amount of whitelisted channels #19
Comments
|
Hi. I'm glad to hear you're using bbyen! I don't see anything in the logs. We can check if all whitelisted channels are recognized. Please try version 2.1.3 which has additional logging for this purpose. There should not be any limits. In the past, I ran into limits sending emails. For example, if I would subscribe to some new channels, it would try to send a bunch of emails to send out the backlog from those channels. But that was a limit with my email provide. After a few attempts, it would clear up. I don't see any logs saying that there are new videos, so it can't be that. |
|
I have 599 channels after filtering, so not so different from you. I've not noticed any problems, but maybe I'm less observant than you and I'm missing stuff 😬 |
|
Thanks for the quick response! I've created a fresh instance of latest bbyen 2.1.3 and copied my config, token and database over. First of all, I think there is an issue with installing the ts-node package as by installing it via the
When it comes to the logging from the latest version, the logging says that it only found 340 channels while I have 411 (Did some cleanup of duplicates, etc in the meantime so it is not the same number as from the config I shared above) I do have like 15 more examples if you want me to send it but I guess those 2 examples should be enough for now. Logs are hereubuntu@linux:/opt/bbyen-test/bbyen$ npm start
2024-11-10T00:05:34.842Z [verbose] [config]: Normalizing channel string 'UCcUNYFu8wNDncymesUOtPrg' |
|
Thanks for doing those tests. That narrows it down to a problem with the subscriptions. I will have a look. How did you install the latest version? That might have something to do with what you saw. Did you do |
|
You're right. ts-node was a devDependency when it should be a dependency |
I followed the instructions in README exactly but you've already found the root cause :) |
|
Since you are downloading through git, could you please try the branch issue_19? That will save me from making a lot of releases just to add some logging to troubleshoot your issue. |
|
Sure. By using that branch, to my eye there isn't anything special in the logs. What I noticed however when I run the branch first time is that the script removed channel |
According to this, everything is working perfectly. The YouTube API thinks you are subscribed to 984 channels and it gave data for 984 channels. |
|
That's very strange, I've just checked this in the Google Dashboard and I'm subbed to I also just reauthorized my token but the amount of subs in the logs is the same. |
|
Seems like it could be this bug: https://issuetracker.google.com/issues/35171641 My experience with Google APIs is that they are riddled with bugs that are never fixed. The above bug is from 2013, which definitely reinforces that. Thinking about it, if you use a whitelist, there is no need to query YouTube to get your subscriptions. It would mean a second code path that I have to maintain, but if Google can't fix their shit, then such a workaround would at least solve your problem. Unfortunately, this only works for people using the whitelist. |
Yeah, that sounds about right..
I was thinking about the same. As I filter my subscriptions to bbyen only from the |
|
I implemented this and pushed it to When I tried it with your whitelist, I got 6 channels that could not be found (possibly the channel was deleted), but 405 successfully added to the database. Logs2024-11-10T14:37:30.872Z [info] [subscriptions]: 1/411 New subscription: Aaron |
|
Wow, I didn't expect to see it implemented so fast. Thanks a bunch! Once I started bbyen from the latest version from the branch, I immediately got 72 new channels added. Seems to work as expected but I'll be testing this during this week and we'll report back if everything is fine. Also, does it mean now that if Whitelist is used, will it no longer require Google Auth or what will be the correct way of setting this up now?
You are correct. I did have some channels that were already deleted, that's why I did the cleanup afterwards. |
It might be possible to just use an API key instead of logging in with your Google account. I'd have to look into it. It's not as trivial as just stripping out Auth.
Sometimes Googling the ID reveals something, at least for video ids.
It deletes them. I could change this, but that doesn't help you with the ones that were already deleted. |
Oh I thought that it is not using Auth now. How does it work now then?
That's exactly how I found the 2 channels but as the last video ids were not in the database, for the rest of the channels it remains a mystery 😄
I know it wouldn't help with the channels that were already deleted, I'm just thinking about the future, in case this would happen again. It would be good if the database could keep them in it even after the channel was deleted. Also, as there is quite a lot of emails in the queue from the new channels, I have run into an issue where the quota has run out. |
The YouTube API (which needs Auth or an API key) is still used to look up the channel name and thumbnail based on the id in the configuration file, and it's also used to look up the information about newly uploaded videos. Now that it doesn't need to know your subscriptions, it could be an API key instead of auth, but it still needs something.
Should be easy.
The way it should work is that it logs something and tries again at your configured video refresh time (45 minutes). It should not crash. That's a bug. Can you please try this: https://stackoverflow.com/a/49505612 |
|
The latest commit should have the channel history thing. To view deleted/unsubscribed channels: |
Ah, got it. In the past before bbyen was created, my friend created a python script that did the same as bbyen but only with the rss feeds without Google Auth. While I do not know much about programming, it seems to me that it opened the video and got all the information from the html. I guess this is not efficient but it worked pretty well. Maybe the same could be implemented in bbyen as a fallback/alternative method? Here is it for inspiration, although the variable names are in our language, it should be understandable
I tried it but it only managed to send 85 emails, afterwards it crashed again with the same error :/
Thanks! Will try it out |
|
That's of course also possible, but it's more effort and I think it's against the terms of service. If I were to do it for bbyen, it would probably be an entire rewrite (maybe a nice opportunity to get off javascript). That error is very strange. I will see what I can find. |
Yes, I guess you are right about that. Although the python script worked decently well for :) In the meantime as I'm testing the latest version, I think bbyen no longer removes the channels that were deleted from the whitelist config. I have removed couple of them but I still received an email about it. |
I think you're right. Hang on. |
|
Please try the latest commit |
|
You're madman 🚀 Now the channels were removed. |
|
So upon further testing, even though the log said that the channels were removed, for some reason they are still being checked. |
|
My bad. Please try again. Thanks for helping me test this stuff out. |
|
No problem, I'm huge fan of this project :) |
|
What I now found out is that when you add a channel, then remove it and add it back in, the sqlite throws an error. By the way, in the meantime I found out that putting the whole channel URL into the config.json does not seem to work convert it to the channel ID anymore. I have tried to use all of the possible URL that Youtube uses with a valid channel (not all of the channel support the older format now), but the script threw an error immediately. Also found a way easier way to get the Channel ID from the user side, just by going into the |
|
Of course. That's annoying. The channel normalization only supported |
|
Thanks! It was fixed with the channel above although when I tried to add |
|
It seems like it was added in node 15, so it was a lie that bbyen supports node >= 14. Would it be a big deal for you to upgrade to 15? 14 is getting pretty old anyways. It's probably good to update this. Another reminder that I should have some automated tests checking each supported version. 14 is what I use on my server, but I don't use this feature so I didn't catch it. |
|
No problem, It's basically a container for me. After the update of nodejs to v15 it worked as expected :) Readme needs an update for that 😉 |
|
I updated the readme about the node version. I also added some rudimentary tests of the config (it would have caught "Promise.any is not a function"). I set this up to run on node 16-22 on every PR. |
|
Awesome, thanks a lot! I've been using it for the past few days and it looks like all of the bugs were squashed now. Seems ready for a release :) One last thing, I hope your paypal did not change in the meantime 😉 |
### Breaking changes - Increase the minimum supported node version from 14 to 16 This is the reason for the major release. However, the previous version was not really compatible with node 14. It was a mistake to list compatibility. ### New features - Do not use YouTube API to get whitelisted subscriptions (#19) - Do not permanently delete subscriptions. This shows you the channel name in the database if the channel is deleted. ### Maintenance - Add simple tests (currently only config) that run for all supported node versions
|
No problem. I just released 3.0.0. Paypal link should still work, but it's really not necessary. |
Hi, I've been using bbyen for quite a while now and I only use the whitelist feature instead of having notified of every new upload from every channel that I am subscribed to.
In the past week or so, I noticed that for some reason bbyen stopped checking all whitelisted channels and stops checking after around 854 channel out of 1162 (Yeah, I do have a lot of them :) )
I have not played with the configuration file other than occasionally adding the whitelisted channels into it since I set it up.
This was perfectly working for a year or so.
I enabled verbose logging but other than saying that it stopped checking, I don't see anything that would be more helpful.
Any idea if the database has some kind of limit that I might have gone over ? :)
Hre is the log and the config that I used with obfuscated credentials.
log_9.11.2024_1.txt
config.json
Edit: After checking Youtube channels that starts with letter T (As that is when the checking stops usually), I found that this has happened around 3 months ago as there videos from the past that I didn't see. Probably not useful to you though.
The text was updated successfully, but these errors were encountered: