Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change the database #549

Open
goto-bus-stop opened this issue Jan 4, 2023 · 6 comments · May be fixed by #637
Open

Change the database #549

goto-bus-stop opened this issue Jan 4, 2023 · 6 comments · May be fixed by #637
Assignees

Comments

@goto-bus-stop
Copy link
Member

goto-bus-stop commented Jan 4, 2023

We have relational data and I've had to spend a lot of time making Mongo do the things that we need. It's a bad fit for this type of application but it's always seemed more trouble than it's worth to change. However there are more issues than just fit for purpose. Mongo requires an annoying amount of maintenance to keep up-to-date, which we've needed to do because they are still adding stuff that lets you simulate relational models every now and then. There also aren't many affordable managed options for small applications.

If we switch to anything SQL-based, we get many more guarantees, and probably much better performance too. As it is we only have a few 100K medias at most on wlk.yt but loading 1 page of a playlist can already take 500ms because of all the relations we need to look up using fancy mongodb pipelines and whatnot. We could also improve it by changing the data model (we really didn't design a data model for mongo back in the day), but at that point we are already doing a complicated migration, and we might as well make a similar-effort migration for much larger gains.

Maybe 2023 is the year for this. I would like to avoid using an ORM. I think it would be nice if we could even work with sqlite (though i would probably recommend postgres in docs).

@goto-bus-stop goto-bus-stop self-assigned this Jan 4, 2023
@goto-bus-stop
Copy link
Member Author

@goto-bus-stop
Copy link
Member Author

@goto-bus-stop
Copy link
Member Author

goto-bus-stop commented May 31, 2024

I'm leaning more towards having sqlite as the standard option. My experience shows that you can decently well run an app on sqlite with tables containing millions of rows--realistically, this will never happen for üWave. And the operational work involved in setting up a sqlite-based app is just so much better than node.js + mongodb + redis. (This implies that I'd also move away from redis :). )
With kysely you probably still could use postgres if you wanted to.

The issue with sqlite in a node.js app is that it blocks the main thread, unless there's a thread pool with connections to use.

@fawaf
Copy link
Member

fawaf commented May 31, 2024

lol, didn't realize this issue is a year old. sqlite is a nice "lightweight" option for sure.

@goto-bus-stop
Copy link
Member Author

time flies when you're getting old 😭

@goto-bus-stop
Copy link
Member Author

Node.js now has an experimental built-in sqlite module: https://nodejs.org/api/sqlite.html

@goto-bus-stop goto-bus-stop linked a pull request Oct 9, 2024 that will close this issue
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants