Plans for API stability or stable releases?

[liujimj]

Have you given any thought to slowing the pace of breaking changes in the future, or is the intention to continue with 5+ major versions per year?

For context, we have a moderately-sized codebase that was migrated from node-pg to slonik@27. We love many of Slonik's out-of-the-box features (e.g. transaction retries, nested transactions mapping to savepoints) and have generally had a great experience!

However, keeping pace with later releases has been a bit of a slog. Some of the larger changes that has affected our team:

1. Moving from sql<T> to sql.type(zodValidator) suddenly made non-zod usage unergonomic (this was before sql.unsafe was introduced, which we would've used instead for a less disruptive migration path).
2. Similarly, we built wrappers around slonik that made liberal use of sql fragments which now require manual annotation as sql.fragment
3. The shift from no validation, to enforced validation, to optional validation via interceptor made us feel a little bit whipsawed. Since it seems like some major design decisions may be softly walked back in the next version, it can be tricky to identify a good version to upgrade to or whether it's worth the cost to refactor our code in the meantime.
4. Database pool creation became async, which forced async initialization to flow all the way to main in multiple entry points.

Again, we think Slonik is a great project, and would only get better with some clarity on the roadmap, or ways for users to upgrade without forcing large refactors.

Just as a point of comparison, similar projects have generally had fewer major versions / year:

• node-pg - 8 versions over 12 years (0.7 / year)
• postgres.js - 3 versions over 3 years (1 / year)
• pg-promise - 11 versions over 7 years (1.6 / year)

While this depends on how rigorously semver is applied, it may serve as a rough gauge of stability.

[markcellus]

Yeah, it appears major versions are used frequently because it's easier than having to deal with backwards compatibility. Some major versions just change syntax without no underlying improvement, similar to point 1 in your list. For example, #438 (reply in thread) is where I expressed how a recent major version has required me to have to update code across multiple repos for a syntax change in the API that did not improve the code logic in any way and could've been easily backwards-compatible. Really hope some more thought goes into backwards-compatibility in future developments of the package.

[AndrewJo]

On one hand, I appreciate that Slonik introduces opinionated structure around making safe queries to the database. On the other hand, I do agree with some sentiments around sudden major backwards breaking changes introduced throughout the past year.

Constant backwards breaking API changes hurts the community and the ecosystem around the library. I myself have built packages that depended on Slonik and having to release major version changes just to stay compatible with latest sudden changes to the API surface has been challenging to keep up.

If you dig around, it's not a secret that it's been putting people off from wanting to develop an ecosystem around Slonik:

• Sql typing no longer works with interface types #410 (comment)

I think @liujimj did a great job of being solutions oriented when he brought up the problems he perceived with the project so in the same spirit, I would like to expand on that with several more.

• Option 1: Having a deprecation path

  I think what would make Slonik better is an adoption of deprecation policy/path prior to major version changes so that the community has time to prepare and adjust as necessary. Here are few examples of open source projects that do a good job of this:

  • Vue.js
  • Django
  • Angular.js
  • Ember.js
  • Adonis.js

• Option 2: Pre-releases

  Another option is doing pre-releases so that the community has time to react to the changes and are aware of the changes coming up. Zod project itself was excellent at how they handled this with multiple -alpha releases and a -beta release prior to v3 changeover. Seeing the Slonik release history, there were no fewer than 3 major version releases last quarter: v31, v32, v33.

  No one would complain about the backwards breaking changes @liujimj mentioned above had these been released as 32.0.0-alpha.1, 32.0.0-alpha.2, 32.0.0-beta.1 since no one expects API stability in pre-releases. It also gives the community an ample opportunity to see the upcoming breaking changes and possibly provide feedback before the API surface is "locked" until next major version release.

• Option 3: RFCs

  A common way projects introduce changes is through RFC processes. Vue.js, Ember.js, Adonis.js are examples of projects that foster community involvement around proposed changes. I think this is harder to implement without a group of core maintainers.

• Option 4: LTS versions

  There's a reason why LTS versions are a thing in open source projects like Node.js. Many companies want their engineers to build products and features that deliver value and see any time spent paying down tech debt as time and money sink. It's why big enterprises pay the big bucks for RHEL (if they've got deep pockets) or go with distros like CentOS¹ instead of installing upstream-but-bleeding-edge Fedora. Perceived stability is a real deciding factor for lot of engineering managers (including myself).

Finally, I understand that it's not easy to run an open source project as a single individual and some of the options I proposed above requires dedicated group of core maintainers.

Honestly, it's amazing how much work @gajus has put in and building out a library like this from scratch. Heck, he didn't need to open source it but did it anyways. Looking at his BuyMeACoffee profile, he's barely getting any monetary support to continue working on it. Regardless, he keeps pushing code anyway even if it comes at the cost of breaking changes. So I understand he is passionate about the impact he's making with his code.

But I when I see feedback like this, it's usually a sign that an open source project is maturing and it has a group of passionate followers that feel strongly about the project one way or the other to provide the feedback. If Slonik wasn't a compelling library for anyone to care, I doubt people would bother leaving feedback like this. So I do hope that he takes the community feedback to heart.

PS: Just to be clear, I don't think Gajus is in any way obligated to listen to our feedback. He's not getting paid for this and nothing stops you from forking this project if you don't agree the direction Slonik is heading. That's the beauty of open source. :)

---

¹: CentOS itself lost a lot of community goodwill when they were put upstream of RHEL by Red Hat which resulted in mass community exodus to Rocky Linux and AlmaLinux. The community understandably felt that they turned into unpaid beta testers for RHEL with introduction of CentOS Stream.

[gajus]

Thank you for your input. All great points and valid observations.

Have you given any thought to slowing the pace of breaking changes in the future, or is the intention to continue with 5+ major versions per year?

It is definitely not an "intention". However, Slonik development has always been driven by learning from integrating it into new projects (most recently Contra, a company I co-founded), and hearing feedback from the community. It is an iterative process, and every layer of improvement has revealed a new milestone. So it is not an "intention" to continue with breaking changes, but neither do I want to commit to no breaking changes (as this is how innovation dies).

That being said, I am personally aware of at least a handful of large companies using Slonik as their primary PostgreSQL client. So it may require rethinking the project governance structure, and I am open to it.

Andrew shared some great suggestions. However, they all have a significant overhead or tradeoffs. I personally cannot commit to them, but I will talk with a few people that I know who maintain open-source projects professionally and get their opinion about it. There could be a path to creating more structured governance with financial incentives to participants.

In the interim, there are a few things that I can commit to. Namely, I can be more pro-active in starting conversations about ideas for future improvements. I've already tried that with a few more recent changes. However, they did not get much engagement. I am open to ideas how to execute this better. I am also aware that the documentation for breaking changes when they do happen needs to be better and must include a migration guide.

Finally, as we continue to search for the path forward, just to give some confidence about the near- to medium-term future of Slonik – I have personally no plans to make further breaking changes to the API. We have been using it in a very large codebase for a while, and are very happy with how everything comes together.

[akutruff]

Reposting my comment from slonik tools here.

[ghost]

As I noted in a separate discussion, the documentation doesn't seem to incorporate the various changes in syntax so slonik just seems broken for the user/developer.

[gajus]

As I noted in a separate discussion, the documentation doesn't seem to incorporate the various changes in syntax so slonik just seems broken for the user/developer.

What's an example?

I am not aware of documentation being out of sync with the API.

[gajus]

Closing as there is no active work here to be done.