Adding comfylite3

[davidroman0O]

Adding the comfylite3 wrapper of the famous go-sqlite3 driver which compensate the lack of goroutine support by giving the illusion of it.

Most other libraries that re-implement the entire sqlite3 driver won't support the latest features, like the recent JSON datatype, sometimes you just want it all!

No dependency added, very similar to the mysql implementation, eventually someone else would want to add a specific sqlite implementation for a pure golang version since go-sqlite3 might require CGO!

At least, it gives more options! Using sqlite3 with watermill for small side projects is very cool, especially when using an adapter pattern to switch between a real mysql to sqlite3 with the same implementation!

Nothing fancy in the code! 😄

[davidroman0O]

Also here an example

package main

import (
	"context"
	"log"
	"time"

	"github.com/ThreeDotsLabs/watermill"
	"github.com/ThreeDotsLabs/watermill-sql/v3/pkg/sql"
	"github.com/ThreeDotsLabs/watermill/message"
         "github.com/davidroman0O/comfylite3"
)

func main() {

	db, err := comfylite3.Comfy(
		comfylite3.WithMemory(),
	)
	if err != nil {
		panic(err)
	}

	logger := watermill.NewStdLogger(false, false)

	subscriber, err := sql.NewSubscriber(
		db,
		sql.SubscriberConfig{
			SchemaAdapter:    sql.DefaultSQLite3Schema{},
			OffsetsAdapter:   sql.DefaultSQLite3OffsetsAdapter{},
			InitializeSchema: true,
		},
		logger,
	)
	if err != nil {
		panic(err)
	}

	messages, err := subscriber.Subscribe(context.Background(), "example_topic")
	if err != nil {
		panic(err)
	}

	go process(messages)

	publisher, err := sql.NewPublisher(
		db,
		sql.PublisherConfig{
			SchemaAdapter: sql.DefaultSQLite3Schema{},
		},
		logger,
	)
	if err != nil {
		panic(err)
	}

	publishMessages(publisher)
}

func publishMessages(publisher message.Publisher) {
	for {
		msg := message.NewMessage(watermill.NewUUID(), []byte(`{"message": "Hello, world!"}`))

		if err := publisher.Publish("example_topic", msg); err != nil {
			panic(err)
		}

		time.Sleep(time.Second)
	}
}

func process(messages <-chan *message.Message) {
	for msg := range messages {
		log.Printf("received message: %s, payload: %s", msg.UUID, string(msg.Payload))

		// we need to Acknowledge that we received and processed the message,
		// otherwise, it will be resent over and over again.
		msg.Ack()
	}
}

[pbedat]

Hi @davidroman0O

I'm a big fan of SQLite and run a successful SaaS on it. While integrating Watermill, I looked into your pub/sub implementation (https://github.com/davidroman0O/watermill-comfymill) but struggled to understand its value proposition:

Aren't you tired to be forced to write a different code for sqlite3 because you can't use it with multiple goroutines?

From my experience, the main challenge with SQLite is handling nested transactions within the same goroutine, which often leads to deadlocks—usually a sign of structural issues.

Our service handles many concurrent reads and writes. While SQLite doesn’t support multiple concurrent write transactions, SetMaxOpenConns(1), WAL mode, and a proper busy timeout usually mitigate this. Even if a transaction fails due to a timeout, event-driven architectures can handle retries gracefully. Personally, I’d rather surface such bottlenecks at the application level than abstract them away with a wrapper.

So my main question is: Compared to a pure go-sqlite3 implementation, what are the concrete benefits of using comfylite for pub/sub?

P.S. One of the key benefits of SQL-based pub/sub is transactional integrity, which works well with databases like PostgreSQL. However, with SQLite, you can’t consume an event and process it transactionally in the same way—you'd have to pass the transaction context to the event handler, which impacts separation of concerns. Any SQLite integration with Watermill-SQL should come with proper documentation and warnings.
To work around this, we use the outboxer pattern and a second SQLite database dedicated to pub/sub. However, we're closely monitoring its throughput and already planning a move to PostgreSQL, as SQLite’s single-writer limitation becomes a major bottleneck (no concurrent event handling).

[davidroman0O]

Hey there, thanks for the thorough feedback!

I built comfylite3 mainly for my own convenience—especially for smaller side projects and end-to-end tests where spinning up a full-fledged DB like PostgreSQL feels like overkill. It’s basically a drop-in replacement for go-sqlite3 that takes care of the usual “database locked” headaches behind the scenes. That way, I can just toss in SQLite without worrying too much about concurrency pitfalls in my own code.

I totally agree, though, that SQLite’s single-writer limit is very real, and if concurrency or transactional consistency is critical, you’re going to hit a wall eventually. The outbox approach or switching to a dedicated DB like PostgreSQL is still the best path for serious workloads.

As for the PR, I’d be happy to improve docs and warnings around these trade-offs—making sure folks know that comfylite3 just helps with lightweight concurrency friction, but doesn’t turn SQLite into a magically scalable solution. Let me know if there’s anything specific you’d like me to add or clarify!

Thanks again for your thoughts. I appreciate your insights and will keep them in mind.

[pbedat]

Continuing the discussion here davidroman0O/comfylite3#7
Don't want to spam this watermill-sql PR ;)