Migration of Backend from NestJS to Go

[devleejb]

What would you like to be added:
The current backend is structured using NestJS, and there is a proposal to migrate it to Go. The primary reason for this migration is to engage contributors who have a stronger interest in backend development.

The migration should take into account the following considerations:

• Implement LangChain functionality in the Go environment.
• Utilize OAuth for user authentication and authorization.
• Ensure proper validation mechanisms are in place.
• Verify that the new backend operations are consistent with the existing functionality.
• Generate Swagger documentation for the API in the new setup.

Why is this needed:
Migrating to Go may enhance our ability to attract backend-focused contributors and improve the overall performance and scalability of our application. This transition also aims to maintain functionality while leveraging Go's efficiency and capabilities.

Additional Information:

[devleejb]

@blurfx @krapie @window9u Could you share your thoughts on migration and any considerations?

[window9u]

@devleejb

I completely agree that migrating the CodePair backend to Go aligns well with Yorkie contributors' expertise, making collaboration more natural. If this proposal moves forward, I’m excited to join the migration team and contribute to building a robust Go-based backend!

Once a new repository for the Go-based backend is created, I suggest we discuss key decisions like:

1. HTTP Framework: Should we use Gin, Echo, or native middleware?
2. MongoDB Integration: Options include the official mongo-go-driver or tools like mgm for simplifying operations.

[devleejb]

@window9u

Thank you for your answer!

1. HTTP Framework: Should we use Gin, Echo, or native middleware?

I understand that yorkie uses native middleware. Is that correct? If so, we can use it as a valuable reference.

2. MongoDB Integration: Options include the official mongo-go-driver or tools like mgm for simplifying operations.

yorkie also uses mongo-go-driver. Additionally, we reference the database architecture in yorkie to support other options (e.g., MySQL, and more).

[blurfx]

I think fiber is a good option for the http framework. It's very fast and has some nice features.

[window9u]

@devleejb

HTTP Framework

As far as I know, yes, Yorkie uses native middleware. However, this decision likely stems from its need to handle more complex logic, such as RPC connections and health checks. Here's an example of Yorkie code:

mux := http.NewServeMux()
mux.Handle(v1connect.NewYorkieServiceHandler(newYorkieServer(yorkieServiceCtx, be), opts...))
mux.Handle(v1connect.NewAdminServiceHandler(newAdminServer(be, tokenManager), opts...))
mux.Handle(v1connect.NewClusterServiceHandler(newClusterServer(be, opts...)))
mux.Handle(grpchealth.NewHandler(healthChecker))
mux.Handle(httphealth.NewHandler(healthChecker))

For CodePair, if the logic mainly involves simple CRUD operations, using a framework could be a good choice to streamline development. But I’ve only worked with native HTTP handlers, so I don’t have much experience with frameworks. I trust @blurfx’s recommendation would be more informed in this case.

MongoDB Integration

Since Yorkie uses the official mongo-go-driver, it’s a dependable choice for CodePair as well. I also agree with adding support for relational databases as an option, as it could increase flexibility and scalability.

Additionally, I’ve been interested in exploring semantic search using vector databases for CodePair. When considering other database options, it might be valuable to account for this use case. I’ve heard that PostgreSQL offers excellent scalability and extensions for features like vector search, making it worth considering for future-proofing the architecture.

[blurfx]

I've used some go http frameworks like net/http, fasthttp, and fiber before and had no problem adapting to any of the three libraries/frameworks. the three of them provides a similar developer experience to most web frameworks. If you have a experience with net/http in go, or express or koa in node.js, I think you're probably already familiar with it.

Database:

I'm curious why you chose MongoDB for your existing CodePair, and what you've liked and disliked about it.
Changing databases is a quite large task and MongoDB is relatively scalable than other databases, so I think we can keep it, but as @window9u said, PostgreSQL might be a good option. there are many (third-party) extensions that can do full text search or message queue, etc...

[devleejb]

@blurfx I chose MongoDB to utilize Yorkie’s existing database. To clarify, I am referring to supporting a new database type like yorkie (MongoDB, In-Memory), not replacing the production database.

[blurfx]

I misread that. I understand now.

[devleejb]

@blurfx Could you recommend a framework? Actually, I’ve been considering using Yorkie’s architecture to accommodate contributions from Yorkie contributors.

[blurfx]

@devleejb I don't know the workload of CodePair server, but I can suggest the following options:

Performance - Fiber
Ecosystem - Echo
Minimal, lightweight, (not framework) - chi

---

I personally prefer fiber but is not compatible net/http interface so there might be issues with using third-party packages.

So I think echo would be great with a large ecosystem and a well-documented and built-in features.

[hackerwins]

@blurfx @devleejb
I think it would be good to create a few backend scaffolding PRs and compare them.

[krapie]

@devleejb @hackerwins @devleejb @window9u

I have experimentally ported the backend to a Go server application using both echo and net/http for comparison, with the help of Cursor(Claude).

• Echo-based Go server
• Net/http-based Go server

Please note that some modules might be missing or not fully implemented.

[devleejb]

@krapie
Thank you for your hard work! It’s truly amazing!
I think echo works really well for codepair. I’ll take a closer look at the code this weekend.
If you have any suggestions, please feel free to share them!

I’ll put together a migration plan (including target APIs, timeline, validation, etc.).

I’m aiming to release version v0.2 along with the Go migration. :)

[devleejb]

@sejongk I’m also curious about your thoughts.