Implement a Codex library so that clients can use Codex directly from their apps

[jrainville]

As a client of Codex, I would like to have access to its APIs and functionalities as part of a library instead of having to spin it in a different process manually.

This also aligns with the new goals of the IFT of having all the subprojects available as libraries that can be used by anyone.

The end goal is that as part of another app, we can start and use Codex features by calling them directly from our code instead of relying on a REST API. This can be done using C Bindings.

[emizzle]

Waku has put together some requirements for an ffi generation library that could possibly help here, however I believe work likely has not started yet. Have asked about latest status.

[emizzle]

11 Sept Update

• First version of go wrapper
• Can start/stop Codex node from go
• Draft PR
• Next: Add event callback for Codex started

[2-towns]

11 Sept Update

• First version of go wrapper
• Can start/stop Codex node from go
• Draft PR
• Next: Add event callback for Codex started

PR link: #1322

[2-towns]

I have this code:

  echo "dispatch the request..."

  # Dispatch the request to be processed asynchronously  
  asyncSpawn CodexThreadRequest.process(request, addr codex)
  
  echo "send received ack..."

  # Notify the main thread that we picked up the request
  let fireRes = ctx.reqReceivedSignal.fireSync()
  if fireRes.isErr():
    error "could not fireSync back to requester thread", error = fireRes.error

Then when I run the Go wrapper I can see the following logs:

dispatch the request...
processing request: LIFECYCLE
DBG 2025-09-11 16:50:55.141+02:00 About to accept incoming connection        topics="libp2p switch" tid=345515
...
INF 2025-09-11 16:51:03.165+02:00 Starting transport                         topics="discv5" tid=345515 bindAddress=0.0.0.0:8090
DBG 2025-09-11 16:51:03.165+02:00 Total nodes in routing table after populate topics="discv5" tid=345515 total=0
NTC 2025-09-11 16:51:03.165+02:00 Started codex node                         topics="codex node" tid=345515 id=16U*RMHtJj addrs="@[/ip4/127.0.0.1/tcp/42689, /ip4/192.168.68.55/tcp/42689]"
NTC 2025-09-11 16:51:03.165+02:00 REST service started                       tid=345515 address=127.0.0.1:8080
Codex node started!
send received ack...

So the ack, reqReceivedSignal, is fired after the Codex node is started.

Then I found this following quote :

The procedures are tagged with the {.async.} pragma which we will look more into later, and when calling other asynchronous procedures they use the await keyword to wait until that procedure is done. This means that when calling await the procedure tells the dispatcher that it is done executing for now and that it is free to execute other things until the awaited thing completes.

So what I understand is, the code will be ran in the same tick in the event loop, until a await keyword is used. In the library template, the await is used at this point:

handleRes(await retFut, request)

So it looks like the Codex start method run in the same tick because the await is called lately, so the ack is sent after that start method.

If I change the asyncSpawn to :

  asyncSpawn (proc() {.async.} =
    await sleepAsync(0)  # yield immediately
    await CodexThreadRequest.process(request, addr codex)
  )()

I get the ACK right after the dispatch

dispatch the request...
send received ack...
waiting for a request...

Am I right ?