Doc: documentation overhaul, cljdoc compatibility, version bumps (#9)

* Doc: documentation overhaul, cljdoc compatibility, version bumps

- Updated dependencies
- Adapter libraries now depend on the core SDK. There is no longer a
  need to explicitly depend on the core SDK.
- New articles have been added to the `doc` directory
- Several docstrings have been added / updated
- New babashka tasks to help running a local cljdoc instance and
  ingesting the docs locally
- The build process now adds libraries interdependencies automatically
- libraries are now in a libraries directory instead of the root of the
  project
- Malli schemas have been split into 3 libraries

Author: JeremS

.github/workflows/release-sdk.yml

@@ -59,40 +59,54 @@ jobs:
         run: bb test:bb
 
       - name: Build jar artifacts
-        run: bb jar:all
+        run: bb install:all
 
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # tag: v4.6.2
         with:
           name: sdk.jar
-          path: sdk/target/*.jar
+          path: libraries/sdk/target/*.jar
+          if-no-files-found: error
+          compression-level: 0
+
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # tag: v4.6.2
+        with:
+          name: brotli.jar
+          path: libraries/sdk-brotli/target/*.jar
           if-no-files-found: error
           compression-level: 0
 
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # tag: v4.6.2
         with:
           name: adapter-http-kit.jar
-          path: sdk-adapter-http-kit/target/*.jar
+          path: libraries/sdk-http-kit/target/*.jar
           if-no-files-found: error
           compression-level: 0
 
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # tag: v4.6.2
         with:
-          name: adapter-ring.jar
-          path: sdk-adapter-ring/target/*.jar
+          name: adapter-http-kit-malli-schemas.jar
+          path: libraries/sdk-http-kit-malli-schemas/target/*.jar
           if-no-files-found: error
           compression-level: 0
 
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # tag: v4.6.2
         with:
           name: malli-schemas.jar
-          path: sdk-malli-schemas/target/*.jar
+          path: libraries/sdk-malli-schemas/target/*.jar
           if-no-files-found: error
           compression-level: 0
 
       - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # tag: v4.6.2
         with:
-          name: brotli.jar
-          path: sdk-brotli/target/*.jar
+          name: adapter-ring.jar
+          path: libraries/sdk-ring/target/*.jar
+          if-no-files-found: error
+          compression-level: 0
+
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # tag: v4.6.2
+        with:
+          name: adapter-ring-malli-schemas.jar
+          path: libraries/sdk-ring-malli-schemas/target/*.jar
           if-no-files-found: error
           compression-level: 0
 

.gitignore

@@ -10,3 +10,7 @@ test-resources/test.config.edn
 /.nfnl.fnl
 /.nvim.fnl
 /.nvim.lua
+
+.cljdoc-preview
+.lazy.fnl
+.lazy.lua

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Release notes for the Clojure SDK
 
+## 2025-09-14 - RC2
+
+This release is mostly centered around documentation and cljdoc compatibility.
+
+### Changes
+
+- Several dependencies have been updated
+- Libraries interdependencies are now explicit in their `deps.edn`
+- Malli schemas are now split into 3 libraries. This comes from the need to be
+  cljdoc compatible
+
+### Added
+
+- New articles have been added to the `doc` directory
+- Several docstrings have been added / updated
+- There are new babashka tasks to help running a local cljdoc instance and
+  ingesting the docs locally
+- The build process updates libraries interdependencies versions automatically
+- A warning about which Http-kit version to use has been added in the proper
+  README
+
+### Fixed
+
+- cljdoc ingestion
+
 ## 2025-06-22
 
 ### Changed
@@ -19,7 +44,7 @@
 
 ### Fixed
 
-- A superflous newline character was send when marking the end of a SSE event
+- A superfluous newline character was send when marking the end of a SSE event
 - The clj-kondo config file for the SDK has been moved in a
   `clj-kondo.exports/starfederation.datastar.clojure/sdk` directory. This change
   allows for other projects to use

README.md

@@ -12,30 +12,30 @@ We provide several libraries for working with [Datastar](https://data-star.dev/)
 
 There currently are adapter implementations for:
 
-- [ring](https://github.com/ring-clojure/ring)
+- [ring compliant adapters](https://github.com/ring-clojure/ring)
 - [http-kit](https://github.com/http-kit/http-kit)
 
 If you want to roll your own adapter implementation, see
 [implementing-adapters](/doc/implementing-adapters.md).
 
-## Installation
+## Library coordinates
 
-To your `deps.edn` file you can add the following coordinates:
 
-| library       | deps coordinate                                                                                                                                         |
-| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| library       | deps coordinate                                     |
+| ------------- | ----------------------------------------------------|
 | SDK           | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/sdk.svg)](https://clojars.org/dev.data-star.clojure/sdk)                     |
 | http-kit      | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/http-kit.svg)](https://clojars.org/dev.data-star.clojure/http-kit)           |
 | ring          | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/ring.svg)](https://clojars.org/dev.data-star.clojure/ring)                   |
 | brotli        | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/brotli.svg)](https://clojars.org/dev.data-star.clojure/brotli)               |
-| malli-schemas | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/malli-schemas.svg)](https://clojars.org/dev.data-star.clojure/malli-schemas) |
+| Core SDK malli schemas | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/malli-schemas.svg)](https://clojars.org/dev.data-star.clojure/malli-schemas) |
+| Http-kit malli schemas | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/http-kit-malli-schemas.svg)](https://clojars.org/dev.data-star.clojure/http-kit-malli-schemas) |
+| Ring malli schemas | [![Clojars Project](https://img.shields.io/clojars/v/dev.data-star.clojure/ring-malli-schemas.svg)](https://clojars.org/dev.data-star.clojure/ring-malli-schemas) |
 
-Notes:
-
-- You need the SDK and either the http-kit or the ring library to get started.
+- To get started you'll need either the http-kit library or the ring one.
+- Other libraries are optional
 - The ring library works with ring compliant adapters (adapter using the
   `ring.core.protocols/StreamableResponseBody`)
-- Currently the brotli library works only with http-kit
+- Currently the Brotli library works only with http-kit
 
 ## Usage
 
@@ -96,9 +96,6 @@ it somewhere and use it later:
 
 ```
 
-Check the docstrings in the `starfederation.datastar.clojure.api` namespace for
-more details.
-
 ### Advanced features
 
 This SDK is essentially a tool to manage SSE connections with helpers to format
@@ -118,40 +115,6 @@ You can find more information in several places:
   lists the conventions by which SDK adapters are implemented if the need to
   implement your own ever arises.
 
-## Adapter behaviors:
-
-Ring adapters are not made equals. This leads to our SSE generators not having
-the exact same behaviors in some cases.
-
-### SSE connection lifetime in ring when trying to store the sse-gen somewhere
-
-#### With ring sync
-
-| Adapter  |                                                                        |
-| -------- | ---------------------------------------------------------------------- |
-| ring     | same as the thread creating the sse response                           |
-| http-kit | alive until the client or the server explicitely closes the connection |
-
-You may keep the connection open in ring sync mode by somehow blocking the thread
-handling the request. This is a valid strategy when using java's virtual threads.
-
-#### With ring async
-
-| Adapter      |                                                                        |
-| ------------ | ---------------------------------------------------------------------- |
-| all adapters | alive until the client or the server explicitely closes the connection |
-
-### Detecting a closed connection
-
-| Adapter  |                                                                                                 |
-| -------- | ----------------------------------------------------------------------------------------------- |
-| ring     | Sending events on a closed connection will fail at some point and the sse-gen will close itself |
-| http-kit | Http-kit detects closed connections by itself and closes the sse-gen                            |
-
-At this moment, when using the ring adapter and jetty, our SSE generators need
-to send 2 small events or 1 big event to detect a closed connection.
-There must be some buffering happening independent of our implementation.
-
 ## TODO:
 
 - Streamlined release process (cutting releases and publish jar to a maven repo)

bb.edn

@@ -1,16 +1,23 @@
 {:paths ["src/bb"
-         "sdk/src/main"
-         "sdk-adapter-http-kit/src/main/"
-         "sdk-malli-schemas/src/main"
+         "libraries/sdk/src/main"
+         "libraries/sdk-http-kit/src/main/"
+         "libraries/sdk-http-kit-malli-schemas/src/main"
+         "libraries/sdk-malli-schemas/src/main"
          "./src/test/core-sdk"
          "./src/test/malli-schemas"]
 
- :deps {io.github.noahtheduke/lazytest {:mvn/version "1.8.0"}
-        metosin/malli                  {:mvn/version "0.17.0"}}
+ :deps {borkdude/rewrite-edn           {:mvn/version "0.4.9"}
+        io.github.noahtheduke/lazytest {:mvn/version "1.8.0"}
+        metosin/malli                  {:mvn/version "0.19.1"}}
 
  :tasks
- {:requires ([tasks :as t])
+ {:requires ([tasks        :as t]
+             [tasks.build  :as build]
+             [tasks.cljdoc :as cljdoc])
 
+  ;; ---------------------------------------------------------------------------
+  ;; Dev tasks
+  ;; ---------------------------------------------------------------------------
   dev
   {:doc "Starts a dev repl. Additional deps aliases can be passed as arguments."
    :task (t/dev :http-kit :ring-jetty :malli-schemas)}
@@ -22,11 +29,14 @@
   dev:empty
   {:doc "Dev repl with no adapters."
    :task (t/dev :malli-schemas)}
-  
+ 
   dev:bb
   {:doc "Start a babashka nrepl server. An addr like `localhost:1234` can be passed as argument."
    :task (t/dev-bb)}
 
+  ;; ---------------------------------------------------------------------------
+  ;; Test Tasks
+  ;; ---------------------------------------------------------------------------
   test:all
   {:doc "Run all tests once."
    :task (t/lazytest [:http-kit
@@ -67,31 +77,37 @@
   {:doc "Run core unit tests with babashka."
    :task (t/bb-lazytest [:test.paths/core-sdk
                          :test.paths/malli-schemas])}
-  
+
   test:sdk-common
   {:doc "Run the server used for the SDKs' common tests."
    :task (t/start-test-server)}
 
+  test:sdk-common-go
+  {:doc "Run the server used for the SDKs' common tests."
+   :task (t/run-go-tests)}
 
+  ;; ---------------------------------------------------------------------------
+  ;; Build tasks
+  ;; ---------------------------------------------------------------------------
   jar:sdk
   {:doc  "Build jar for the common sdk"
-   :task (t/lib-jar! t/sdk-dir)}
+   :task (build/lib-jar! build/sdk-dir)}
 
   jar:sdk-adapter-ring
   {:doc  "Build jar for the adapter-ring"
-   :task (t/lib-jar! t/sdk-adapter-ring-dir)}
+   :task (build/lib-jar! build/sdk-adapter-ring-dir)}
 
   jar:sdk-adapter-http-kit
   {:doc  "Build jar for the adapter-http-kit"
-   :task (t/lib-jar! t/sdk-adapter-http-kit-dir)}
+   :task (build/lib-jar! build/sdk-adapter-http-kit-dir)}
 
   jar:sdk-brotli
   {:doc  "Build jar for the Brotli library"
-   :task (t/lib-jar! t/sdk-brotli-dir)}
+   :task (build/lib-jar! build/sdk-brotli-dir)}
 
   jar:sdk-malli-schemas
   {:doc  "Build jar for the malli-schemas"
-   :task (t/lib-jar! t/sdk-malli-schemas-dir)}
+   :task (build/lib-jar! build/sdk-malli-schemas-dir)}
 
   jar:all
   {:doc     "Build the jar for all the libs"
@@ -102,30 +118,54 @@
              jar:sdk-brotli
              jar:sdk-malli-schemas]}
 
+  install:clean
+  {:doc "Deletes all datastar libs from the local maven repository."
+   :task (build/clean-maven-dir!)}
+
+
   install:all
-  {:doc "Install all libraries"
+  {:doc "Install all libraries locally."
    :depends [clean]
-   :task (doseq [dir t/sdk-lib-dirs]
-           (t/lib-install! dir))}
+   :task (doseq [dir build/sdk-lib-dirs]
+           (build/lib-install! dir))}
 
   clean
-  {:doc  "Clean build artifacts"
-   :task (doseq [dir t/sdk-lib-dirs]
+  {:doc  "Clean build artifacts in each library directory."
+   :task (doseq [dir build/sdk-lib-dirs]
+           (println "----------------")
+           (println "Cleaning: " dir)
+           (println "----------------")
            (clojure {:dir dir} "-T:build clean"))}
 
   set-version
   {:doc  "Set the version in all libs"
-   :task (doseq [dir t/sdk-lib-dirs]
-           (t/lib-set-version! dir (first *command-line-args*)))}
+   :task (doseq [dir build/sdk-lib-dirs]
+           (build/lib-set-version! dir (first *command-line-args*)))}
 
   bump-version
   {:doc  "Bump the version component in all libs. First argument must be one of: major, minor, patch"
-   :task (doseq [dir sdk-lib-dirs]
+   :task (doseq [dir build/sdk-lib-dirs]
            (let [component (first *command-line-args*)]
-             (t/lib-bump! dir component)))}
+             (build/lib-bump! dir component)))}
 
   publish:all
   {:doc     "Publish the clojure sdk libs to clojars"
    :depends [jar:all]
-   :task    (doseq [dir t/sdk-lib-dirs]
-              (t/lib-publish! dir))}}}
+   :task    (doseq [dir build/sdk-lib-dirs]
+              (build/lib-publish! dir))}
+
+  ;; ---------------------------------------------------------------------------
+  ;; Cljdoc tasks
+  ;; ---------------------------------------------------------------------------
+  cljdoc:clean
+  {:doc "Clean local cljdoc data."
+   :task (cljdoc/clean!)}
+
+  cljdoc:server
+  {:doc "Start a local cljdoc server."
+   :task (cljdoc/start-server!)}
+
+  cljdoc:ingest
+  {:doc "Tell local cljdoc server to ingest a library. Needs a argument (:sdk, :http-kit, :ring, ...)"
+   :task (cljdoc/ingest! (first *command-line-args*))}}}
+