Feature/connection loaders

.github/workflows/docgen.yml

@@ -33,19 +33,20 @@ jobs:
         run: |
           TEMP_CONFIG="$(mktemp)"
           TEMP_README="$(mktemp)"
-          # Retrieve default config and put it in a temp file
+          # Retrieve default config and put it in a temp file.
           {
             echo '```lua'
             awk '/DOCGEN_END/{f=0} f; /DOCGEN_START/{f=1}' lua/dbee/config.lua
             echo '```'
           } > "$TEMP_CONFIG"
-          # Insert the default config between DOCGEN_CONFIG tags in the README
+          # Insert the default config between DOCGEN_CONFIG tags in the README.
+          # And remove stuff between DOCGEN_IGNORE_START and DOCGEN_IGNORE_END tags from README.
           {
             sed -e '
               /DOCGEN_CONFIG_START/,/DOCGEN_CONFIG_END/!b
               /DOCGEN_CONFIG_START/r '"$TEMP_CONFIG"'
               /DOCGEN_CONFIG_END:/!d
-            ' <(sed '0,/DOCGEN_START/d' README.md)
+            ' <(sed '/DOCGEN_IGNORE_START/,/DOCGEN_IGNORE_END/d' README.md)
             cat ARCHITECTURE.md
           } > "$TEMP_README"
           # Generate docs

ARCHITECTURE.md

@@ -28,11 +28,11 @@ responsibilities.
                          │
                          │
                          │
-                         │      ┌─────────────┐
-                         │      │             │
-                         │      │   install   │
-                         │      │             │
-                         │      └─────────────┘
+                         │      ┌─────────────┐     ┌────────────┐
+                         │      │             │     │            │
+                         │      │   install   │     │   loader   │
+                         │      │             │     │            │
+                         │      └─────────────┘     └────────────┘
                          │
 ```
 
@@ -43,6 +43,8 @@ Description:
 - `install` package is independent of the other packages and is used for
   installation of the compiled go binary using the manifest generated by the CI
   pipeline.
+- `loader` package is also independent and is used as the default loading and
+  saving method in the config, which is later consumed by the handler
 - `drawer` is the "tree" view in UI - it consumes the editor (to provide
   scratchpad view and to manage scratchpads) and the handler (for managing
   connections and providing layout of each database).

README.md

@@ -1,11 +1,13 @@
-<!-- Any html tags, badges etc. go before this tag. -->
+<!-- DOCGEN_IGNORE_START -->
+
+<!-- This tag controlls what's ignored by the docgen workflow. -->
 
 ![Linting Status](https://img.shields.io/github/actions/workflow/status/kndndrj/nvim-dbee/lint.yml?label=linting&style=for-the-badge)
 ![Docgen Status](https://img.shields.io/github/actions/workflow/status/kndndrj/nvim-dbee/docgen.yml?label=docgen&logo=neovim&logoColor=white&style=for-the-badge)
 ![Backend](https://img.shields.io/badge/go-backend-lightblue?style=for-the-badge&logo=go&logoColor=white)
 ![Frontend](https://img.shields.io/badge/lua-frontend-blue?style=for-the-badge&logo=lua&logoColor=white)
 
-<!--DOCGEN_START-->
+<!-- DOCGEN_IGNORE_END -->
 
 # Neovim DBee
 
@@ -69,6 +71,12 @@
 
 ### Platform Support
 
+<!-- DOCGEN_IGNORE_START -->
+
+<details>
+  <summary>Click to expand</summary>
+<!-- DOCGEN_IGNORE_END -->
+
 This project aims to be as cross-platform as possible, but there are some
 limitations (for example some of the go dependencies only work on certain
 platforms).
@@ -82,8 +90,19 @@ created.
 So to check if your platform is currently supported, check out the mentioned
 manifest
 
+<!-- DOCGEN_IGNORE_START -->
+
+</details>
+<!-- DOCGEN_IGNORE_END -->
+
 ### Manual Binary Installation
 
+<!-- DOCGEN_IGNORE_START -->
+
+<details>
+  <summary>Click to expand</summary>
+<!-- DOCGEN_IGNORE_END -->
+
 The installation examples include the `build`/`run` functions, which get
 triggered once the plugin updates. This should be sufficient for the majority of
 users. If that doesn't include you, then you have a few options:
@@ -109,13 +128,36 @@ users. If that doesn't include you, then you have a few options:
   go build [-o ~/.local/share/nvim/dbee/bin/dbee]
   ```
 
-## Quick Start
+<!-- DOCGEN_IGNORE_START -->
+
+</details>
+<!-- DOCGEN_IGNORE_END -->
+
+## Configuration
+
+You can pass an optional table parameter to `setup()` function.
+
+Here are the defaults:
+
+<!--DOCGEN_CONFIG_START-->
+
+<!-- Contents from lua/dbee/config.lua are inserted between these tags for docgen. -->
+
+[`config.lua`](lua/dbee/config.lua)
+
+<!--DOCGEN_CONFIG_END-->
+
+## Usage
 
 Call the `setup()` function with an optional config parameter. If you are not
 using your plugin manager to lazy load for you, make sure to specify
 `{ lazy = true }` in the config.
 
-Here is a brief refference of the most useful functions:
+<!-- DOCGEN_IGNORE_START -->
+
+<details>
+  <summary>Brief reference (click to expand):</summary>
+<!-- DOCGEN_IGNORE_END -->
 
 ```lua
 -- Open/close the UI.
@@ -131,19 +173,201 @@ require("dbee").execute(query)
 require("dbee").save(format, file)
 ```
 
-## Configuration
+<!-- DOCGEN_IGNORE_START -->
 
-As mentioned, you can pass an optional table parameter to `setup()` function.
+</details>
+<!-- DOCGEN_IGNORE_END -->
 
-Here are the defaults:
+### Getting Started
 
-<!--DOCGEN_CONFIG_START-->
+Here are a few steps to quickly get started:
 
-<!-- Contents from lua/dbee/config.lua are inserted between these tags for docgen. -->
+- call the `setup()` function in your `init.lua`
 
-[`config.lua`](lua/dbee/config.lua)
+- Specify connections using one or more sources (reffer to
+  [this section](#specifying-connections)).
 
-<!--DOCGEN_CONFIG_END-->
+- When you restart the editor, call `lua require("dbee").open()` to open the UI.
+
+- Navigate to the drawer (tree) and use the following key-bindings to perform
+  different actions depending on the context (the mappings can all be changed in
+  the config):
+
+  - All nodes:
+
+    - Press `o` to toggle the tree node.
+    - Press `r` to manually refresh the tree.
+
+  - Connections:
+
+    - Press `cw` to edit the connection
+    - Press `dd` to delete it (if source supports saving, it's also removed from
+      there - see more below.)
+    - Press `<CR>` to perform an action - view history or look at helper
+      queries.
+
+  - Scratchpads:
+
+    - Press `<CR>` on the `new` node to create a new scratchpad.
+    - When you try to save it to disk (`:w`), the path is automatically filled
+      for you. You can change the name to anything you want, if you save it to
+      the suggested directory, it will load the next time you open DBee.
+    - Press `cw` to rename the scratchpad.
+    - Press `dd` to delete it (also from disk).
+    - Pressing `<CR>` on an existing scratchpad in the drawer will open it in
+      the editor pane.
+
+  - Help:
+
+    - Just view the key bindings.
+
+- Once you selected the connection and created a scratchpad, you can navigate to
+  the editor pane (top-right by default) and start writing queries. In editor
+  pane, you can use the following actions:
+
+  - Highlight some text in visual mode and press `BB` - this will run the
+    selected query on the active connection.
+  - If you press `BB` in normal mode, you run the whole scratchpad on the active
+    connection.
+
+- If the request was successful, the results should appear in the "result"
+  buffer (bottom one by default). If the total number of results was lower than
+  the `page_size` parameter in config (100 by default), all results should
+  already be present. If there are more than `page_size` results, you can "page"
+  thrugh them using one of the following:
+
+  - Using `require("dbee").next()` and `require("dbee").prev()` from anywhere
+    (even if your cursor is outside the result buffer).
+  - Using `L` for next and `H` for previous page if the cursor is located inside
+    the results buffer.
+
+- The current result (of the active connection) can also be saved to a file
+  using `require("dbee").save()` command. Use:
+
+  - `require("dbee").save("csv", "/path/to/file.csv")` for csv and
+  - `require("dbee").save("json", "/path/to/file.json")` for json.
+
+- Once you are done or you want to go back to where you were, you can call
+  `require("dbee").close()`.
+
+### Specifying Connections
+
+Connection represents an instance of the database client (i.e. one database).
+This is how it looks like:
+
+```lua
+{
+  id = "optional_identifier" -- only mandatory if you edit a file by hand. IT'S YOUR JOB TO KEEP THESE UNIQUE!
+  name = "My Database",
+  type = "sqlite", -- type of database driver
+  url = "~/path/to/mydb.db",
+}
+```
+
+The connections are loaded to dbee using so-called "sources". They can be added
+to dbee using the `setup()` function:
+
+```lua
+  require("dbee").setup {
+    sources = {
+      require("dbee.sources").MemorySource:new({
+        {
+          name = "...",
+          type = "...",
+          url = "...",
+        },
+        -- ...
+      }),
+      require("dbee.sources").EnvSource:new("DBEE_CONNECTIONS"),
+      require("dbee.sources").FileSource:new(vim.fn.stdpath("cache") .. "/dbee/persistence.json"),
+    },
+    -- ...
+  },
+  -- ... the rest of your config
+  }
+
+```
+
+The above sources are just built-ins. Here is a short description of them:
+
+- `MemorySource` just loads the connections you give it as an argument.
+
+- `EnvSource` loads connection from an environment variable Just export the
+  variable you gave to the loader and you are good to go:
+
+  ```sh
+    export DBEE_CONNECTIONS='[
+        {
+            "name": "DB from env",
+            "url": "mysql://...",
+            "type": "mysql"
+        }
+    ]'
+  ```
+
+- `FileSource` loads connections from a given json file. It also supports
+  editing and adding connections interactively
+
+If the source supports saving and editing you can add connections manually using
+the "add" item in the drawer. Fill in the values and write the buffer (`:w`) to
+save the connection. By default, this will save the connection to the global
+connections file and will persist over restarts (because default `FileSource`
+supports saving)
+
+Another option is to use "edit" item in the tree and just edit the source
+manually.
+
+If you aren't satisfied with the default capabilities, you can implement your
+own source. You just need to fill the following interface and pass it to config
+at setup.
+
+```lua
+---@class Source
+---@field name fun(self: Source):string function to return the name of the source
+---@field load fun(self: Source):connection_details[] function to load connections from external source
+---@field save? fun(self: Source, conns: connection_details[], action: "add"|"delete") function to save connections to external source (optional)
+---@field file? fun(self: Source):string function which returns a source file to edit (optional)
+```
+
+#### Secrets
+
+If you don't want to have secrets laying around your disk in plain text, you can
+use the special placeholders in connection strings (this works using any method
+for specifying connections).
+
+NOTE: *Currently only envirnoment variables are supported*
+
+Example:
+
+Using the `DBEE_CONNECTIONS` environment variable for specifying connections and
+exporting secrets to environment:
+
+```sh
+# Define connections
+export DBEE_CONNECTIONS='[
+    {
+        "name": "{{ env.SECRET_DB_NAME }}",
+        "url": "postgres://{{ env.SECRET_DB_USER }}:{{ env.SECRET_DB_PASS }}@localhost:5432/{{ env.SECRET_DB_NAME }}?sslmode=disable",
+        "type": "postgres"
+    }
+]'
+
+# Export secrets
+export SECRET_DB_NAME="secretdb"
+export SECRET_DB_USER="secretuser"
+export SECRET_DB_PASS="secretpass"
+```
+
+If you start neovim in the same shell, this will evaluate to the following
+connection:
+
+```lua
+{ {
+  name = "secretdb",
+  url = "postgres://secretuser:secretpass@localhost:5432/secretdb?sslmode=disable",
+  type = "postgres",
+} }
+```
 
 ## Projector Integration
 
@@ -153,7 +377,11 @@ code-runner/project-configurator.
 
 To use dbee with it, simply use `"dbee"` as one of it's outputs.
 
+<!-- DOCGEN_IGNORE_START -->
+
 ## Development
 
 Reffer to [ARCHITECTURE.md](ARCHITECTURE.md) for a brief overview of the
 architecture.
+
+<!-- DOCGEN_IGNORE_END -->

dbee/clients/mongo.go

@@ -57,7 +57,7 @@ func NewMongo(url string) (*MongoClient, error) {
 	// get database name from url
 	dbName, err := getDatabaseName(url)
 	if err != nil {
-		return nil, fmt.Errorf("mongo: invalid url: %s -- %v", url, err)
+		return nil, fmt.Errorf("mongo: invalid url: %v", err)
 	}
 
 	opts := options.Client().ApplyURI(url)

dbee/conn/cache.go

@@ -106,7 +106,7 @@ func (c *cache) set(iter models.IterResult) error {
 	})
 	c.active = id
 
-	// process everything else in a seperate goroutine
+	// process everything else in a separate goroutine
 	if !drained {
 		go func() {
 			i := 0