Merge pull request #10 from ChaosNyaruko/todo

Feature: support all registered MDX sources result display

README.md

@@ -1,7 +1,14 @@
 # Introduction
 Yet another simple dictionary application. Support multiple sources, including Longman online dictionary, and user-loaded MDX/MDD dictionary files.
 
-![Gif](./assets/ondict_example1.gif)
+# Disclaimer
+It is trying to be just a **dictionary**, which you may need during your English study or writing some English posts. 
+
+It is _NOT_ a "translator", in which scenario an LLM model based tool is more suitable in modern days.
+
+Web mode is recommended, because both the online engine and MDX engine are based on HTML/CSS stuff. So when you need its output to be rendered as markdown, a independent parser and renderer need to written for each source, that's quite a lot of work and almost impossible.
+
+I just write a simple markdown renderer for [Longman Dictionary of Contemporary English](https://github.com/ChaosNyaruko/ondict/releases/download/v0.0.5/Longman.Dictionary.of.Contemporary.English.mdx) MDX dictionary, which I uploaded in some releases, so that you can roughly use its markdown rendered output in some cases, such working as a TUI editor(which has no "web core") plugin, or just using it in a terminal.
 
 # Other choices
 - [Golden](http://www.goldendict.org/)
@@ -35,20 +42,33 @@ ondict -h
 
 ### Examples
 #### One-shot query
+A one-shot query, it will take some time when you call it the first time, it needs some loading work.
+It will launch an local server using unix domain socket.
+
+##### online engine (you don't have to specify the -e option):
 ```console
-ondict -q <word>
+ondict -q <word> [-e anything]
 ```
+![Gif](./assets/e1_online.gif)
+##### mdx engine (ldoce5):
+```console
+ondict -q <word> -e mdx
+```
+![Gif](./assets/e1_mdx.gif)
+
 
 #### One-shot query, but from remote server
 ```console
-ondict -q <word> -remote auto 
+ondict -q <word> -remote localhost:1345
 ```
+![Gif](./assets/e1_mdx_remote.gif)
 
 #### A "repl" querier
 ```console
-ondict -i
+ondict -i -e mdx
 ```
 input `.help` for commands that can be used.
+![Gif](./assets/e1_mdx_interactive.gif)
 
 #### Work as a server
 This app can also serve as a HTTP server, allowing remote fetch and query, with cache and acceleration.
@@ -59,16 +79,24 @@ Launch a http request
 ```console
 curl "http://localhost:1345/?query=apple&engine=mdx&format=x"
 ```
+![Gif](./assets/e1_mdx_web.gif)
 If you are visiting the URL with a web browser, setting format to "html" is recommended. The browser will automatically render a more beautiful page than it is in the "CLI" interface.
 
 You can also deploy it on your server, as an upstream of Nginx/, or just exposing it with a suitable ip/port.
 
 You can run `make serve` locally for an easy example. My front-end skill is poor, so the page is ugly and rough, don't hate it :(. 
 
-There are still a lot of [TODOs](./todo.md), free free to give me PRs and contribute to the immature project, thanks in advance.
+There are still a lot of [TODOs](./todo.md), feel free to give me PRs and contribute to the immature project, thanks in advance.
 
+#### Work with Neovim
+See [Integrated with Neovim](#neovim)
+![Gif](./assets/e1_mdx_nvim.gif)
 
-## Working with Neovim
+#### For MacOS, work with [hammerspoon](https://www.hammerspoon.org)
+![Gif](./assets/e1_mdx_hammerspoon.gif)
+
+
+## <a name="neovim"> </a>Integrated with Neovim
 1. Install the plugin with a plugin manager or manually. 
 2. Use `:lua require("ondict").query()` to query \<cword\>.
 3. Define a mapping for yourself to call it easier. NOTE: in visual mode, use "\<cmd\>lua require("ondict").query()\<cr\>" instead. It will capture the "SELECTED" word. Otherwise, the "mode" will be changed and only "\<cword\>" can be queried.
@@ -101,13 +129,35 @@ vnoremap <leader>d <cmd>lua require("ondict").query()<cr>
 vim.keymap.set("n", "<leader>d", require("ondict").query)
 vim.keymap.set("v", "<leader>d", require("ondict").query)
 ```
-# Offline dictionary files
-Put the decoded JSON files in $HOME/.config/ondict/dicts
+# <a name="offline"></a>Offline dictionary files
+Put dictionary files in $HOME/.config/ondict/dicts, support formats are:
+- "key-value" organized pairs JSON files.
+- MDX files, refer to [mdict](https://mdict.org) or [pdawiki](https://pdawiki.com/forum/).
 
+## Configuration
+
+### file tree
+```
+// cd ~/.config/ondict
+.
+├── config.json
+└── dicts
+    └── Longman\ Dictionary\ of\ Contemporary\ English.mdx
+```
+### config.json
+```json
+{
+    "dicts": [
+        "Longman Dictionary of Contemporary English",
+        "xxx",
+        "yyy"
+    ]
+}
+```
 # Features
-- Online query support based on [Longman online dictionary](https://ldoceonline.com)
+- **Online** query support based on [Longman online dictionary](https://ldoceonline.com)
 - Integrated with (n)vim, feel free to use it in whatever editor you are using!
-- Offline engine/mode is supported. The online engine may be more comprehensive and updated, but they are slow since an HTTP request is made for the first time.
+- In the offline mode, MDX engine is supported. The online engine may be more comprehensive and updated, but they are slow since an HTTP request is made for the first time. The offline mode, however, can work without internet connection, but pre-loaded [dictionary files](#offline) are needed.
 
 # LICENSE
 [LICENSE](./LICENSE)

go.mod

@@ -7,7 +7,6 @@ require (
 	github.com/C0MM4ND/go-ripemd v0.0.0-20200326052756-bd1759ad7d10
 	github.com/fatih/color v1.15.0
 	github.com/stretchr/testify v1.8.4
-	golang.org/x/crypto v0.16.0
 	golang.org/x/net v0.15.0
 )
 

go.sum

@@ -15,8 +15,6 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
 github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
-golang.org/x/crypto v0.16.0 h1:mMMrFzRSCF0GvB7Ne27XVtVAaXLrPmgPC7/v0tkwHaY=
-golang.org/x/crypto v0.16.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
 golang.org/x/net v0.15.0 h1:ugBLEUaxABaB5AJqW9enI0ACdci2RUd4eP51NTBvuJ8=
 golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=

main.go

@@ -32,13 +32,13 @@ var interactive = flag.Bool("i", false, "Launch an interactive CLI app")
 var server = flag.Bool("serve", false, "Serve as a HTTP server, default on UDS, for cache stuff, make it quicker!")
 var idleTimeout = flag.Duration("listen.timeout", defaultIdleTimeout, "Used with '-serve', the server will automatically shut down after this duration if no new requests come in")
 var listenAddr = flag.String("listen", "", "Used with '-serve', address on which to listen for remote connections. If prefixed by 'unix;', the subsequent address is assumed to be a unix domain socket. Otherwise, TCP is used.")
-var remote = flag.String("remote", "", "Connect to a remote address to get information, 'auto' means it will try to launch a request by UDS. If no local server is working, a new server will be created, with -listen.timeout 1 min.")
+var remote = flag.String("remote", "auto", "Connect to a remote address to get information, 'auto' means it will try to launch a request by UDS. If no local server is working, a new server will be created, with -listen.timeout 1 min.")
 var colour = flag.Bool("color", false, "This flags controls whether to use colors.")
 var renderFormat = flag.String("f", "", "render format, 'md' (for markdown, only for mdx engine now), or 'html'")
 var engine = flag.String("e", "", "query engine, 'mdx' or others(online query)")
 
 // TODO: prev work, for better source abstractions
-var g sources.Source = &sources.GlobalDict
+var g = sources.G
 
 func main() {
 	flag.Parse()
@@ -68,7 +68,7 @@ func main() {
 	}
 
 	if *interactive {
-		g.Register() // TODO(ch): lazy loading for performance?
+		g.Load()
 		startLoop()
 		return
 	}
@@ -93,7 +93,7 @@ func main() {
 			}
 		}
 		log.Printf("start a new server: %s/%s/%s/%s", network, addr, *renderFormat, *engine)
-		g.Register()
+		g.Load()
 		l, err := net.Listen(network, addr)
 		if err != nil {
 			log.Fatal("bad Listen: ", err)
@@ -195,7 +195,7 @@ func main() {
 
 	if *engine == "mdx" {
 		// io.Copy(os.Stdout, fd)
-		g.Register()
+		g.Load()
 		fmt.Println(sources.QueryMDX(*word, *renderFormat))
 		return
 	}

render/mdx_test.go

@@ -72,15 +72,15 @@ func Test_MultiMatch(t *testing.T) {
 }
 
 func Test_play(t *testing.T) {
-	var g sources.MdxDict
+	var g *sources.MdxDict
 	if os.Getenv("FULLTEST") == "1" {
 		sources.LoadConfig()
-		g = sources.GlobalDict
+		g = (*sources.G)[0]
 	} else {
 		d := sources.MdxDict{
 			MdxFile: "../testdata/test_dict",
 		}
-		g = d
+		g = &d
 	}
 	g.Register()
 	dict := g.MdxDict

server.go

@@ -7,7 +7,6 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ChaosNyaruko/ondict/sources"
 	"github.com/ChaosNyaruko/ondict/util"
 )
 
@@ -49,9 +48,6 @@ func (s *proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		res := query(word, e, f)
 		w.Header().Set("Content-Type", "text/html; charset=utf-8")
 		w.Write([]byte(res))
-		if f == "html" {
-			w.Write([]byte("<style>" + sources.GlobalDict.CSS() + "</style>"))
-		}
 		// w.Write([]byte("<style>" + odecss + "</style>"))
 		// w.Write([]byte(fmt.Sprintf(`<link ref="stylesheet" type="text/css", href=/d/static/oald9.css />`)))
 		return

sources/aho_corasick_test.go

@@ -6,15 +6,15 @@ import (
 )
 
 func Test_New(t *testing.T) {
-	var g MdxDict
+	var g *MdxDict
 	if os.Getenv("FULLTEST") == "1" {
 		LoadConfig()
-		g = GlobalDict
+		g = (*G)[0]
 	} else {
 		d := MdxDict{
 			MdxFile: "../testdata/test_dict",
 		}
-		g = d
+		g = &d
 	}
 	g.Register()
 	ack := New(g.MdxDict)

sources/config.go

@@ -28,8 +28,13 @@ func LoadConfig() error {
 	if len(c.Dicts) == 0 {
 		return nil
 	}
-	GlobalDict.MdxFile = filepath.Join(util.DictsPath(), c.Dicts[0])
-	GlobalDict.MdxCss = filepath.Join(util.DictsPath(), c.Dicts[0]+".css")
-	log.Printf("get global dicts: %v", GlobalDict)
+	for _, name := range c.Dicts {
+		dict := &MdxDict{}
+		dict.MdxFile = filepath.Join(util.DictsPath(), name)
+		dict.MdxCss = filepath.Join(util.DictsPath(), name+".css")
+		log.Printf("get global dict: %v", dict.MdxFile)
+		*G = append(*G, dict)
+	}
+	log.Printf("get global dicts: %v", G)
 	return nil
 }

sources/mdx.go

@@ -3,6 +3,7 @@ package sources
 import (
 	"encoding/json"
 	"errors"
+	"fmt"
 	"log"
 	"os"
 	"strings"
@@ -14,28 +15,55 @@ import (
 var Gbold = "**"
 var Gitalic = "*"
 
-var GlobalDict MdxDict
+type Dicts []*MdxDict
+
+var G = &Dicts{}
+
+func (g *Dicts) Load() error {
+	for _, d := range *g {
+		d.Register()
+	}
+	log.Printf("loading g")
+	return nil
+}
 
 func QueryMDX(word string, f string) string {
-	defs := GlobalDict.Get(word)
-	log.Printf("def of %v: %q", defs, word)
+	type mdxResult struct {
+		defs []string
+		css  string
+	}
+	var defs []mdxResult
+	for _, dict := range *G {
+		defs = append(defs, mdxResult{dict.Get(word), dict.CSS()})
+		log.Printf("def of %q, %v: %q", dict.MdxFile, defs, word)
+	}
 	// TODO: put the render abstraction here?
 	if f == "html" { // f for format
 		var res []string
-		for _, def := range defs {
-			h := render.HTMLRender{Raw: def}
-			// m1 := regexp.MustCompile(`<img src="(.*?)\.png" style`)
-			// replaceImg := m1.ReplaceAllString(def, `<img src="`+"data/"+`${1}.png" style`)
-			// log.Printf("try to replace %v", replaceImg)
-			res = append(res, h.Render())
+		for _, dict := range defs {
+			for _, def := range dict.defs {
+				h := render.HTMLRender{Raw: def}
+				// m1 := regexp.MustCompile(`<img src="(.*?)\.png" style`)
+				// replaceImg := m1.ReplaceAllString(def, `<img src="`+"data/"+`${1}.png" style`)
+				// log.Printf("try to replace %v", replaceImg)
+				// TODO: it might be overriden
+				rs := fmt.Sprintf("<div>%s<style>%s</style></div> ", h.Render(), dict.css)
+				res = append(res, rs)
+			}
 		}
 		return strings.Join(res, "<br><br>")
 	}
 
 	var res string
-	for _, def := range defs {
-		fd := strings.NewReader(def) // TODO: find a "close" one when missing?
-		res += "\n---\n" + render.ParseMDX(fd, f)
+	for i, dict := range defs {
+		// TODO: different markdown parser here
+		for _, def := range dict.defs {
+			if i > 0 {
+				break
+			}
+			fd := strings.NewReader(def) // TODO: find a "close" one when missing?
+			res += "\n---\n" + render.ParseMDX(fd, f)
+		}
 	}
 	return res
 }
@@ -48,6 +76,13 @@ func loadDecodedMdx(filePath string) Dict {
 		log.Printf("JSON file not exist: %v", filePath+".json")
 		m := &decoder.MDict{}
 		err := m.Decode(filePath + ".mdx")
+		go func() {
+			if err := m.Decode(filePath + ".mdd"); err != nil {
+				log.Printf("[WARN] parse %v.mdd err: %v", filePath, err)
+			} else {
+				log.Printf("[INFO] successfully decode %v.mdd", filePath)
+			}
+		}()
 		if err != nil {
 			log.Fatalf("Failed to load mdx file[%v], err: %v", filePath, err)
 		}

todo.md

@@ -23,9 +23,10 @@
 - [x] Appropriate module abstraction, and unit testing automation. (sources/decoder/renderer)
 - [x] Lazy-loading for launch performance and memory usage?
 - [x] Parse the MDD file, and serve the picture sources when working in HTML mode.
-- [ ] A independent MDD parser command.
+- [x] Support multiple mdx libs at the same time and provide a user interface (no UI yet)
+- [ ] Auto history.
+- [ ] A independent parser cmd.
 - [ ] See the TODOs in the code.
-- [ ] Support multiple mdx libs at the same time and provide a user interface
 
 ---
 The following are less important things that I want to finish.