Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V0.2.0 #38

Merged
merged 84 commits into from
Aug 31, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
ae17d11
feat(bot.run): rename
iyear Aug 4, 2022
f3c1d82
feat(cmd): add usr run
iyear Aug 5, 2022
ab074bb
refactor!(bot): path and default values
iyear Aug 5, 2022
0a573c2
feat(usr): add login
iyear Aug 5, 2022
5d259b8
feat(logger): support settings
iyear Aug 5, 2022
ff3db73
feat(usr): run init
iyear Aug 5, 2022
1e85599
refactor!(proxy): change proxy to url type
iyear Aug 5, 2022
37afebd
refactor(logger): zapcore parse level
iyear Aug 6, 2022
8b14332
feat(usr): add login completely
iyear Aug 6, 2022
6174509
feat(usr): add default values for account config
iyear Aug 6, 2022
563f278
feat(usr): add device identifier
iyear Aug 6, 2022
35fd22b
feat(usr): init
iyear Aug 8, 2022
0ab9ed0
fix(usr): state and accesshasher
iyear Aug 8, 2022
77eb463
fix(usr): remove gaps log
iyear Aug 9, 2022
070ff76
feat(bot): add sender name and optimize search result style
iyear Aug 9, 2022
8f8bf3f
refactor(usr): state manager
iyear Aug 9, 2022
154a89f
refactor(usr): rename scope to bot scope
iyear Aug 10, 2022
1afc7e0
feat(usr): add usr scope
iyear Aug 10, 2022
25592dd
fix(bot): index date
iyear Aug 10, 2022
ccc8c29
fix(bot): edit unix time
iyear Aug 11, 2022
fd77de2
feat(usr): add usr handlers
iyear Aug 11, 2022
b5423b2
feat(search): add chat name field
iyear Aug 11, 2022
c1b09cd
feat(search): chat name render
iyear Aug 12, 2022
86d0716
refactor!(bot): use html style and solve escape problem. close #15
iyear Aug 13, 2022
48a8124
chore(dep): tidy
iyear Aug 13, 2022
60cc994
chore(bot): optimize search result style
iyear Aug 13, 2022
c1da9e0
feat(bot): view full result
iyear Aug 14, 2022
1644e73
refactor(utils): add scopes
iyear Aug 14, 2022
23e58e1
refactor(utils): rename
iyear Aug 14, 2022
a72d4b9
refactor(utils): extract to new scope
iyear Aug 14, 2022
c78d472
feat(usr): clean index message text
iyear Aug 14, 2022
04aa80b
feat(usr): don't index sticker
iyear Aug 14, 2022
faabe6f
refactor(index): message text
iyear Aug 14, 2022
cb66747
feat!(prj): add view message and modify search msg indexing model
iyear Aug 15, 2022
121462a
feat(bot): remove useless dot and optimize indexing
iyear Aug 15, 2022
981a3c9
refactor(storage): extract Init func
iyear Aug 17, 2022
5aac62d
refactor(usr): add channel too long handler
iyear Aug 17, 2022
42d35ec
feat(usr): add search handler
iyear Aug 18, 2022
ba833e2
fix(usr): nil peer pointer. close #16
iyear Aug 18, 2022
b05b795
chore(gomod): upgrade deps
iyear Aug 19, 2022
5afb1bd
feat(search): add chat type field. close #24
iyear Aug 20, 2022
714bf1d
feat(storage): add context support. close #20
iyear Aug 20, 2022
bdeb2de
feat(source): add chat type field
iyear Aug 20, 2022
f3b0c4a
feat(view): add chat type field
iyear Aug 20, 2022
51543ab
refactor(bot): search
iyear Aug 20, 2022
4a56c82
refactor(usr): search
iyear Aug 21, 2022
860f71d
fix(usr): sometimes don't receive messages
iyear Aug 21, 2022
f2efd90
chore(usr): add gaps params comments
iyear Aug 21, 2022
fb2f828
refactor(i18n): extract hooks
iyear Aug 21, 2022
6146593
fix(bot): nil chat. close #22
iyear Aug 21, 2022
6ac0410
refactor(config): extract conf pkg
iyear Aug 21, 2022
92f9996
refactor(usr): cmd
iyear Aug 21, 2022
a58d10d
refactor(usr): separate config
iyear Aug 21, 2022
1bf3f2f
refactor(usr): separate login
iyear Aug 21, 2022
4e460b2
feat(usr): add updates and client logger
iyear Aug 24, 2022
e77cdee
refactor(usr): extract index pkg to gen msg data
iyear Aug 25, 2022
fb2a94d
feat(usr): add source
iyear Aug 26, 2022
f51e787
feat(usr): enhance source. close #19
iyear Aug 27, 2022
39d8e87
refactor(util): telegram utils
iyear Aug 27, 2022
2da9d89
fix(utils): missing return
iyear Aug 27, 2022
c094761
chore(dep): add go-pretty
iyear Aug 27, 2022
27e876e
refactor(usr): use rate limit middleware for tg client. close #33
iyear Aug 27, 2022
c8c33ff
fix(usr.run): remove ratelimit
iyear Aug 27, 2022
ea097f6
refactor(prj): bot middleware. close #30
iyear Aug 27, 2022
cf9b374
fix(usr.source): can not get user's name
iyear Aug 28, 2022
12db10f
feat(usr): only account owner can use search bot
iyear Aug 28, 2022
ee5d7aa
refactor(bot): cmd & config. close #35
iyear Aug 28, 2022
e67ff86
refactor(cmd): pass errors to root cmd
iyear Aug 28, 2022
a940fd2
chore(cmd): improve desc of commands
iyear Aug 28, 2022
f3fed26
chore(version): update
iyear Aug 28, 2022
a1261ff
feat(cmd): add cmd docs and examples
iyear Aug 28, 2022
86a8c05
wip(searchbot): private go link
iyear Aug 29, 2022
7b3e22e
feat(pkg): add hashids
iyear Aug 29, 2022
4a88f51
feat(searchbot): private go link
iyear Aug 29, 2022
d4701b2
chore(docs.cmd): remove gen tag
iyear Aug 29, 2022
3893911
chore(i18n): improve
iyear Aug 29, 2022
bff52a2
fix(usr.source): peer entities(tmp solution)
iyear Aug 29, 2022
a608928
refactor(usr.config): rename
iyear Aug 29, 2022
1a73da3
feat(usr): config examples
iyear Aug 30, 2022
807875e
fix(cmd): root error judge
iyear Aug 30, 2022
23ffb81
fix(kv): set timeout for bolt db
iyear Aug 30, 2022
e835b2e
docs(all): update. WIP
iyear Aug 30, 2022
5c46b9c
ci(goreleaser): add new files
iyear Aug 30, 2022
a6b864e
docs(all): update
iyear Aug 30, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
.idea
data
log
config/config.yaml
.searchx
9 changes: 4 additions & 5 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,11 @@ archives:
- goos: windows
format: zip
files:
- README.md
- README*.md
- LICENSE
- config/i18n/*
- config/dict.txt
- config/config.full.yaml
- config/config.min.yaml
- config/**/*
- data/**/*
- docs/**/*
changelog:
use: github
sort: asc
Expand Down
99 changes: 18 additions & 81 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
## Introduction

![](https://img.shields.io/github/go-mod/go-version/iyear/searchx?style=flat-square)
![](https://img.shields.io/github/license/iyear/searchx?style=flat-square)
![](https://img.shields.io/github/v/release/iyear/searchx?color=red&style=flat-square)
![](https://img.shields.io/github/last-commit/iyear/searchx?style=flat-square)

English | [简体中文](README_zh.md)
English | [简体中文](README.zh.md)

🔍 Enhance Telegram Group/Channel Search In 5 Minutes 🚀
🔍 Enhance Telegram Search In 5 Minutes 🚀

[DEMO](https://t.me/e5subs_bot) (Join [E5SubBot Group](https://t.me/e5subbot) in advance)

Expand All @@ -20,103 +21,39 @@ English | [简体中文](README_zh.md)
- Customizable i18n message templates
- ......

## Deployment

1. **Prepare `Telegram Bot`**
1. Get `Bot Token`: [How to create a Bot](https://core.telegram.org/bots#6-botfather)
2. **Disable** `Group Privacy`: send `/setprivacy` to [@BotFather](https://t.me/BotFather)
2. **Get `Telegram User ID`: [@userinfobot](https://t.me/userinfobot)**
3. **Prepare `searchx`**
1. Download `searchx`: go to [GitHub Releases](https://github.com/iyear/searchx/releases) and uncompress it.
2. Change `YOUR_BOT_TOKEN` `YOUR_ADMIN_ID` in `config/config.min.yaml` to `Bot Token` `Telegram User ID`
4. **Start: `./searchx run -c config/config.min.yaml`**
5. **Invite `Bot` to groups/channels**

## Customization
For the vast majority of users, the default minimal configuration is sufficient.

If you have more customization needs, please refer to [config.full.yaml](config/config.full.yaml) and its comments.

## Command
### version
View version information

```shell
./searchx version
```

```
Version: 0.0.0
Commit: 5cae8dc
Date: 2022-07-30T07:58:05Z

go1.17.3 windows/amd64
```

### run
Start Bot

- `-c`: path to config file
## Usage

```shell
# start with the minimal config
./searchx run -c config/config.min.yaml
```
In versions after `v0.2.0`, `searchx` combines `bot` and `usr`.

### source
Bot only indexes messages during the join period, use this command if you want to index history messages.
`bot` is used by channel owners/group owners to provide members with an enhanced search function.

Support groups/channels, support very large `JSON` file.
`usr` is used by individual users to provide enhanced search functionality for themselves.

Use **official client** to export history messages. Export options: **Uncheck all, format as `JSON`**.
[bot document](docs/bot/README.md) | [usr document](docs/usr/README.md)

- `-f`: exported history messages in `JSON` file
- `-d`: search engine driver
- `-o`: search engine options
## Commands

```shell
# Using bleve to import messages
./searchx source -f YOUR_PATH/result.json -d bleve -o path="index" -o dict="config/dict.txt"
```

### query
Command line query

- `-d`: search engine driver
- `-o`: search engine options
- `-q`: keyword
- `--pn`: page number, start from 0, default is 0
- `--ps`: number of result per page, default is 10
- `--json`: output as `JSON` format

```shell
# Using bleve to query
./searchx query -d bleve -o path="index" -o dict="config/dict.txt" -q KEYWORD --pn 0 --ps 10 --json
```
Go to [command documentation](docs/command/searchx.md) for full command docs.

## FAQ

**Q: Why don't I use the search that comes with Telegram?**

A: As we all know, the search function that comes with Telegram is not very useful. The purpose of this project is to solve these problems.
A: As we all know, the search function that comes with Telegram is not very useful. The purpose of this project is to
solve these problems.

**Q: I'm having problems using it?**

A: If you still can't solve the problem after searching, give feedback by [SUBMIT ISSUE](https://github.com/iyear/searchx/issues/new).
A: If you still can't solve the problem after searching, give feedback
by [SUBMIT ISSUE](https://github.com/iyear/searchx/issues/new).

When submit an `ISSUE`, we recommend describing the problem in English and providing relevant screenshots and steps to reproduce it.
When submit an `ISSUE`, we recommend describing the problem in English and providing relevant screenshots and steps to
reproduce it.

**Q: I want to add a feature?**

A: Same as above

**Q: Why do I need to disable `Group Privacy`? Does it cause security issues?**

A: When `Group Privacy` is enabled, it will cause `Bot` to not receive all messages in the group, resulting in missing indexes. Please refer to: https://core.telegram.org/bots#privacy-mode
At the same time, it does not create security issues. The project is self-deployed and the data is stored locally, so it will not lead to data leakage.

**Q: Can I use it for my personal account? Will it be supported in the future?**

A: Not currently, it may be supported in the future, but the use of `userbot` has the risk of blocking. The current target group of this project is group/channel owners, self-built to provide convenience to members.

## LICENSE

Apache License 2.0
55 changes: 55 additions & 0 deletions README.zh.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
## 简介

![](https://img.shields.io/github/go-mod/go-version/iyear/searchx?style=flat-square)
![](https://img.shields.io/github/license/iyear/searchx?style=flat-square)
![](https://img.shields.io/github/v/release/iyear/searchx?color=red&style=flat-square)
![](https://img.shields.io/github/last-commit/iyear/searchx?style=flat-square)

[English](README.md) | 简体中文

🔎 五分钟强化 Telegram 搜索 🚀

[DEMO](https://t.me/e5subs_bot) (需先加入[E5SubBot交流群](https://t.me/e5subbot))

## 特性

- 最小化配置 & 单文件一键启动,轻松运行在你的路由器上
- 组件化扩展设计,可选多种存储后端
- 跨平台支持 macOS/Windows/Linux
- 特别为中文优化的搜索引擎
- 大文件历史消息导入
- 自定义国际化消息模板
- ……

## 使用

在 `v0.2.0` 之后的版本,`searchx` 将 `bot` `usr` 集于一身。

`bot` 的目标群体为频道主/群主,为成员提供该会话的强化搜索功能;

`usr` 的目标群体为个人用户,为自己提供强化搜索功能。

[bot 文档](docs/bot/README.zh.md) | [usr 文档](docs/usr/README.zh.md)

## 命令

前往 [命令文档](docs/command/searchx.md) 查看完整命令文档

## FAQ

**Q: 为什么我不使用 Telegram 自带的搜索?**

A: 众所周知,Telegram 自带的搜索功能并不好用,尤其是对中文的支持很差。本项目的目的就是解决这些搜索痛点。

**Q: 我在使用过程中遇到了问题?**
A: 在确认搜索后依旧无法解决,通过 [发起 ISSUE](https://github.com/iyear/searchx/issues/new) 的方式反馈。

在发起 `ISSUE` 的过程中,我们提倡使用英文描述问题,并在 `ISSUE` 中提供相关的截图和复现步骤。

**Q: 我想要增加一个功能?**

A: 同上

## LICENSE

Apache License 2.0
119 changes: 0 additions & 119 deletions README_zh.md

This file was deleted.

31 changes: 31 additions & 0 deletions app/bot/internal/config/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package config

import (
"github.com/iyear/searchx/pkg/configs"
"github.com/iyear/searchx/pkg/logger"
"github.com/iyear/searchx/pkg/storage"
)

var C config

func Init(path string) error {
return configs.Init(path, &C)
}

type config struct {
Bot struct {
Token string `mapstructure:"token" validate:"required"`
Admin []int64 `mapstructure:"admin" validate:"required"`
} `mapstructure:"bot"`
Proxy string `mapstructure:"proxy" validate:"omitempty,url"`
Storage storage.Config `mapstructure:"storage"`
Log logger.Config `mapstructure:"log"`
Ctrl struct {
Notice string `mapstructure:"notice" default:"NO NOTICE"`
I18N string `mapstructure:"i18n" validate:"dir" default:"config/bot/i18n"`
DefaultLanguage string `mapstructure:"default_language" validate:"iso6391" default:"zh-cn"`
Search struct {
PageSize int `mapstructure:"page_size" validate:"gte=1,lte=20" default:"10"`
} `mapstructure:"search"`
} `mapstructure:"ctrl"`
}
Loading