Add support for incoming emails (#22056)

closes #13585 fixes #9067 fixes #2386 ref #6226 ref #6219 fixes #745 This PR adds support to process incoming emails to perform actions. Currently I added handling of replies and unsubscribing from issues/pulls. In contrast to #13585 the IMAP IDLE command is used instead of polling which results (in my opinion 😉) in cleaner code. Procedure: - When sending an issue/pull reply email, a token is generated which is present in the Reply-To and References header. - IMAP IDLE waits until a new email arrives - The token tells which action should be performed A possible signature and/or reply gets stripped from the content. I added a new service to the drone pipeline to test the receiving of incoming mails. If we keep this in, we may test our outgoing emails too in future. Co-authored-by: silverwind <me@silverwind.io> Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
go-gitea · Jan 14, 2023 · fc037b4 · fc037b4
1 parent 20e3ffd
commit fc037b4
Show file tree

Hide file tree

Showing 26 changed files with 1,525 additions and 39 deletions.
diff --git a/.drone.yml b/.drone.yml
@@ -230,6 +230,10 @@ services:
       MINIO_ACCESS_KEY: 123456
       MINIO_SECRET_KEY: 12345678
 
+  - name: smtpimap
+    image: tabascoterrier/docker-imap-devel:latest
+    pull: always
+
 steps:
   - name: fetch-tags
     image: docker:git

diff --git a/assets/go-licenses.json b/assets/go-licenses.json
diff --git a/custom/conf/app.example.ini b/custom/conf/app.example.ini
@@ -1664,6 +1664,47 @@ ROUTER = console
 ;; convert \r\n to \n for Sendmail
 ;SENDMAIL_CONVERT_CRLF = true
 
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;[email.incoming]
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;;
+;; Enable handling of incoming emails.
+;ENABLED = false
+;;
+;; The email address including the %{token} placeholder that will be replaced per user/action.
+;; Example: incoming+%{token}@example.com
+;; The placeholder must appear in the user part of the address (before the @).
+;REPLY_TO_ADDRESS =
+;;
+;; IMAP server host
+;HOST =
+;;
+;; IMAP server port
+;PORT =
+;;
+;; Username of the receiving account
+;USERNAME =
+;;
+;; Password of the receiving account
+;PASSWORD =
+;;
+;; Whether the IMAP server uses TLS.
+;USE_TLS = false
+;;
+;; If set to true, completely ignores server certificate validation errors. This option is unsafe.
+;SKIP_TLS_VERIFY = true
+;;
+;; The mailbox name where incoming mail will end up.
+;MAILBOX = INBOX
+;;
+;; Whether handled messages should be deleted from the mailbox.
+;DELETE_HANDLED_MESSAGE = true
+;;
+;; Maximum size of a message to handle. Bigger messages are ignored. Set to 0 to allow every size.
+;MAXIMUM_MESSAGE_SIZE = 10485760
+
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 ;[cache]

diff --git a/docs/content/doc/advanced/config-cheat-sheet.en-us.md b/docs/content/doc/advanced/config-cheat-sheet.en-us.md
@@ -750,6 +750,20 @@ and
 - `SEND_BUFFER_LEN`: **100**: Buffer length of mailing queue. **DEPRECATED** use `LENGTH` in `[queue.mailer]`
 - `SEND_AS_PLAIN_TEXT`: **false**: Send mails only in plain text, without HTML alternative.
 
+## Incoming Email (`email.incoming`)
+
+- `ENABLED`: **false**: Enable handling of incoming emails.
+- `REPLY_TO_ADDRESS`: **\<empty\>**: The email address including the `%{token}` placeholder that will be replaced per user/action. Example: `incoming+%{token}@example.com`. The placeholder must appear in the user part of the address (before the `@`).
+- `HOST`: **\<empty\>**: IMAP server host.
+- `PORT`: **\<empty\>**: IMAP server port.
+- `USERNAME`: **\<empty\>**: Username of the receiving account.
+- `PASSWORD`: **\<empty\>**: Password of the receiving account.
+- `USE_TLS`: **false**: Whether the IMAP server uses TLS.
+- `SKIP_TLS_VERIFY`: **false**: If set to `true`, completely ignores server certificate validation errors. This option is unsafe.
+- `MAILBOX`: **INBOX**: The mailbox name where incoming mail will end up.
+- `DELETE_HANDLED_MESSAGE`: **true**: Whether handled messages should be deleted from the mailbox.
+- `MAXIMUM_MESSAGE_SIZE`: **10485760**: Maximum size of a message to handle. Bigger messages are ignored. Set to 0 to allow every size.
+
 ## Cache (`cache`)
 
 - `ENABLED`: **true**: Enable the cache.

diff --git a/docs/content/doc/features/comparison.en-us.md b/docs/content/doc/features/comparison.en-us.md
@@ -106,7 +106,7 @@ _Symbols used in table:_
 | Issue search                  | ✓                                                   | ✘    | ✓         | ✓         | ✓         | ✓         | ✘            |
 | Global issue search           | [/](https://github.com/go-gitea/gitea/issues/2434)  | ✘    | ✓         | ✓         | ✓         | ✓         | ✘            |
 | Issue dependency              | ✓                                                   | ✘    | ✘         | ✘         | ✘         | ✘         | ✘            |
-| Create issue via email        | [✘](https://github.com/go-gitea/gitea/issues/6226)  | ✘    | ✘         | ✘         | ✓         | ✓         | ✘            |
+| Create issue via email        | [✘](https://github.com/go-gitea/gitea/issues/6226)  | ✘    | ✘         | ✓         | ✓         | ✓         | ✘            |
 | Service Desk                  | [✘](https://github.com/go-gitea/gitea/issues/6219)  | ✘    | ✘         | ✓         | ✓         | ✘         | ✘            |
 
 ## Pull/Merge requests

diff --git a/docs/content/doc/features/comparison.zh-cn.md b/docs/content/doc/features/comparison.zh-cn.md
@@ -92,7 +92,7 @@ _表格中的符号含义:_
 | 工单搜索            | ✓                                                  | ✘                                             | ✓         | ✓                                                                       | ✓         | ✓              | ✘            |
 | 工单全局搜索        | [✘](https://github.com/go-gitea/gitea/issues/2434) | ✘                                             | ✓         | ✓                                                                       | ✓         | ✓              | ✘            |
 | 工单依赖关系        | ✓                                                  | ✘                                             | ✘         | ✘                                                                       | ✘         | ✘              | ✘            |
-| 通过 Email 创建工单 | [✘](https://github.com/go-gitea/gitea/issues/6226) | [✘](https://github.com/gogs/gogs/issues/2602) | ✘         | ✘                                                                       | ✓         | ✓              | ✘            |
+| 通过 Email 创建工单 | [✘](https://github.com/go-gitea/gitea/issues/6226) | [✘](https://github.com/gogs/gogs/issues/2602) | ✘         | ✓                                                                       | ✓         | ✓              | ✘            |
 | 服务台              | [✘](https://github.com/go-gitea/gitea/issues/6219) | ✘                                             | ✘         | [✓](https://gitlab.com/groups/gitlab-org/-/epics/3103)                  | ✓         | ✘              | ✘            |
 
 #### Pull/Merge requests

diff --git a/docs/content/doc/features/comparison.zh-tw.md b/docs/content/doc/features/comparison.zh-tw.md
@@ -93,7 +93,7 @@ menu:
 | 問題搜尋             | ✓                                                  | ✘                                             | ✓         | ✓                                                                       | ✓         | ✓         | ✘            |
 | 全域問題搜尋         | [✘](https://github.com/go-gitea/gitea/issues/2434) | ✘                                             | ✓         | ✓                                                                       | ✓         | ✓         | ✘            |
 | 問題相依             | ✓                                                  | ✘                                             | ✘         | ✘                                                                       | ✘         | ✘         | ✘            |
-| 從電子郵件建立問題   | [✘](https://github.com/go-gitea/gitea/issues/6226) | [✘](https://github.com/gogs/gogs/issues/2602) | ✘         | ✘                                                                       | ✓         | ✓         | ✘            |
+| 從電子郵件建立問題   | [✘](https://github.com/go-gitea/gitea/issues/6226) | [✘](https://github.com/gogs/gogs/issues/2602) | ✘         | ✓                                                                       | ✓         | ✓         | ✘            |
 | 服務台               | [✘](https://github.com/go-gitea/gitea/issues/6219) | ✘                                             | ✘         | [✓](https://gitlab.com/groups/gitlab-org/-/epics/3103)                  | ✓         | ✘         | ✘            |
 
 ## 拉取/合併請求

diff --git a/docs/content/doc/usage/incoming-email.en-us.md b/docs/content/doc/usage/incoming-email.en-us.md
@@ -0,0 +1,47 @@
+---
+date: "2022-12-01T00:00:00+00:00"
+title: "Incoming Email"
+slug: "incoming-email"
+draft: false
+toc: false
+menu:
+  sidebar:
+    parent: "usage"
+    name: "Incoming Email"
+    weight: 13
+    identifier: "incoming-email"
+---
+
+# Incoming Email
+
+Gitea supports the execution of several actions through incoming mails. This page describes how to set this up.
+
+**Table of Contents**
+
+{{< toc >}}
+
+## Requirements
+
+Handling incoming email messages requires an IMAP-enabled email account.
+The recommended strategy is to use [email sub-addressing](https://en.wikipedia.org/wiki/Email_address#Sub-addressing) but a catch-all mailbox does work too.
+The receiving email address contains a user/action specific token which tells Gitea which action should be performed.
+This token is expected in the `To` and `Delivered-To` header fields.
+
+Gitea tries to detect automatic responses to skip and the email server should be configured to reduce the incoming noise too (spam, newsletter).
+
+## Configuration
+
+To activate the handling of incoming email messages you have to configure the `email.incoming` section in the configuration file.
+
+The `REPLY_TO_ADDRESS` contains the address an email client will respond to.
+This address needs to contain the `%{token}` placeholder which will be replaced with a token describing the user/action.
+This placeholder must only appear once in the address and must be in the user part of the address (before the `@`).
+
+An example using email sub-addressing may look like this: `incoming+%{token}@example.com`
+
+If a catch-all mailbox is used, the placeholder may be used anywhere in the user part of the address: `incoming+%{token}@example.com`, `incoming_%{token}@example.com`, `%{token}@example.com`
+
+## Security
+
+Be careful when choosing the domain used for receiving incoming email.
+It's recommended receiving incoming email on a subdomain, such as `incoming.example.com` to prevent potential security problems with other services running on `example.com`.
diff --git a/go.mod b/go.mod
@@ -20,11 +20,13 @@ require (
 	github.com/buildkite/terminal-to-html/v3 v3.7.0
 	github.com/caddyserver/certmagic v0.17.2
 	github.com/chi-middleware/proxy v1.1.1
-	github.com/denisenkom/go-mssqldb v0.12.3
+	github.com/denisenkom/go-mssqldb v0.12.2
+	github.com/dimiro1/reply v0.0.0-20200315094148-d0136a4c9e21
 	github.com/djherbis/buffer v1.2.0
 	github.com/djherbis/nio/v3 v3.0.1
 	github.com/dustin/go-humanize v1.0.0
 	github.com/editorconfig/editorconfig-core-go/v2 v2.5.1
+	github.com/emersion/go-imap v1.2.1
 	github.com/emirpasic/gods v1.18.1
 	github.com/ethantkoenig/rupture v1.0.1
 	github.com/felixge/fgprof v0.9.3
@@ -58,6 +60,7 @@ require (
 	github.com/hashicorp/golang-lru v0.6.0
 	github.com/huandu/xstrings v1.4.0
 	github.com/jaytaylor/html2text v0.0.0-20211105163654-bc68cce691ba
+	github.com/jhillyerd/enmime v0.10.1
 	github.com/json-iterator/go v1.1.12
 	github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51
 	github.com/keybase/go-crypto v0.0.0-20200123153347-de78d2cb44f4
@@ -145,6 +148,7 @@ require (
 	github.com/blevesearch/zapx/v15 v15.3.8 // indirect
 	github.com/boombuler/barcode v1.0.1 // indirect
 	github.com/bradfitz/gomemcache v0.0.0-20190913173617-a41fca850d0b // indirect
+	github.com/cention-sany/utf7 v0.0.0-20170124080048-26cad61bd60a // indirect
 	github.com/cespare/xxhash/v2 v2.1.2 // indirect
 	github.com/cloudflare/circl v1.2.0 // indirect
 	github.com/couchbase/go-couchbase v0.0.0-20210224140812-5740cd35f448 // indirect
@@ -155,6 +159,7 @@ require (
 	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
 	github.com/dlclark/regexp2 v1.7.0 // indirect
 	github.com/dsnet/compress v0.0.2-0.20210315054119-f66993602bf5 // indirect
+	github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21 // indirect
 	github.com/felixge/httpsnoop v1.0.3 // indirect
 	github.com/fxamacker/cbor/v2 v2.4.0 // indirect
 	github.com/go-ap/errors v0.0.0-20221205040414-01c1adfc98ea // indirect