Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish to telegram #15

Merged
merged 25 commits into from
Dec 6, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
d938e0f
Add telegram client, for send notification
sgaynetdinov Nov 16, 2019
908cda8
Send message telegram if get new item RSS
sgaynetdinov Nov 16, 2019
3ae8267
Add new package in vendor
sgaynetdinov Nov 16, 2019
d8fb331
Not raise error if telegram token empty
sgaynetdinov Nov 18, 2019
c741b89
Add 'TestSendIfChannelIDEmpty'
sgaynetdinov Nov 18, 2019
ee439da
Fix: 'NewTelegramClient' return 'TelegramClient' not 'nil'
sgaynetdinov Nov 18, 2019
635a674
Add in environment 'TELEGRAM_TOKEN'
sgaynetdinov Nov 18, 2019
80ca198
Refactoring test
sgaynetdinov Nov 18, 2019
be4eeb7
Add test for 'TelegramClient.tagLinkOnlySupport'
sgaynetdinov Nov 18, 2019
117ef40
Add test for 'TelegramClient.getMessageHTML'
sgaynetdinov Nov 18, 2019
bec8544
Rename 'html_expected' -> 'htmlExpected'
sgaynetdinov Nov 18, 2019
010a331
Pretty log, if error send telegram message
sgaynetdinov Nov 19, 2019
d3d0adb
dd prefix "@" for "channel" if not found
sgaynetdinov Nov 19, 2019
b8312f0
Add interface 'Notification'
sgaynetdinov Nov 19, 2019
69f08f9
Set timeout for telegram
sgaynetdinov Nov 20, 2019
2b98130
Revert "Set timeout for telegram"
sgaynetdinov Nov 26, 2019
336fa15
Use timeout from http.Client without context timeout
sgaynetdinov Nov 26, 2019
c5a94a5
Pretty log
sgaynetdinov Dec 4, 2019
c2f5de1
Send audio if file size less 50 Mb or send only text
sgaynetdinov Dec 4, 2019
727df45
Add const 'maxTelegramFileSize'
sgaynetdinov Dec 5, 2019
395831a
Refactoring, message error
sgaynetdinov Dec 5, 2019
ec63f0e
Get filename audio file from url
sgaynetdinov Dec 5, 2019
588a63f
Fix: []byte is a slice backed by pointer already
sgaynetdinov Dec 5, 2019
e27a045
Telegram timeout moved to opts
sgaynetdinov Dec 5, 2019
7193518
Not reads file content to memory
sgaynetdinov Dec 5, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions app/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"fmt"
"io/ioutil"
"os"
"time"

log "github.com/go-pkgz/lgr"
"github.com/jessevdk/go-flags"
Expand All @@ -14,9 +15,11 @@ import (
)

var opts struct {
DB string `short:"c" long:"db" env:"FM_DB" default:"var/feed-master.bdb" description:"bolt db file"`
Conf string `short:"f" long:"conf" env:"FM_CONF" default:"feed-master.yml" description:"config file (yml)"`
Dbg bool `long:"dbg" env:"DEBUG" description:"debug mode"`
DB string `short:"c" long:"db" env:"FM_DB" default:"var/feed-master.bdb" description:"bolt db file"`
Conf string `short:"f" long:"conf" env:"FM_CONF" default:"feed-master.yml" description:"config file (yml)"`
Dbg bool `long:"dbg" env:"DEBUG" description:"debug mode"`
TG string `long:"telegram_token" env:"TELEGRAM_TOKEN" description:"Telegram token"`
TelegramTimeout int64 `long:"telegram_timeout" env:"TELEGRAM_TIMEOUT" description:"Telegram timeout"`
}

var revision = "local"
Expand All @@ -36,8 +39,12 @@ func main() {
if err != nil {
log.Fatalf("[ERROR] can't open db %s, %v", opts.DB, err)
}
tg, err := proc.NewTelegramClient(opts.TG, time.Duration(opts.TelegramTimeout))
if err != nil {
log.Fatalf("[ERROR] failed initilization telegram client %s, %v", opts.TG, err)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when starting without TELEGRAM_TOKEN there will be an error,

  • is this the right behavior?
  • need to register this env in docker-compose?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not right, telegram support should be optional

Copy link
Owner

@umputun umputun Nov 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for "need to register this env in docker-compose?" - I think you are asking if it should be passed via docker-compose? Yes, if a user needs to set (or pass) TELEGRAM_TOKEN it has to be defined/declared in the compose

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


p := &proc.Processor{Conf: conf, Store: db}
p := &proc.Processor{Conf: conf, Store: db, Notification: tg}
go p.Do()

server := api.Server{
Expand Down
40 changes: 28 additions & 12 deletions app/proc/processor.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,28 @@ import (
"github.com/umputun/feed-master/app/feed"
)

// Notification is interface implemented send message
type Notification interface {
Send(string, feed.Item) error
}

// Processor is a feed reader and store writer
type Processor struct {
Conf *Conf
Store *BoltDB
Conf *Conf
Store *BoltDB
Notification Notification
}

// Conf for feeds config yml
type Conf struct {
Feeds map[string]struct {
Title string `yaml:"title"`
Description string `yaml:"description"`
Link string `yaml:"link"`
Image string `yaml:"image"`
Language string `yaml:"language"`
Sources []struct {
Title string `yaml:"title"`
Description string `yaml:"description"`
Link string `yaml:"link"`
Image string `yaml:"image"`
Language string `yaml:"language"`
TelegramChannel string `yaml:"telegram_channel"`
Sources []struct {
Name string `yaml:"name"`
URL string `yaml:"url"`
} `yaml:"sources"`
Expand All @@ -51,9 +58,9 @@ func (p *Processor) Do() {
swg := syncs.NewSizedGroup(p.Conf.System.Concurrent, syncs.Preemptive)
for name, fm := range p.Conf.Feeds {
for _, src := range fm.Sources {
name, src := name, src
name, src, fm := name, src, fm
swg.Go(func(_ context.Context) {
p.feed(name, src.URL, p.Conf.System.MaxItems)
p.feed(name, src.URL, fm.TelegramChannel, p.Conf.System.MaxItems)
})
}
// keep up to MaxKeepInDB items in bucket
Expand All @@ -71,7 +78,7 @@ func (p *Processor) Do() {
}
}

func (p *Processor) feed(name, url string, max int) {
func (p *Processor) feed(name, url, telegramChannel string, max int) {

rss, err := feed.Parse(url)
if err != nil {
Expand All @@ -91,9 +98,18 @@ func (p *Processor) feed(name, url string, max int) {
continue
}

if err := p.Store.Save(name, item); err != nil {
created, err := p.Store.Save(name, item)
if err != nil {
log.Printf("[WARN] failed to save %s (%s) to %s, %v", item.GUID, item.PubDate, name, err)
}

if !created {
return
}

if err := p.Notification.Send(telegramChannel, item); err != nil {
log.Printf("[WARN] failed send telegram message, url=%s to channel=%s, %v", item.Enclosure.URL, telegramChannel, err)
}
}
}

Expand Down
18 changes: 14 additions & 4 deletions app/proc/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@ func NewBoltDB(dbFile string) (*BoltDB, error) {
}

// Save to bolt, skip if found
func (b BoltDB) Save(fmFeed string, item feed.Item) error {
func (b BoltDB) Save(fmFeed string, item feed.Item) (bool, error) {
var created bool

key, err := func() ([]byte, error) {
ts, err := time.Parse(time.RFC1123Z, item.PubDate)
Expand All @@ -50,10 +51,10 @@ func (b BoltDB) Save(fmFeed string, item feed.Item) error {
}()

if err != nil {
return err
return created, err
}

return b.DB.Update(func(tx *bolt.Tx) error {
err = b.DB.Update(func(tx *bolt.Tx) error {
bucket, e := tx.CreateBucketIfNotExists([]byte(fmFeed))
if e != nil {
return e
Expand All @@ -66,9 +67,18 @@ func (b BoltDB) Save(fmFeed string, item feed.Item) error {
if jerr != nil {
return jerr
}

log.Printf("[INFO] save %s - %s - %s - %s", string(key), fmFeed, item.Title, item.GUID)
return bucket.Put(key, jdata)
e = bucket.Put(key, jdata)
if e != nil {
return e
}

created = true
return e
})

return created, err
}

// Load from bold for given feed, up to max
Expand Down
186 changes: 186 additions & 0 deletions app/proc/telegram.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
package proc

import (
"fmt"
"io"
"net/http"
"path"
"strings"
"time"

log "github.com/go-pkgz/lgr"
"github.com/microcosm-cc/bluemonday"
tb "gopkg.in/tucnak/telebot.v2"

"github.com/umputun/feed-master/app/feed"
)

const (
maxTelegramFileSize = 50_000_000
)

// TelegramClient client
type TelegramClient struct {
Bot *tb.Bot
Timeout time.Duration
}

// NewTelegramClient init telegram client
func NewTelegramClient(token string, timeout time.Duration) (*TelegramClient, error) {
if timeout == 0 {
timeout = time.Duration(60 * 10)
sgaynetdinov marked this conversation as resolved.
Show resolved Hide resolved
}

if token == "" {
return &TelegramClient{
Bot: nil,
Timeout: timeout,
}, nil
}

bot, err := tb.NewBot(tb.Settings{
Token: token,
Client: &http.Client{Timeout: timeout * time.Second},
})
if err != nil {
return nil, err
}

result := TelegramClient{
Bot: bot,
Timeout: timeout,
}
return &result, err
}

// Send message, skip if telegram token empty
func (client TelegramClient) Send(channelID string, item feed.Item) error {
if client.Bot == nil {
return nil
}

if channelID == "" {
return nil
}

contentLength, err := getContentLength(item.Enclosure.URL)
if err != nil {
return err
}

var message *tb.Message

if contentLength < maxTelegramFileSize {
message, err = client.sendAudio(channelID, item)
} else {
message, err = client.sendText(channelID, item)
}

if err != nil {
return err
}

log.Printf("[DEBUG] send telegram message: \n%s", message.Text)
return err
}

func getContentLength(url string) (int64, error) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@umputun how mocking http response?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see httptest. You can run test server and provide whatever response you need

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is an example of httptest.Server use in real life, from one of my projects https://github.com/go-pkgz/rest/blob/master/httperrors_test.go#L16

resp, err := http.Head(url) //nolint:gosec
if err != nil {
return 0, err
}

defer resp.Body.Close()

if resp.StatusCode != 200 {
return 0, fmt.Errorf("resp.StatusCode: %d, not equal 200", resp.StatusCode)
}

log.Printf("[DEBUG] Content-Length: %d, url: %s", resp.ContentLength, url)
return resp.ContentLength, err
}

func (client TelegramClient) sendText(channelID string, item feed.Item) (*tb.Message, error) {
message, err := client.Bot.Send(
recipient{chatID: channelID},
client.getMessageHTML(item),
tb.ModeHTML,
tb.NoPreview,
)

return message, err
}

func (client TelegramClient) sendAudio(channelID string, item feed.Item) (*tb.Message, error) {
httpBody, err := client.downloadAudio(item.Enclosure.URL)
defer httpBody.Close() //nolint:staticcheck
if err != nil {
return nil, err
}

audio := tb.Audio{
File: tb.FromReader(httpBody),
FileName: client.getFilenameByURL(item.Enclosure.URL),
MIME: "audio/mpeg",
Caption: client.getMessageHTML(item),
Title: item.Title,
}

message, err := audio.Send(
client.Bot,
recipient{chatID: channelID},
&tb.SendOptions{
ParseMode: tb.ModeHTML,
},
)

return message, err
}

func (client TelegramClient) downloadAudio(url string) (io.ReadCloser, error) {
clientHTTP := &http.Client{Timeout: client.Timeout * time.Second}

resp, err := clientHTTP.Get(url)
if err != nil {
return nil, err
}

log.Printf("[DEBUG] start download audio: %s", url)

return resp.Body, err
}

// https://core.telegram.org/bots/api#html-style
func (client TelegramClient) tagLinkOnlySupport(html string) string {
p := bluemonday.NewPolicy()
p.AllowAttrs("href").OnElements("a")
return p.Sanitize(html)
}

func (client TelegramClient) getMessageHTML(item feed.Item) string {
title := strings.TrimSpace(item.Title)

description := client.tagLinkOnlySupport(string(item.Description))
description = strings.TrimSpace(description)

messageHTML := fmt.Sprintf("%s\n\n%s\n\n%s", title, description, item.Enclosure.URL)

return messageHTML
}

func (client TelegramClient) getFilenameByURL(url string) string {
_, filename := path.Split(url)
return filename
}

type recipient struct {
chatID string
}

func (r recipient) Recipient() string {
if !strings.HasPrefix(r.chatID, "@") {
return "@" + r.chatID
}

return r.chatID
}
Loading