Allow >50M audio upload to telegram

[umputun]

With #19 and #15 in place, it was tested in a few real configurations and works as expected. However, the limitation of 50M for upload feels strange from the user's point of view and we need to allow such big files.

Recently, as a part of radio-t/super-bot#3 I've played with github.com/go-telegram-bot-api/telegram-bot-api library and it seems to provide complete support of all telegram's API functionality, for both bots as well as other parts. @sgaynetdinov - maybe switching to this lib may help you to complete the task.

[kfimchenko]

Is there a good way how to do it? I know two ways, but all of them are meh
First one is send audio files by parts and telegram will play all audios continuesly, but playing order depending on telegram client.
And second one is about one big audio file, but you need upload .mp3 as a user(telegram api, not bot api), share media with a bot, and then bot can use that audio file by media id. It may be tricky in implementing.

https://core.telegram.org/bots/api#sending-files link to telegram documentation, about 50 MB limits

[sgaynetdinov]

@EatsLemons you talking about this #13 (comment) ?

[sgaynetdinov]

@sgaynetdinov - maybe switching to this lib may help you to complete the task.

@umputun I will try and create a PR

[kfimchenko]

@EatsLemons you talking about this #13 (comment) ?

Yes, it is. I didn't see that comment with solution

[sgaynetdinov]

feed-master    | ..... 05:53:27.378 [WARN]  {proc/processor.go:80} failed to remove, no bucket for all-umputun
feed-master    | ..... 05:53:27.706 [INFO]  {proc/store.go:71} save 1578179638-dcefc1c4332fdabcb88ed601edf1b67c3b7dcd6a - all-umputun - Радио-Т 683 - https://radio-t.com/p/2020/01/04//podcast-683/
feed-master    | ..... 05:53:28.251 [DEBUG] {proc/telegram.go:131} start download audio: http://cdn.radio-t.com/rt_podcast683.mp3
feed-master    | ..... 06:01:14.368 [WARN]  {proc/processor.go:119} failed to send telegram message, url=http://cdn.radio-t.com/rt_podcast683.mp3 to channel=-1001409915930, can't send to telegram for {URL:http://cdn.radio-t.com/rt_podcast683.mp3 Length:117205573 Type:audio/mp3}: Request Entity Too Large

@umputun an error while loading a large file remains, can try on this article to do (https://habr.com/ru/post/348234/) ?

[umputun]

see radio-t/super-bot#5 (comment)

[chuhlomin]

I've made some progress on this: https://github.com/chuhlomin/telegram-large-files-upload
Couldn't set the correct title yet.

[chuhlomin]

That is working now. Follow instructions in README and it will send large file to a channel or bot.
Not sure what to do next here.

[paskal]

I thought the idea would be writing some code in Go which will allow large files upload. The problem I stumbled upon is there are hundreds of Telegram Bot API libraries but I haven't found one for Telegram API, and documentation of Telegram API looks like a foreign language to me so I haven't figured out how to make necessary API calls without a library yet.

If somebody would find a Go Telegram API library, it would allow me to fix large files upload relatively easy.

[chuhlomin]

I tried to do it with go first but gave up. Here are my findings:

https://github.com/sdidyk/mtproto – MTProto implementation in Golang, hardcoded api_id and api_hash, probably using some old API version because temporary code comes via SMS, also not working – getting PHONE_NUMBER_UNOCCUPIED error.

https://github.com/cjongseok/mtproto – Telegram MTProto and proxy (over gRPC), haven't tried.

https://github.com/Arman92/go-tdlib – promising one, uses https://github.com/tdlib/td, couldn't make it work – reports missing *.h files on build (which are there)

[paskal]

@chuhlomin could you please help me figure this out? Is MTProto is the thing I should be looking for, is it an only way to access Telegram API? Is TDLib a superset of Telegram API which we could use, or it is something else?

[chuhlomin]

As I understand it MTProto is a protocol for talking to Telegram servers
Telegram TdLib is a relatively new library for building Telegram Clients.
So we can use either. Python library I ended up using uses MTProto

[paskal]

Thanks a lot! I'll give MTProto a shot. If that's possible from Python, I would be able to do the same in Go.

[paskal]

Three potential candidates for implementation: shelomentsevd/mtproto and cjongseok/mtproto which work with MTProto 1.0 which is deprecated and 9seconds/mtg which is actively developed but I don't know if it's supposed to be used like that.

Just for reference, actual (official?) TDLib library for Go is Arman92/go-tdlib.

[paskal]

Per off-GitHub talk with one library developer, it looks like TDLib is the way to go.

[chuhlomin]

I like go (and prefer to use it), but in this case does it really matter which language to use?
Especially when it's just a few lines in Python: https://github.com/chuhlomin/telegram-large-files-upload/blob/master/main.py

[paskal]

I have no objections if you know how to tie it all together. Feel free to proceed.

[umputun]

I would prefer all of this to stay in go for 2 reasons:

• ability to run without docker or without py & packages preinstalled on the machine
• single-lang programs are easier to maintain

However, if we don't have any other practical choice we may try the py route, but it has to be clearly an optional dependency (set as cli/env) doing just upload of the file and returning back the id, or whatever else API needs. This utility should be called from the primary process as a black box and receives status as the exit code and the id (or whatever else needed) as the last printed (to stdout) line.

In case if the cli param (or env) didn't set/passed it should continue to work the way it is doing now. Probably the same failback should be initiated if the upload failed.

[paskal]

I've spent ~4 hours trying to make TDLib work in Go and I haven't had complete success yet. It seems that using TDLib would require switching docker image to Debian and building the C++ library before compiling the Go program. With cmake building takes 1Gb of RAM and dies after 1 hour on 76% of compilation, g++ eats all my RAM and dies on ~65% of compilation.

I think that executing a program in Python is a simpler option in this particular case: Go code with C++ slows build by orders of magnitude and make Dockerfile ugly.

[umputun]

I think that executing a program in Python is a simpler option in this particular case

Makes sense. As much as I don't like mixing python code with go code, dealing with c++ interop feels even worse.

So, let's make it an external call of py script. We will also need a Makefile to prepare py (making sure the proper version installed and install dependencies).

[paskal]

@chuhlomin to clarify: I'm not doing anything here anymore, it's up to you to proceed.

[paskal]

As an alternative to Python implementation (#31) now we can try to use native Go xelaj/mtproto. I'll look into it when I'll have time.