Bot api server general questions

[Fedorus]

1. Is it right that this is bot-api to client-api proxy for bots?
2. Can we extend this server api with methods that exist in client api and can be called by bots but does not exist in bot api?

[levlam]

1. Yes. And it ever was.
2. No. Bots have access only to methods required for the Bot API implementation.

[Fedorus]

This is list of methods from client api which can be called while logged in as bot:
https://tl.telethon.dev/methods/botindex.html
And among them there is GetFullUser method that does not exist in official bot API... but still can be used by bots...

[giautm]

Hey, Is there an docker version to deploy bot server to any cloud that support docker?

[levlam]

@Fedorus This is not true. GetFullUser is used for getChat implementation. All UserFull fields not included in getChat response are useless for bots: bots don't have peer settings, can't change chat notification settings, can't block users, can't call users, don't have privacy settings, don't have scheduled messages, don't have video calls and can't move chat with the user to Archive.

[levlam]

@giautm No. You need to build the Bot API server yourself using provided build instructions to use.

[gabbhack]

What about limits on sending messages and so on? As far as I know, the limits for bots using the client-api (telethon/pyrogram) are stricter than for bots using official bot-api. Does this depend on whether the app_id is "trusted"? If bot support has increased limits for my bot, will they still apply on my server?

[fskuratov]

So, briefly, as far as I understand no new methods for Bots are available? No Recent actions history, or chat members list, right? Any exclusions maybe, or someone can add them by forking and rewriting the current BotAPI server?

[levlam]

@gabbhack

What about limits on sending messages and so on?

Limits always was server-side and never depended on api_id or something similar.

As far as I know, the limits for bots using the client-api (telethon/pyrogram) are stricter than for bots using official bot-api.

This is false.

If bot support has increased limits for my bot, will they still apply on my server?

Yes.

[levlam]

So, briefly, as far as I understand no new methods for Bots are available?

Briefly, you can run Bot API server locally now.

No Recent actions history, or chat members list, right?

Yes, because that polling of those lists wouldn't work. You can't poll Recent actions for million chats in runtime. Stay tuned for the future updates, though.

Any exclusions maybe, or someone can add them by forking and rewriting the current BotAPI server?
No, it isn't possible to add them currently.

[RememberTheAir]

No. Bots have access only to methods required for the Bot API implementation.

Just a clarification: as far as I remember, using any MTProto client and logging in as a bot account, the server doesn't forbid you to send a channels.getParticipants request. A similar method is not available in the bot api, as the bot api doesn't expose any method to fetch a chat's members list. What is currently stopping someone from forking this project and add a method to the current available methods to request the members list using tdlib's getSupergroupMembers? Maybe tdlib doesn't let you use such methods if you login as a bot - or underneath getSupergroupMembers uses some other mtproto method that is not available to bots (so one would have to fork this project and tdlib to add this functionality)?

[levlam]

@RememberTheAir
Just a clarification: getSupergroupMembers is required for the Bot API implementation, and you can use it with filters not exposed in the Bot API. But flood limits for such usages aren't increased for bots, so you are likely to exceed them very fast.

[RememberTheAir]

@levlam ah, I see. One should expect mtproto/tdlib to accept just method requests strictly required to what the bot api allows to do - getSupergroupMembers is required to the current set of functionalities offered by the bot API (eg, getting a group's administrators), so it's not something Telegram left out "just because". Makes sense, thanks for the clarification. That also means that there are some MTProto methods that can be more broadly extended by a bot api fork (such as getSupergroupMembers), if you take in account the tight limits enforced by telegram's servers

[funnyflowerpot]

Thanks @levlam for proving quick and helpful answers here!

Question: What parts of the data transfer between Telegram client, Telegram bot, and local Bot API server are going through official Telegram servers?

In the previous version of the Bot API the data flows were pretty clear to me, but now I find it more difficult to see which network host is talking to whom and when. I was searching for details or maybe something like a sequence diagram, but I couldn't find related information so far.

I am also asking from a data privacy perspective. So in other words: What data (and metadata) could be still read by official Telegram servers, when a local Bot API server is used? I think including such information in the readme file of this repository would be great in terms of transparency.

[funnyflowerpot]

Here is how I understand the network communication between the relevant hosts Telegram client (C), Telegram bot (B), local Bot API server (BAS), and official Telegram servers (OTS). Is the following outline correct? If I missed a section in official documents describing this, please let me know!

1. BAS starts (no network requests here).
2. B starts and registers with BAS.
3. B tells OTS that it can be reached via BAS.
4. Using C a user attempts to start a chat with B.
5. C asks OTS for information on B.
6. OTS tells C that B's API server is BAS.
7. C sends data to BAS, BAS forwards data to B, and vice versa.

And so:

• All subsequent communication between C and B happen with BAS as a proxy.
• C and B never communicate directly with each other. (This also was the case before local Bot API servers.)
• OTS are only needed to establish communication between C and BAS in the beginning, but not afterwards.
• OTS and BAS never communicate directly with each other.

Is this description correct? Thank you!

[RememberTheAir]

Sorry for the low effort diagram. This is a map of how the various involved entities interact with each other.

Using your abbreviations, C never directly interact with BAS (that is, the user's Telegram app never directly connects to the server where an instance of telegram-bot-api is running). All the communications go through Telegram's servers (OTS), which forwards them to the two mtproto clients involved: the Telegram app the user C is using on their device, and the tdlib instance telegram-bot-api (BAS) is running under the hood.

BAS just acts as a front-end to the Telegram API, which is way more difficult to implement than a common Rest API (which is what BAS is supposed to be - an additional level of abstraction to interact with the Telegram API as a bot account). A rest api is more familiar and friendly to developers, so it's easier to pick up.

Note: in the image above, you can see only one telegram-bot-api instance, but there can be a number of these now, each one of them used by a number of bots. Also, the machine where the api is hosted can also be used to host the actual bot code, of course

[itsMoji]

Is there anyway to get file upload/download progress?

[funnyflowerpot]

@RememberTheAir Thank you very much for the explanation! The diagram also helps a lot, particularly the details that the Bot API server is a tdlib client and communicates via MTProto. This helped me to better grasp the idea behind tdlib and puts the benefits of a local Bot API server into perspective. Thanks again!

[micheleselea]

Why using the bot server api locally should improve performance? There is just one more layer instead to communicate directly with telegram servers.. I don't get why should be better instead using bot api as usual.
If the architecture whould like @funnyflowerpot said There will be benefit otherwise I don't understand

[JrooTJunior]

Hey, Is there an docker version to deploy bot server to any cloud that support docker?

https://hub.docker.com/r/aiogram/telegram-bot-api

[kolayne]

Why using the bot server api locally should improve performance? There is just one more layer instead to communicate directly with telegram servers. I don't get why should be better instead using bot api as usual
If the architecture whould like @funnyflowerpot said There will be benefit otherwise I don't understand

Absolutely agree.
At first, I supposed it was created to accept PRs implementing new methods or something like that from the developers using Telegram API. Then I figured out that's not an instance of what is running on the Telegram side and started thinking something similar to what @RememberTheAir has described. After reading his explanation I felt like bingo!.. But that's wrong too...

@RememberTheAir, @funnyflowerpot, could you, please, help me understand, what are the advantages of a local bot API server?

[gabbhack]

@micheleselea

I don't get why should be better instead using bot api as usual.

The regular bot api is also an "extra" layer between bot and Telegram servers. A local bot api can improve performance due to a low ping and an extended limits.

[kolayne]

@gabbhack, does it mean that usual bots interact with Telegram servers via Telegram's instance of this project (global Bot API Server?)?
Are files stored on bot API servers rather than on main Telegram servers (tdlib)? Will they send a request to my local Bot API Server each time a user wants to download a file sent by my bot?

[gabbhack]

@kolayne

does it mean that usual bots interact with Telegram servers via Telegram's instance of this project (global Bot API Server?)?

Yes.

Are files stored on bot API servers rather than on main Telegram servers (tdlib)? Will they send a request to my local Bot API Server each time a user wants to download a file sent by my bot?

All files are stored on Telegram servers, not on bot api server, but I think bot api can cache [1] [2] files (or something else) that the bot requested to download.

[micheleselea]

@gabbhack ok for the limitation extended with local server, but the lower ping time can be just from you and your bot api, and than you have the "ping time" between bot api server and telegram. I see only one more node that can be slow down communication, because you always speak with a bot api even in the standard configuration, but is a bot api that run on telegram server so the mtptoto communication I think it's faster than which one with local server api and telegram.
I keep thinking that is for limitation extension and not for performance reason. @levlam what do you think about? that whould really improve performance? and if so why?

[gabbhack]

@micheleselea Popular bots have a problem - their servers can handle a large number of requests, but bot api sends maximum of 100 simultaneous requests (see setWebhok max_connections parameter). This causes some users to get a response from the bot later, but it's not about the power of the bot's server. Local bot api can send 100,000 simultaneous requests, which may increases the bot's throughput.

These results may not be relevant, but a simple query was faster with a local server. Of course, this depends on the server location: https://t.me/aiogram_ru/334063

[micheleselea]

ok understand, because the MTProto communication between local bot and telegram bot is not limited as direct bot API are. So the local-bot-api implementation is not restricted from the point of view of maximum number of https request?

[micheleselea]

So basically the main advantage of this infrastrutture is that the conversion between HTTP BOT API and MTPROTO is done locally and than MTPROTO si used to "multiplex" the BOT API CALL in a already established channel. That way can be interesting to test...I'll try

[funnyflowerpot]

@kolayne @micheleselea From another issue:

[A server run by means of this project] is exactly the same server as run at https://api.telegram.org.

This is where I see the advantages of a bot API server: In the diagram above we see an official Telegram server, the bot API server and bots. If the bot API server does not run on the same host as an official Telegram server, but bot API server and bots share the same host, then network communication between network hosts does not happen over HTTP, but MTProto. Compared to HTTP the MTProto protocol is most surely providing better performance for network communication with Telegram apps in most cases.

And as @gabbhack said web hooks for real-time communication might be a good reason for a local bot API server too. After all HTTP and in particular an additional TLS layer (HTTPS) can provide quite some overhead. Keeping this kind of communication within a host and using MTProto's (builtin) encryption can help a lot I guess.

[funnyflowerpot]

I think for certain use cases the choice of using a local API server or not is an advantage for the community. That being said, a decentralised version (as I wondered about above), where Telegram client and bot API server bypass the official Telegram servers, might speed things further up considerably. There would be other advantages too, for example lower load of the official Telegram servers or improved data privacy by giving more control to the bot providers (there are risks too though to be fair). But I am not sure whether MTProto would support that change in network communication anyhow.

[levlam]

Connection; Keep-Alive is deprecated since HTTP/1.1, because this is a default behavior for servers. You don't need to specify the header. Instead your client HTTP library should be able to reuse network connections.

[micheleselea]

@levlam HTTP header is Connection: Keep-Alive and I don't think is deprecated, and so I don't think that reuse connection is the default behavior, instead some servers DO NOT allow reusing connections. My client HTTP lib can reuse connections, but usually I have to set that behavior. Anyway that is going to be an off topic, I'll try to reuse connections to api.telegram.org that I wrongly thought was not allowed

[makisukurisu]

Even though I can modify some parameter I can't change how much data can be passed trough callback_data parameter? Or can I?

If not, will it be increased in near future?

Thanks for reply in advance.

[levlam]

I can't change how much data can be passed trough callback_data parameter?

You can't.

If not, will it be increased in near future?

No. It should be as small as possible to minimize client traffic and 64 bytes is more than enough to store unique ID for a data of any size.

[gabbhack]

Why should the client see and transmit the date? Why not store the original date on the telegram server and send a minimal ID to the client? When you click on the button, the server would substitute the original date.

[jcmag]

currently bots are limited to 50MB when they send files; could we bypass this limit by running a local API server?

[RememberTheAir]

@jcmag yes: https://github.com/tdlib/telegram-bot-api#usage

[puppy0cam]

@micheleselea @kolayne This is exactly the same server bots communicate with on api.telegram.org. There are no additional or removed layers.

There can be various performance improvements, that can be achieved with the local server. For example, api.telegram.org is located in the Europe, but some bots are bound to Asia or America datacenters. These bots would work much faster if switched to the local Bot API server located in the corresponding region. Another performance improvements is ability to upload and download local files. This way the files don't need to be additionally transferred over HTTP. Also, responses to webhook requests can be sent much faster to a local server, which increases theoretical webhook throughput. Despite that improvements most of the bots don't need to switch to a local server and can continue using api.telegram.org as a gateway.

Is there any way to easily determine what datacenter our bot is bound to?

[micheleselea]

ping api.telegram.com and you will see that is somewhere in Nederland.
For your bot, just look at datacenter IP where the your bot is running

[micheleselea]

@levlam about my question on "delete" requests I posted yesterday:
what I want to understand is if the HTTP requests to our local bot api server are faster than requests sent directly to the api.telegram.org. I mean: suppose I need to delete 5000 messages (sometimes happens in my bot), because the delete API can be called for one message at a time, I have to do 5000 call to api.telegram.org. Sometimes I get a message that told me to wait a while because I did to much requests. Do you think that from this point of view can be better to have a local server bot api server? Do you think I can have less flood reply?

[gabbhack]

@micheleselea #1 (comment)
If I understand correctly, the limits on the local server are the same as those indicated in FAQ, because these limits are regulated by Telegram servers.

[micheleselea]

ok understand, so I think that I can keep running on api.telegram.org and I'm trying the reuse https connections

[gabbhack]

ping api.telegram.com and you will see that is somewhere in Nederland.
For your bot, just look at datacenter IP where the your bot is running

This is not about where the bot is located, but about which telegram data center it is linked to, because users from different regions are linked to different data centers. If I understand correctly, this may cause a situation when a bot linked to an Asian data center, for example, connects to api.telegram.org, which is located in Europe, and after the bot api connects to the Asian data center.

[micheleselea]

Ok but just ping api.telegram.org from location where you have bot running and you will find where your bot is connecting to

[RememberTheAir]

@micheleselea a bot using the official bot API (api.telegram.org) will always connect to some Netherlands server, because this is where the api is hosted. Then the Netherlands server will connect to the user Data Center where the bot account is stored (there are 5 across the world, as far as I remember) to pass the request to Telegram.

So, let's say I'm hosting the bot in an US server, and I have to send a message to an user - the message my bot sends from my server takes this route: US (server where the bot is hosted) -> Netherlands (api.telegram.org) -> US datacenter -> [some additional routing probably telegram servers do based on where the involved account's data is stored] -> user phone. The goal here is to cut the US -> Europe -> US part, which in some cases is unnecessary network overhead. If, in this scenario, you host an intance of telegram-bot-api in an US server, you don't probably have to have your requests routed all the way to Europe, because telegram-bot-api will connect (through tdlib) directly to the US datacenter (via mtproto).

As far as I know (please someone correct me if I'm wrong), each Telegram account has its data stored in a specific datacenter, and it never migrates (or migrates only if some parameters are met, very unlikely nonetheless). Every bot (or supergroup or channel) an account creates, will be stored in the account's datacenter. When you create your account, Telegram picks your datacenter based on proximity (again talking from personal experience and what I'm saying might not be 100% correct). That means that if your account is atored in the US datacenter, your bots will be too, and that means that when you send a request to api.telegram.org, your request will be routed back to the US datacenter. If your account is bound to the European datacenter, which is very near where the bot api is hosted, and you host your bot in an US server, then hosting your own telegram-bot-api won't probably help that much with that, because your requests would do US (machine where you host the bot) -> US (your instance of telegram-bot-api) -> Netherlands (european datacenter) (opposed to US (bot) -> Netherlands (api.telegram.org) -> Netherlands (dc)); so your requests have to cross the ocean anyway. In the former scenario (US -> Netherlands -> Netherlands), it's more a matter of picking a proper server location where to host your bot. To decide whether it is worth to host a telegram-bot-api instance or not, it would be helpful to know to which datacenter your bot would connect through mtproto. In case it's not the European one, it might be worth hosting your instance to avoid your requests to be routed to Europe and then to the bot account's datacenter

[Poolitzer]

As far as I know (please someone correct me if I'm wrong), each Telegram account has its data stored in a specific datacenter, and it never migrates (or migrates only if some parameters are met, very unlikely nonetheless)

They do migrate, if their account connects consistently from a different location. If you move to Europe from the US, after some time, your account will move with you :)

[levlam]

@Poolitzer No, they don't migrate currently, but this could be implemented in the future.

[puppy0cam]

@Poolitzer No, they don't migrate currently, but this could be implemented in the future.

Could this be implemented for bot accounts sooner rather than later? Even if we can run an instance of the bot API locally, many people may still prefer to use the official servers.
This could be a setting in BotFather to change the region our bot is located on to help migrate our bot to the region with the largest amount of our users.

[Poolitzer]

Could this be implemented for bot accounts sooner rather than later? Even if we can run an instance of the bot API locally, many people may still prefer to use the official servers.

For people using the cloud, it shouldn't make a difference. If I understood this correctly (apparently gotta be careful about my formulation here), all the bot API instances hosted by telegram (the cloud) are running on servers in the Netherlands. So changing the home DC of a bot wouldn't bypass the Netherlands if Telegram doesn't also provide cloud instances in places where the other DCs are. But now you are already requesting two things ;P

This could be a setting in BotFather to change the region our bot is located on to help migrate our bot to the region with the largest amount of our users.

That would indeed work with local bot API instances, yes. For reasons pointed out further up in the thread.

[abdoo9]

how many bots I can run on the local server? I've reached 1789 and start getting this error

{
  ok: false,
  error_code: 400,
  description: 'Bad Request: bad webhook: Failed to create a socket'
}
{
  ok: false,
  error_code: 500,
  description: 'Internal Server Error: restart'
}

[Poolitzer]

As stated in #73 you should only be limited by your servers resources