Dumping channels with the test branch in cygwin

[alinafariyeh]

I have built the test branch like this:

1. Installed Complete Cygwin
2. Then Cloned the test branch:
   git clone --recursive https://github.com/vysheng/tg.git -b test --single-branch tg-test
3. configured (./configure)
4. Patched the makefile and loop.c as Instructed Here
5. executed make command.
6. The telegram-cli is built without any problem I run telegram-cli with the following parameters:
   telegram-cli --json -P 9009
   as Instructed.
   I also run the "ruby telegram-history-dump/telegram-history-dump.rb" command in another terminal.
   But it skips all the dialogs and says:

I, [2016-01-30T21:07:27.425554 #4304]  INFO -- : Loading dumper module 'json'
I, [2016-01-30T21:07:27.427640 #4304]  INFO -- : Attaching to telegram-cli control socket at localhost:9009
I, [2016-01-30T21:09:53.108433 #4304]  INFO -- : Skipping 16 dialogs: "کتابخانه_الف#1", "++maryam", "Maman", "خانوادمون", "کتابخانه_الف", "Top10kala.ir", "Ali_Nfr", "Amani", "Mohamad_Ferdosi", "Ali_Sedaghat", "Saeed_Zhiani", "جوكستان_الف", "بچه_های_رجایی", "PWUTX", "کتابخانه_الف", "Top10kala.ir"
I, [2016-01-30T21:09:53.108510 #4304]  INFO -- : Backing up 0 dialogs: (none)
I, [2016-01-30T21:09:53.108537 #4304]  INFO -- : Finished

(It gets stuck a little bit after Attaching to telegram-cli control socket at localhost:9009 and then skips the dialogs and finishes)
I have tried changing the config.json5 But it doesn't affect anything. I have dumped history using the master branch build of telegram-cli and I didn't have any problem. but as you can see I can't dump anything with the test branch build. Any help would be appreciated.

[tvdstaaij]

I prefer to ensure compatibility once the new features are merged into master, because the test branch may change / break randomly anyway. If you want to dump channels, you can use a known working tg revision from the test branch (it worked for me with revision c69f2a2, also see #4). Or is there another reason you want to use the test branch?

But considering how long it is taking tg to merge test into master, I may attempt to get telegram-history-dump working with tg's test branch. I can't work on it right now however because of ISP issues at home.

[tvdstaaij]

I finally looked into this, it seems telegram-cli changed some names in the API. Added a compatibility workaround, it works for me now with telegram-history-dump@master and tg@test (tg@ca8c525 at the time of writing).

[alinafariyeh]

Thanks for your response. It seems the error I mentioned above has been fixed I am backing up channels that their names are unicode (persian language) and the messages also contain unicode characters so I wonder if it is the problem:

$ ruby telegram-history-dump.rb
I, [2016-03-04T11:50:35.244738 #3972]  INFO -- : Attaching to telegram-cli control socket at localhost:9009
I, [2016-03-04T11:50:37.305275 #3972]  INFO -- : Skipping 1 dialogs: "Telegram"
I, [2016-03-04T11:50:37.305609 #3972]  INFO -- : Backing up 40 dialogs: "خط_ویژه", "کارگزاران_جوان", "اعتماد_ملی", "سازمان_عدالت_و_آزادی", "Radiofarda_رادیوفردا", "پیک_خبری_اراده_ملت_ایران", "صادق_زیباکلام_-_Sadegh_Zibakalam", "BBCPersian", "کانال_فرارو", "حمید_رسایی", "نواندیش", "hashemirafsanjani.ir", "خبرنامه_نداي_ايرانيان", "مجمع_اصلاح_طلبان", "اصلاحات_نیوز_✌", "لينك_رسانه_هاي_اصلاح_طلب", "سایت_خبری-تحلیلی_کلمه", "آخرين_خبر", "عصر_ایران", "خبرهای_فوری_/_مهم🔖", "سایت_خبری-تحلیلی_کلمه", "مجمع_اصلاح_طلبان", "حمید_رسایی", "BBCPersian", "hashemirafsanjani.ir", "Radiofarda_رادیوفردا", "خبرهای_فوری_/_مهم🔖", "خط_ویژه", "لينك_رسانه_هاي_اصلاح_طلب", "اصلاحات_نیوز_✌", "نواندیش", "آخرين_خبر", "کانال_فرارو", "عصر_ایران", "صادق_زیباکلام_-_Sadegh_Zibakalam", "سازمان_عدالت_و_آزادی", "خبرنامه_نداي_ايرانيان", "اعتماد_ملی", "کارگزاران_جوان", "پیک_خبری_اراده_ملت_ایران"
I, [2016-03-04T11:50:37.308582 #3972]  INFO -- : Dumping "خط_ویژه" (range 1-100)
I, [2016-03-04T11:50:38.966975 #3972]  INFO -- : Dumping "خط_ویژه" (range 101-200)
telegram-history-dump.rb:39:in `exec_tg_command': Telegram command <history> failed: {"error_code"=>71, "result"=>"FAIL", "error"=>"RPC_CALL_FAIL 400: OFFSET_INVALID"} (RuntimeError)
        from telegram-history-dump.rb:63:in `block in dump_dialog'
        from /usr/share/ruby/2.2.0/timeout.rb:88:in `block in timeout'
        from /usr/share/ruby/2.2.0/timeout.rb:32:in `block in catch'
        from /usr/share/ruby/2.2.0/timeout.rb:32:in `catch'
        from /usr/share/ruby/2.2.0/timeout.rb:32:in `catch'
        from /usr/share/ruby/2.2.0/timeout.rb:103:in `timeout'
        from telegram-history-dump.rb:62:in `dump_dialog'
        from telegram-history-dump.rb:270:in `block in <main>'
        from telegram-history-dump.rb:266:in `each'
        from telegram-history-dump.rb:266:in `each_with_index'
        from telegram-history-dump.rb:266:in `<main>'

I have compiled the test branch that you have mentioned tg@ca8c525 .
I can give you my telegram account so you can test if it is a problem with me doing this or it's a bug in history dumper. I am really eager to help solve this issue.

[tvdstaaij]

Looks like you are running into this issue now:

vysheng/tg#947

Try the temporary solution suggested there :)

[tvdstaaij]

Alternatively, you could make a special configuration file for dumping channels only (set backup_users and backup_groups to null), with chunk_size set to a very large number, and maybe a larger chunk_timeout. This will prevent the error, but I don't know if it will work with channels that have a lot of messages.

[alinafariyeh]

Thanks changing the chunk size and chunk timeout makes it work. but it only gives me 100 messages from each channels I tried changing the chunk size and chunk timeout to make right but it has no effect on the result.

[tvdstaaij]

Did you try the first suggestion, recompiling tg@test with the code change suggested in vysheng/tg#947, then running with normal chunk size? If that does not work either I'm afraid I don't know a solution other than waiting until this is properly fixed by tg devs.

[alinafariyeh]

Yes I have tried it. it Gives me the following error:

$ ruby telegram-history-dump.rb
I, [2016-03-04T15:54:02.140417 #5196]  INFO -- : Attaching to telegram-cli contr                                            ol socket at localhost:9009
I, [2016-03-04T15:54:05.326741 #5196]  INFO -- : Skipping 1 dialogs: "Telegram"
I, [2016-03-04T15:54:05.327031 #5196]  INFO -- : Backing up 40 dialogs: "خط_ویژه                                            ", "کارگزاران_جوان", "اعتماد_ملی", "سازمان_عدالت_و_آزادی", "پیک_خبری_اراده_ملت_ا                                            یران", "صادق_زیباکلام_-_Sadegh_Zibakalam", "حمید_رسایی", "آخرين_خبر", "خبرنامه_ن                                            داي_ايرانيان", "نواندیش", "hashemirafsanjani.ir", "سایت_خبری-تحلیلی_کلمه", "لينك                                            _رسانه_هاي_اصلاح_طلب", "BBCPersian", "خبرهای_فوری_/_مهم🔖", "مجمع_اصلاح_طلبان", "                                            کانال_فرارو", "عصر_ایران", "اصلاحات_نیوز_✌", "Radiofarda_رادیوفردا", "سایت_خبری-                                            تحلیلی_کلمه", "مجمع_اصلاح_طلبان", "حمید_رسایی", "BBCPersian", "hashemirafsanjani                                            .ir", "Radiofarda_رادیوفردا", "خبرهای_فوری_/_مهم🔖", "خط_ویژه", "لينك_رسانه_هاي_ا                                            صلاح_طلب", "اصلاحات_نیوز_✌", "نواندیش", "آخرين_خبر", "کانال_فرارو", "عصر_ایران",                                             "صادق_زیباکلام_-_Sadegh_Zibakalam", "سازمان_عدالت_و_آزادی", "خبرنامه_نداي_ايران                                            يان", "اعتماد_ملی", "کارگزاران_جوان", "پیک_خبری_اراده_ملت_ایران"
I, [2016-03-04T15:54:05.330844 #5196]  INFO -- : Dumping "خط_ویژه" (range 1-100)
I, [2016-03-04T15:54:07.432302 #5196]  INFO -- : Dumping "خط_ویژه" (range 101-20                                            0)
I, [2016-03-04T15:54:09.885462 #5196]  INFO -- : Dumping "خط_ویژه" (range 201-30                                            0)
I, [2016-03-04T15:54:12.213595 #5196]  INFO -- : Dumping "خط_ویژه" (range 301-40                                            0)
I, [2016-03-04T15:54:14.135582 #5196]  INFO -- : Dumping "خط_ویژه" (range 401-50                                            0)
I, [2016-03-04T15:54:15.985013 #5196]  INFO -- : Dumping "کارگزاران_جوان" (range                                             1-100)
I, [2016-03-04T15:54:19.729348 #5196]  INFO -- : Dumping "کارگزاران_جوان" (range                                             101-200)
I, [2016-03-04T15:54:22.229332 #5196]  INFO -- : Dumping "کارگزاران_جوان" (range                                             201-300)
/usr/share/gems/gems/json-1.8.2/lib/json/common.rb:155:in `initialize': no impli                                            cit conversion of nil into String (TypeError)
        from /usr/share/gems/gems/json-1.8.2/lib/json/common.rb:155:in `new'
        from /usr/share/gems/gems/json-1.8.2/lib/json/common.rb:155:in `parse'
        from telegram-history-dump.rb:36:in `exec_tg_command'
        from telegram-history-dump.rb:63:in `block in dump_dialog'
        from /usr/share/ruby/2.2.0/timeout.rb:88:in `block in timeout'
        from /usr/share/ruby/2.2.0/timeout.rb:32:in `block in catch'
        from /usr/share/ruby/2.2.0/timeout.rb:32:in `catch'
        from /usr/share/ruby/2.2.0/timeout.rb:32:in `catch'
        from /usr/share/ruby/2.2.0/timeout.rb:103:in `timeout'
        from telegram-history-dump.rb:62:in `dump_dialog'
        from telegram-history-dump.rb:270:in `block in <main>'
        from telegram-history-dump.rb:266:in `each'
        from telegram-history-dump.rb:266:in `each_with_index'
        from telegram-history-dump.rb:266:in `<main>'

[tvdstaaij]

That's too bad, I wish I could do something about it but I can't. Let's hope telegram-cli development gets more active again and issues like these get fixed.

The issue is unrelated to cygwin by the way, I think you probably realized that already.

[tvdstaaij]

Current status of channel dumping:

• Telegram-cli test branch is no longer needed, channels are now supported with master. I'll update the readme and config to reflect this.
• The telegram-cli offset issue that crashes dumps of channels with more than 100 messages is still not fixed.
• Supergroups seem to have some characteristics in common with channels but do not have the offset problem. Properly supporting supergroups needs some changes on my end though, working on that now.

[tvdstaaij]

Forgot to update this issue, both channels and supergroups can be dumped using the latest version of telegram-cli and telegram-history-dump.