Automatic renewal of OAuth2 token

[Rafael-Ayala]

Problem

Firstly, thanks a lot for the excellent package and all the efforts to maintain it.

I have read on the changelog (https://docs.ropensci.org/rtweet/news/index.html) that it should be possible to automatically renew OAuth2 tokens (required by the V2 API for posting tweets through tweet_post()). This would be required in order to set up a bot aiming just to post tweets every few hours (without collecting any data or interacting in any way with replies).

However, when I include in my code calls to auth_renew() every hour, I get the following error message when calling it on an expired OAuth2 token:

Error in `load_client()`:                            
! Can't find saved client with name 'Application_name'
Run `rlang::last_trace()` to see where the error occurred.

I would like to note that I manage to successfully post through tweet_post() using the free plan as long as I provide an OAuth2 token. But basically I would like to have to avoid to provide authorization manually every 2 hours to renew the token (if possible).

Expected behavior

I expected that it would just renew the token and enable continued bot operation.

Reproduce the problem

clientID <- ""
clientSecret <- ""
client <- rtweet_client(clientID, clientSecret, "Application_name")
oauth2 <- rtweet_oauth2(client)
while(TRUE) {
    tweet_post("test", token = oauth2)
    Sys.sleep(3600)
    oauth2 <- rtweet:::auth_renew(oauth2)
    Sys.sleep(3600)
    oauth2 <- rtweet:::auth_renew(oauth2)
    Sys.sleep(3600)
    oauth2 <- rtweet:::auth_renew(oauth2)
}

rtweet version

packageVersion("rtweet")
‘1.2.0.9005’

[llrs]

In order to automatically renew the token you must have the client and the oauth2 saved in the system. If you save them with auth_save(oauth2, "my_oauth2") and client_save(client), rtweet should be able to renew it without you calling auth_renew. I'll try to modify the documentation and make a more sensible error message but the error "Can't find saved client with name" is hinting to that.

If the httr2 version is 0.2.3.9000 or higher it should automatically renew
the access without your explicit confirmation via the browser (sorry, I forgot to add this to the DESCRIPTION in the devel branch).

[Rafael-Ayala]

Thanks a lot! I have now tested it over multiple days and have been able to renew the token automatically with no problem as described :)

[llrs]

Thanks for confirming it works well.

[guga31bb]

Thanks a lot! I have now tested it over multiple days and have been able to renew the token automatically with no problem as described :)

Would you be able to share a bit more how you got this working @Rafael-Ayala? I saved the client and auth as instructed, do

rtweet::client_as("PATH/NAME.rds")
rtweet::auth_as("PATH/NAME.rds")

rtweet::tweet_post("test")

It works for the first two hours but then 2 hours after that I get

Enter authorization code:

With it opening a browser to get me to authenticate manually again. How does one get the auto-refresh to work?

[llrs]

I'm closing this @guga31bb but you need the devel version of both httr2 and rtweet.

[Rafael-Ayala]

My apologies for the late reply. As @llrs mentioned, you need those devel versions.
I did not do anything special. Just included

client <- rtweet_client(clientID, clientSecret, "app_name")
oauth2 <- rtweet_oauth2(client)
client_save(client)
auth_save(oauth2, "name_fot_oauth2")

Before the bot loop, and then the following in the bot loop:

oauth2 <- rtweet:::auth_renew(oauth2)

[llrs]

That last step, calling the internal function auth_renew, shouldn't be used (It is an internal function used internally when checking if the token should be updated). Did it work without it?

[Rafael-Ayala]

I see! I have not tried without it. I have just restarted my bot without that call. Will post here the results

[guga31bb]

For what it's worth, I followed the exact steps above (without auth_renew) with rtweet 1.2.0.9003 and httr2 0.2.3.9000 and it still asks for an authorization code after 2 hours. So for anyone else reading this having problems getting it to work without manual intervension, you aren't alone!

rtweet::tweet_post("another test")
Reading client from 'PATH/FILE.rds'
Enter authorization code: 

[llrs]

Thanks for double checking. The joys of using an in-development version (which commit of httr2 are you using? check it out with sessioninfo::session_info("httr2")).
This message was added 2 weeks ago in httr2. I will try to fix it, as auth_renew shouldn't be needed and the call to the httr2 function that triggers the message shouldn't happen.

I'll need to figure the way to do it with the upcoming release of httr2 (which seems closer, as there are a lot of commits recently). Hopefully this is resolved before httr2 update, which would mean I could release rtweet 1.3 to CRAN.

[Rafael-Ayala]

I have now also verified that it does not work for me unless I place a explicit call to auth_renew in the bot loop.
Please let me know if I can be of any help debugging this. Happy to share more details if required!

[guga31bb]

The joys of using an in-development version (which commit of httr2 are you using? check it out with sessioninfo::session_info("httr2")).

httr2 0.2.3.9000 2023-09-08 [2] Github (r-lib/httr2@8f77d41)

This is the most recent commit as of now.

Sounds like I could get it to work by using the httr2 version that you both were using back in July, but obviously that's a short term fix (and not helpful to you!). Looking at the commit history of httr2 it does seem like the auth stuff is a moving target right now so probably makes sense to just wait until they finish.

[Rafael-Ayala]

I am not sure if the problem has to do with using a more recent version of httr2. This is because I tested not placing a explicit call to auth_renew without updating httr2 (I still have the same devel version I was using back in July), and that resulted in automatic renewal not working. Just explicitly calling again auth_renew made it work

[llrs]

Oh, thanks @Rafael-Ayala. Then the problem is in the logic rtweet uses to renew the credentials. Luckily today I have some surprise free time and I will look into it.

If you can provide the trail/conditions about how this message appears I would appreciate it. I.e. if anyone can show the process from rtweet::post_tweet to httr2::oauth_flow_auth_code (which triggers the requests) it will be helpful. If not, don't worry I'll do that myself.

[guga31bb]

A bit more information if it's helpful- it's not just re-authorizing that is problematic with the current dev version of httr2. I can't even get it to authenticate in the first place:

# old httr2: this works
  
  # july 12 commit https://github.com/r-lib/httr2/commit/7f795cd4ae688ad71c445b8b3225e1a65fdf6b0a
  remotes::install_github("r-lib/httr2@7f795cd4ae688ad71c445b8b3225e1a65fdf6b0a")

  library(rtweet)
  
  client <- rtweet_client(
    client_id = "XX", 
    client_secret = "XX", 
    "Baldwin bot")
  
  oauth2 <- rtweet_oauth2(client)
  
  Waiting for authentication in browser...
  Press Esc/Ctrl + C to abort
  
# new httr2: this does not work
  
  # (uninstall httr2, restart R)
  # most recent commit
  remotes::install_github("r-lib/httr2@8f77d4145a7d887e7f30c449f28cda150a61d744")
  
  library(rtweet)
  
  client <- rtweet_client(
    client_id = "XX", 
    client_secret = "XX", 
    "Baldwin bot")
  
  oauth2 <- rtweet_oauth2(client)
  
  Enter authorization code: 
    

In the browser (on twitter), I see

  Something went wrong
  You weren’t able to give access to the App. Go back and try logging in again.

I should have mentioned this in the first place, but I never had authenticating working and assumed it was a problem on my end since it was working for you guys. So I was authenticating with an older version of httr2 and then trying to refresh credentials with the new version. But it seems clear now that there's an issue with a combination of the current versions of rtweet / httr2

[llrs]

I recently tested with the latest httr2 and the authentication part works even starting from a fresh installation.
I think httr2 changed some code that fixed this (I think I or some user opened a issue related to this error).

To re-authorize your app you might get asked each time via the browser to authenticate the app if too much time passed since last authentication.
But I will not step up to develop a solution to avoid this, further feature requests to improve the re-authentication should go to httr2 package.

I'll plan to submit this week a new version to CRAN, any bug reports will be appreciated.

[guga31bb]

I think I or some user opened a issue related to this error

That was me! r-lib/httr2#319

And thank you for the update