Faraday 2.0

[iMacTia]

Faraday 2.0 🎉

The next major release is here, and it comes almost 2 years after the release of v1.0!
This release changes the way you use Faraday and embraces a new paradigm of Faraday as an ecosystem, rather than a library.
What does that mean? It means that Faraday is less of a bundled tool and more of a framework for the community to build on top of.

As a result, all adapters and some middleware have moved out and are now shipped as standalone gems 🙌!
But this doesn't mean that upgrading from Faraday 1.x to Faraday 2.0 should be hard, in fact we've listed everything you need to do in the UPGRADING.md doc.
Moreover, we've setup a new awesome-faraday repository that will showcase a curated list of adapters and middleware 😎.

This release was the result of the efforts of the core team and all the contributors, new and old, that have helped achieve this milestone 👏.

What's Changed

• Begin v2 release by @iMacTia in Begin v2 release #1297
• Autoloading, dependency loading and middleware registry cleanup by @iMacTia in Autoloading, dependency loading and middleware registry cleanup #1301
• Move JSON middleware (request and response) from faraday_middleware by @iMacTia in Move json middleware (request and response) from faraday_middleware #1300
• Remove deprecated Faraday::Request#method by @olleolleolle in Remove deprecated Faraday::Request#method #1303
• README v2 notice by @iMacTia in README v2 notice #1305
• Remove deprecated Faraday::UploadIO by @iMacTia in Remove deprecated Faraday::UploadIO #1307
• [1.x] Deprecate Authorization helpers in Faraday::Connection by @iMacTia in [1.x] Deprecate Authorization helpers in Faraday::Connection #1306
• Drop deprecated auth helpers from Connection and refactor auth middleware by @iMacTia in Drop deprecated auth helpers from Connection and refactor auth middleware #1308
• Add latest rubocop and use Ruby 2.6+ by @olleolleolle in Add latest rubocop and use Ruby 2.6+ #1311
• Automatically refresh website Team page on commit to default branch by @iMacTia in Automatically refresh website Team page on commit to default branch #1315
• Add Faraday 1.x examples in authentication.md docs by @iMacTia in Add Faraday 1.x examples in authentication.md docs #1320
• Fix passing a URL with embedded basic auth by @iMacTia in Fix passing a URL with embedded basic auth #1324
• Register JSON middleware by @mollerhoj in Register json middleware #1331
• Retry middleware should handle string exception class name consistently by @jrochkind in Retry middleware should handle string exception class name consistently #1334
• Improve request info in exceptions raised by RaiseError Middleware by @willianzocolau in Improve request info in exceptions raised by RaiseError Middleware #1335
• Remove net-http adapter and update docs by @iMacTia in Remove net-http adapter and update docs #1336
• update changelog 1.8.0 by @mcansky in update changelog 1.8.0 #1338
• Explain plan for faraday_middleware in UPGRADING.md by @iMacTia in Explain plan for faraday_middleware in UPGRADING.md #1339
• Scripts folder cleanup by @iMacTia in Scripts folder cleanup #1340
• Add Truffleruby to CI by @gogainda in Add Truffleruby to CI #1341
• Replace Hash#merge with Utils#deep_merge for connection options by @xkwd in Replace Hash#merge with Utils#deep_merge for connection options #1343
• Callable authorizers by @sled in Callable authorizers #1345
• CI: Use latest jruby-openssl release by @jbanahan in CI: Use latest jruby-openssl release #1348
• Default value for exc error by @DariuszMusielak in Default value for exc error #1351
• Don't call retry_block unless a retry is going to happen by @jrochkind in Don't call retry_block unless a retry is going to happen #1350
• Improve documentation for v2 by @iMacTia in Improve documentation for v2 #1353
• Remove default default_adapter (yes, you read that right) by @iMacTia in Remove default default_adapter (yes, you read that right) #1354
• Remove retry middleware by @iMacTia in Remove retry middleware #1356
• Remove multipart middleware and all its documentation and tests by @iMacTia in Remove multipart middleware and all its documentation and tests #1357

New Contributors

• @mollerhoj made their first contribution in Register json middleware #1331
• @jrochkind made their first contribution in Retry middleware should handle string exception class name consistently #1334
• @willianzocolau made their first contribution in Improve request info in exceptions raised by RaiseError Middleware #1335
• @mcansky made their first contribution in update changelog 1.8.0 #1338
• @xkwd made their first contribution in Replace Hash#merge with Utils#deep_merge for connection options #1343
• @sled made their first contribution in Callable authorizers #1345
• @jbanahan made their first contribution in CI: Use latest jruby-openssl release #1348
• @DariuszMusielak made their first contribution in Default value for exc error #1351

Full Changelog: v1.8.0...v2.0.0

---

This discussion was created from the release v2.0.0.

[iMacTia]

Lots of feedback on the lack of a default adapter coming from this tweet: https://twitter.com/getajobmike/status/1478586924998688768?s=20

[jrochkind]

Thanks for opening a discussion! Not sure if it's too late for this, but...

If 'faraday-net_http' were added as a dependency to faraday, and set as default_adapter again... it would assuage the concerns in that twitter thread, and also solve some challenges I'm seeing with having a gem support both faraday 1 and 2.

Personally, at the app level, I don't think it's too hard to make the minor changes needed here. But the trick is that faraday as a very popular sort of plumbing/infrastructural gem is often used in gems (and a dependency of them), and it's a bit trickier there... especially if you want the gem to support both faraday 1 and 2, which you kind of have to since a dependency tree may have multiple things depending on faraday, and synchronizing updating all of them to use faraday 2 at once isn't feasible.

Even if extracting these to seperate gems... what are the reasons to not include them as dependencies of faraday, for 2.x? Trying to think of reasons, one I can think of: I'm not sure what version requirements you'd put there... if you make it too strict (only 1.x), then you need to upgrade faraday to upgrade the extra gems, which defeats the whole purpose of extracting them. But if you make it too generous, then releases of the extra gems will break apps, also defeating the whole purpose.

So that may not work. Hmm.

There are currently challenges though for sure.

It's hard to change things like faraday which have become so pervasive at the "plumbing" level. I regret that we didn't notice this in any of the faraday 2.0 pre-releases, I'm sorry! I think we just take faraday for granted, it's been around so long, and changed so little of late.

[jrochkind]

In one reply Mattia says:

Apps will still break because of the many backwards-incompatible changes, so I’m not sure this would have made much difference?

Wait, what many backwards incompatible changes? I'm not seeing "many" listed in upgrading, only a few that it's likely won't touch many apps!

[iMacTia]

I completely understand why this have been such an issue and I agree the community is right on this one.
It won't solve the upgrading issues for everyone, but it should be enough for the majority of apps relying on the default connection stack.

I've just released Faraday 2.0.1 which brings back faraday-net_http as a dependency and sets it as the default adapter 👍
I'll move the latter point (support for both versions) on a separate comment so we can start a thread there as well.

[iMacTia]

Issues around supporting both v1.0 and v2.0 in gems that depend on Faraday have been raised here: lostisland/faraday-retry#3 (comment)

[jrochkind]

@iMacTia So... one thing I'm learning is how many nooks and crannies in my dependency tree have faraday as a dependency. So I wind up with it many times at many levels in my dependency tree.

I of course can't upgrade the app (or gem) I'm starting at to use faraday 2, until all the references in it's dependency tree allow faraday 2. (And if some support ONLY faraday 2 and otheres ONLY faraday 1... I'm stuck using old versions of the ones whose newest releases are only faraday 2, until everything supports faraday 2).

So I've been jumping around github filing issues and PRs trying to upgrade to faraday 2. Sometimes upgrading to faraday 2 has unexpected barriers, that aren't actually about faraday's own code. Like the gem I'm trying to issue a PR on has it's own dependency that doesn't allow faraday 2 yet (and it could be a dependency only in CI but not a runtime dependency in gemspec). Or the gem's master is already working on a next major release of unknown future release date, that my app may not be using anytime soon because of other issues, so the modification to gemspec to allow faraday 2 really has to be backported to the previous major release they thought they were done with. And of course many gems have busy maintainers it's hard to get the attention of for issue/review/merge. (Some of these gems can be 2, 3, or even 4 levels down in dependency tree from where i started!)

Here a couple recent examples: oauth-xx/oauth2#569 danger/danger#1349

So.. this is a lot. I understand why some gems I am coming across chose to migrate away from faraday as a reponse to the 2.0 release.

In retrospect... faraday is really a sort of "plumbing" gem. One of it's popular historical use cases is in gems, to let the client app (the thing using the gem) customize the http connection in the gem -- by choosing whatever adapter the client app wants, or adding middleware, etc. This is neat, gems can let client apps do a lot of customization on the HTTP connectons it uses via faraday API without having to invent their own architectures and APIs for this . (You're on jruby so want a different adapter? No problem! You want to configure automatic retry or client-side cache? No problem!). It's faraday providing a lot of value!

But as a result, faraday can show up at multiple points deep in a dependency tree.

And as a result of that... it's really hard to deal with any backwards incompatibilities, or any major version releases (that will not be included as allowed in specifications in existing gemspecs). Almost infeasible. Some projects will probably be stuck on faraday 1 for quite some time, others are going to be stuck between a rock and a hard place if recent releases of some dependencies require only faraday 1, but others only faraday 2.

I would urge seriously considering, like, never doing a backwards breaking change/major release again. If that's not realistic, then the serious cost of it should be considered, and hopefully such will be as rare going forward, if not more so, as it has been so far. Upping the cadence of major version releases could be really disastrous.

[pboling]

It is particularly troublesome for apps that are stuck with major foundational structural libraries that haven't been maintained in years, and which won't ever get a Faraday 2 upgrade, and which would make more sense to rip out and refactor than adopt the headaches of a many years dead library (thinking specifically of the her gem).

Practically speaking this means refactoring the entire app to rip our Her, and replace it with something else, so that other tools depending on Faraday 2 can be upgraded.

Of note - this type of pain is why I took pains to support both Faraday v1 and v2 (and even v0, as I did not want to be the obstacle preventing people from upgrading things) in the recent-ish oauth2 v2 release.

Thanks for all the work on this!

[iMacTia]

Fun fact: I actually offered to become a maintainer of her some 3 years ago to keep it updated and was brutally ignored 😅

remi/her#324 (comment)

[pboling]

The state of Her is so brutal. I am brutalized by it daily. Spyke looks like an much improved fork & rewrite, but even moving to that is hellishly difficult for us, since we... <whisper> apparently monkey patched Her all the way to the moon </whisper>. 😭

If any library authors are interested in looking at an example of a gem supporting "faraday", ">= 0.17.3", "< 3.0", it might be helpful... https://gitlab.com/oauth-xx/oauth2/-/commits/v2.0.9?search=Faraday

[iMacTia]

Thanks for the feedback @jrochkind, I can totally relate to pretty much all the points you've raised above.
Faraday is an incredibly popular gem and it's used in many, many Ruby projects for the reasons you've already highlighted above.
That unfortunately makes ANY major release a bit of an issue. We had similar problems when we released v1 a couple of years ago, and we'll have them again when we release v3.

Every time we do a major release, we try and do our best to minimize breaking changes, but the real issue is that gem owners will still apply version locking to avoid the next major version from automatically being used (and rightly so, it's a pretty common and sensible thing to do!).
The downside of that, though, is that every time we do a major release the process you're going through will need to happen: every gem that currently has the next major version locked out will need to update and release a new version.
If we did our job properly, it means that upgrade will be easy and quick, but unfortunately, it can't be avoided.

It's hard to predict when and how a v3 will happen, and we'll obviously postpone it as much as possible by avoiding backwards-incompatible changes as much as possible, but it will happen at some point.
The good news is that v2 is probably a bit harder than the past ones, but it will make future ones easier, thanks to the distributed gems structure that it introduces. A breaking change in a single adapter (e.g. excon) will not require a breaking change in Faraday anymore. This will potentially reduce the need for future major releases and their impact.

I really appreciate you taking the time to go through all these gems and trying to move things forward.
Thanks to that, there will be developers who will be able to seamlessly switch from Faraday v1 to v2 without even knowing that 👏!
We're here to help if there's anything we can do to speed up this process. Faraday v1.10.0 was just released with an out-of-the-box JSON middleware, which will make fading away from faraday_middleware easier, thus removing the double-dependency issue.

I also saw the PR in Sawyer (lostisland/sawyer#70) which happens to be a lostisland repo! So I can definitely help with that as well 😃.

Note taken on the cadence of major releases, I just wanted to provide some extra context around what has been done, what we're doing, and how we hope future major releases will be easier thanks to that ❤️ .

[iMacTia]

FYI -- lostisland/sawyer#72 🙌

[jrochkind]

The basic_auth changes are currently a pain for me. I'm getting this in my logs:

Feb 24 12:18:40 app/web.1 WARNING: `Faraday::Connection#basic_auth` is deprecated; it will be removed in version 2.0.
Feb 24 12:18:40 app/web.1 While initializing your connection, use `#request(:basic_auth, ...)` instead.
Feb 24 12:18:40 app/web.1 See https://lostisland.github.io/faraday/middleware/authentication for more usage info.

When I go to the URL it suggests for more usage info... I don't see any examples that look like #request(:basic_auth, ...) (calling a method request with a first param :basic_auth??), I'm not sure I understand the "instead" advice.

I wonder if it would be helpful if the documentation URL showed the old deprecated (and won't work in faraday 2.x) way too, so to help someone arriving there understand what code in their present codebase is the "bad" way. I'm not sure I understand the "bad" way, can you give me an example of what code that triggers this deprecation warning might look like?

I'm also not totally sure what in my app is triggering this, since there's no stack trace in my logs at present. I can do more detective work to find it. Once i do... and it's probably a gem dependency not my direct app...

If a gem dependency needs to do basic auth, and wants to support both faraday 1.x and 2.x to ease upgrade... do you have any advice for how an app can do basic auth and support both faraday 1 and 2? Some kind of conditional code?

[jrochkind]

Thanks this is helpful!

I see conn.request :basic_auth now -- I understand # for refering to methods in ruby, but just got confused somehow.

I think part of my confusion was the third way to do it:

1. conn.request :basic_auth -- the "new" faraday 1.x way to do it. I think this will not work in faraday 2.x? Will this still generate a deprecation warning in faraday 1.x, or will it not?
2. conn.request :authorization -- okay, this is the faraday 2.x way to do it. It isn't usable in faraday 1.x.
3. conn.request :basic_auth -- the "old" way to do it. This will generate a deprecation warning. And it won't work at all in faraday 2.x, correct?

I think perhaps https://lostisland.github.io/faraday/middleware/authentication should mention the conn.basic_auth way in an additional section (with clear text that this is deprecated and not to be done). And then say, if I have this right, the non-deprecated way to do that differs between faraday 1.x and 2.x.

(Although now that I've reached this point... what's the point of issuing a deprecation warning for conn.basic_auth if it will keep working in Faraday 1.x, and if the "non-deprecated" thing you replace it will will ALSO break in Faraday 2.0 -- a bit confusing! I guess the only reason to change to the new 1.x code is to avoid the deprecation warning in my logs, but the new 1.x way will... also need to be changed for faraday 2.0? The value of the deprecation is a bit unclear, when both the deprecated and undeprecated way will both keep working in faraday 1.x and both not work in faraday 2.x... why make people change to avoid the deprecation warning?)

[iMacTia]

You totally nailed it on the 3 ways of doing it. Faraday 1.x allows you all 3 ways, but will raise a deprecation warning on the "old" way. Faraday 2.x will only allow to use the :authorization middleware.

Point taken on this being a bit confusing though. Just to clarify, I recall the reason is that the decisions to remove the Connection method and to merge the 3 auth middleware were taken at a different stage, and historically we haven't introduced deprecation warning for middleware that would be removed into Faraday v2.0.

But that's meaningless, the important thing is to make the upgrade path as painless as possible.
So on top of updating the documentation as you suggested, I think the following changes would also be highly beneficial:

1. Make the "new Faraday v2.x" conn.request :authorization signature work also in 1.x. This currently doesn't work, and causes the need for an if/else if you want to support both versions.
2. Add a deprecation warning for the basic_auth and token_auth middleware in 1.x, as they won't be available in 2.x anyway.
3. For consistency, we should add deprecation warnings to all other middleware that has been removed from Faraday 2.x (e.g. multipart and retry)

I'll create a couple of issue for these points 👍

UPDATE: #1405 and #1406 have been created 🙌

[jrochkind]

Thanks for the reply! I think one thing you said was contradictory/not right:

Faraday 1.x allows you all 3 ways

It does not currently, correct? But you plan to make it so.

I think that is a good idea -- so conn.request :authorization would work in (latest) 1.x or 2.x, and would be a way to support both. I think the doc page should then outline this, say you can do conn.request :authorization to be compat with both 1.x and 2.x as long as you are using at least version X of 1.x (and say what that is in docs). Cause gems will have to require at least that X to use the new way, or keep doing it the conditional way....

Add a deprecation warning for the basic_auth and token_auth middleware in 1.x

I am honestly less sure about this -- I think the constantly changing deprecation warnings in 1.x are starting to annoy people. In this case, someone may have already changed from conn.basic_auth to conn.request :basic_auth to avoid the deprecation warning, perhaps done that only recently, and now they upgrade faraday 1.x and get another new deprecation warning for what they just moved to? I see your point about how maybe it should have been deprecated all along, but since it wasn't and we can't go back -- and with 2.0 already being out with the breakages visible -- I'm not sure how helpful it is to the community at this point to keep shuffling in new deprecation warnings.

(The copious deprecation warnings in the logs are an annoyance even when you have otherwise working code).

I think people may start to have a "wait, I have to go deal with faraday deprecation warnings again now? Didn't I just do that? Twice? Is every 1.x release going to have more deprecation warnings, even though 2.x is already out?" reaction.

For consistency, we should add deprecation warnings to all other middleware that has been removed from Faraday 2.x (e.g. multipart and retry)

I'm not sure about that either. Becuase of above. Also, to be sure, is there a non-deprecated replacement for those middleware that works in 1.x? You should never issue a deprecation warning for something that can't be replaced (without an upgrade) in my opinion.

The deprecation warnings in the logs for something that is working, and that you already know about, and that you can't easily do anything about right now -- is a real annoyance. Faraday deprecation warnings appear every time the code is called, so can easily end up kind of "spamming" logs.

[iMacTia]

It does not currently, correct? But you plan to make it so.

Yes, apologies, the fact that authorization is actually an available middleware in 1.x made me erroneously say that it suppports all 3, but in fact it's a partial support for 1.x, you're right. authorization will need to be updated so that it can be used for all auth methods as its 2.x counterpart.

I'd happily backtrack on the deprecation warnings, I thought that would be useful and something you were hoping for, but I'm glad we agree is probably unnecessary and more of a nuisance given the current stage Faraday is at. I'll be closing #1406 then 👍 !

[jrochkind]

Thanks for discussion!

Yeah, I'm figuring out what would be best in discussion by talking about it too! It's more clear to me what would have been best a year ago haha but there's no time travel, sometimes we can only figure it out by doing!

[iMacTia]

[iMacTia]

#1405 and #1406 have been created 🙌

[jrochkind]

I am hoping for some advice on how to handle :follow_redirects in a codebase that would ideally support faraday 1.0 or 2.0.

I have an old barely-maintained dependency (ruby gem oai) which depends on faraday_middleware in order to provide follow_redirects functionality.

https://github.com/code4lib/ruby-oai/blob/3cdc12d86c7d1dde00b47105a7dab6def3f6801d/lib/oai/client.rb#L97-L99

Because faraday-middleware requires faraday ~> 1.0, anything depending on oai can't upgrade to faraday 2.0. I'd like to fix this. Ideally, oai could work with either faraday 1.0 or 2.0 -- it is a very old dependency which does not have much maintenance resources, but has been just chugging along and is in use. It might be disruptive if it dropped support for faraday 1.0 (as we can see, it's quite hard to get all your indirect dependencies sufficient to allow an upgrade to 2.0, many apps are going to be in 1.0 for a while because of this).

Is there a way for a codebase to have follow_redirects functionality and work with either faraday 1.x or 2.x?

The faraday_middleware README says "All other middleware, they'll be re-released as independent gems compatible with both Faraday v1 and v2" -- has this happened for faraday-redirect? I can't find it. But it also doesn't seem to be included in faraday core, I don't see any functionality for following redirects in there.

Thanks for any advice!

Certainly one last-ditch solution would just be copying the faraday middleware code into the local codebase and just using it from there.

[olleolleolle]

@jrochkind That sounds promising!

[iMacTia]

@jrochkind it looks like you already found everything you need with @olleolleolle's help, I just wanted to chime in and confirm this is the correct way forward 👍
Middleware in faraday_middleware can be isolated into standalone gems that will support both Faraday v1 and v2, and this will allow replacing the dependency on faraday_middleware with these new gems.

Now the "problem" is that faraday_middleware has accumulated a bunch of middleware over the years, some of which were pushed by external maintainers, which then disappeared.
With these new gems, we're letting the community drive the migration from the "faraday_middleware monolith" to the new gems collection. We are here to assist, obviously, but we really encourage people to step up and take ownership of the gems that they depend on for their own work.

@tisba was kind enough to do this for faraday_middleware, but he probably doesn't currently use Faraday v1, so there was no need for him to support both. This is where I'd expect someone else from the community to come in to help adding that support, so that over time we get to migrate all the middleware into their own gems with both versions support.

For what it's worth, we're not just relying on other's help, we're also trying to help spread Faraday v2 in other ways.
I for example have been looking at the "Depended by" list of https://bestgems.org/gems/faraday and going through those gems to add support for Faraday v2. I thought that was one of the best ways to allow as many people as possible to use Faraday v2, focusing on the most popular gems. signet for example took a significant amount of time.

I think my wording on the README was misleading if it conveyed the message that the core team would be releasing all of those gems.
Our vision for Faraday is to make it more like an ecosystem, providing the tools to the community to create their own path, rather than relying on the limited resources of the core team (like the great https://github.com/lostisland/faraday-middleware-template repo).

I'm sorry if this keeps getting in your way and I want to reiterate I'm glad you've been so proactive on this.
We're here to support as always, but it's up to the community to update individual gems and projects based on their needs.

[tisba]

Hey everyone. My idea was to keep complexity low by only "officially" supporting faraday >= 2.

I now realise, that by extracting it from the [faraday_middleware](https://github.com/lostisland/faraday_middleware) gem it is a bit more difficult to keep this as clean as I'd like.

Thanks @jrochkind for pointing this out and for the PR.

[tisba]

jFYI: I pushed 0.3.0 earlier today with support for both faraday 1.x and 2.x. Thanks to @jrochkind for preparing the PR 🚀

[iMacTia]

Great work you both 👏🎉!
Thanks for contributing to the Faraday ecosystem 🙏