Brew installed version doesn't work with plugins

[rshurts]

Brew is listed as the recommended way of installing httpie on macOS. However, it doesn't work with auth plugins.

For example, if you pip3 install requests-hawk and then run http --help hawk will not show as an auth type. If you pip3 install httpie-oauth it will install httpie via pip as a dependency and overwrite the brew installed link in /usr/local/bin/http and now all the plugins will show because it isn't using the brew installed version.

I suggest changing the documentation to read pip3 install httpie as the recommended method of installing on macOS.

[jkbrzt]

@rshurts you're right. Plugin installation for Homebrew-based installs doesn't work out of the box.

There are advantages to keeping Homebrew as the recommended method, though. It makes updates easy and the main package can depend on Python 3 (improved SSL support, etc.).

The ideal world solution would be to have all plugins available as Homebrew packages (brew install httpie-oauth) . But that would mean extra work for maintainers so not all plugins would end up in Homebrew.

Using pip to install plugins looks like the easier way to go. To make it work with Hombrew, it would mean instructing pip to use (currently) /usr/local/Cellar/httpie/0.9.8_2/libexec/lib/python3.6/site-packages as the installation directory.

But plugins installed there won't survive HTTPie upgrades.

Perhaps it would be possible to tweak the Homebrew formula to makeallow HTTPie load installed plugins from, for example, the Homebrew Python 3 site-packages (not sure if it's permanent). Or from another location.

…or some sort of custom plugin installer.

[rshurts]

In the meantime, maybe a note in the documentation around plugins saying "brew doesn't work with plugins and to pip install httpie for plugin functionality" would save some headaches.

[aheissenberger]

There is also a problem with the read rights when user has no admin rights:

$ http --help
Traceback (most recent call last):
  File "/usr/local/bin/http", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3123, in <module>
    @_call_aside
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3107, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3136, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 569, in _build_master
    ws = cls()
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 562, in __init__
    self.add_entry(entry)
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 618, in add_entry
    for dist in find_distributions(entry, True):
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1965, in find_on_path
    for dist in factory(fullpath):
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2035, in distributions_from_metadata
    root, entry, metadata, precedence=DEVELOP_DIST,
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2492, in from_location
    py_version=py_version, platform=platform, **kw
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2867, in _reload_version
    md_version = _version_from_file(self._get_metadata(self.PKG_INFO))
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2456, in _version_from_file
    line = next(iter(version_lines), '')
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2646, in _get_metadata
    for line in self.get_metadata_lines(name):
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1411, in get_metadata_lines
    return yield_lines(self.get_metadata(name))
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1407, in get_metadata
    value = self._get(self._fn(self.egg_info, name))
  File "/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1519, in _get
    with open(path, 'rb') as stream:
PermissionError: [Errno 13] Permission denied: '/usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/idna-2.7-py3.7.egg-info/PKG-INFO'

I was able to fix it with

chmod +r /usr/local/Cellar/httpie/1.0.0/libexec/lib/python3.7/site-packages/idna-2.7-py3.7.egg-info/PKG-INFO

but switched to pip3 install httpie because of it made it easier to install plugins.

[sensibleish]

Since I upgraded to 10.14 Mojave I'm having the permissions error mentioned by @aheissenberger above. I have the same setup where I install homebrew using a local admin account but do normal operation in a standard account. I'm not a Python person and don't understand this discussion of plugins. I do not recall trying to add anything httpie beyond whatever comes as part of the homebrew formula. pip3 list only shows 3 items which presumably were part of the initial Python bundle.

Is this fundamentally a Python3 problem (at least, Python as installed via homebrew)? Should I be reporting it to the maintainers?

[nicbet]

For what it's worth and perhaps other users googling for an answer, installing a plugin worked fine for me like this:

pip3 install -t /usr/local/Cellar/httpie/2.0.0/libexec/lib/python3.8/site-packages httpie-jwt-auth

Thanks #566 (comment)

[matthew16550]

Just thinking out loud here, I haven't actually tried this.

Perhaps at runtime we add ~/.httpie/plugins/ to the search path (if it exists) and recommend Mac users do this:

pip install -t ~/.httpie/plugins httpie-jwt-auth

Suspect it might work fine for 80-90% of the use cases but no idea what might be in that last 10% !

[BoboTiG]

It is preferable rather my proposal on #1108 IMO.

Still:

• What would happen if someone installed a plugin in the folder using a different Python version (like Python 2.7) when HTTPie is running Python 3.9?
• Plugin requires HTTPie to be installed, so they install it, making the folder bigger (that's an issue on plugin themselves and not really a problem I guess), and having 2 httpie modules available (that may be a problem).

All those solutions are not ideal, I'll see how other projects are doing. If you have examples, I would be glad to check them :)

[matthew16550]

Plugin requires HTTPie to be installed, so they install it, making the folder bigger

Good point, my random sampling of 4 plugins finds they all require httpie as an install dependency.

[matthew16550]

I've done a little work on Nikola, it manages plugin installs itself. Running nikola plugin -i foo will download and extract a foo.zip, then run pip as a subprocess to install any dependencies.

Maybe we could borrow the second half of that idea and do http --install-plugin foo to launch pip as a subprocess with the right settings to install the plugin next to the httpie module so in Homebrew installs it would end up somewhere like /usr/local/Cellar/httpie/2.3.0/libexec/lib/python3.9/site-packages/foo.

Pros

• Plugins will be installed using the same version of Python that Httpie is using.
• No duplicate installation of httpie.
• Does not add plugins to a global dir (e.g. /usr/local/lib/python3.7/site-packages).
• No worrying about which of the many pips on my machine I should use to install the plugin.

Cons

• I think plugins will need to be manually reinstalled every time Homebrew updates Httpie to a new version.
• Needing to reinstall the plugins will be quite annoying!

Kind of suspect it would go against Homebrew philosophy if we modify a dir managed by Homebrew in this way but it's difficult to see a clean solution to all the complications here.

I'm not convinced this is a good idea but maybe it's a step nearer to one.

[matthew16550]

Another example: For JMeter, this blog recommends manually copying plugins into the Homebrew dir.

[isidentical]

From an outsiders look, this issue seem like a problem for the people who are actually maintaing plugins rather than the httpie itself. If an X tool is installed through Y platform, then it is fair to expect to get all the plugins to be installed in the same manner. So if the plugin maintainers also package their plugins for brew to support those users, that would be the general and (somewhat) unconvient solution (which is what we currently have with all those python-<xxx> packages on distro indexes).

Possible solutions

If we turn the other way around, and try to offer a more user friendly solution on the httpie side as far as I can tell there are 2 options that I can think of:

--interpreter (less human friendly & more error prone)

Since we don't have any actual chance of knowing what is the primary python environment they have installed the plugin (we can only assume it is the global one, or whatever python resolves [which might python2 or python3] on the current PATH), we need an explicit way for the users to pass a Python environment.

The general solution that I've seen (e.g on pyperf/pyperformance) for this sort of problem is that you pass an --interpreter option with the interpreter you want python3 or .venv/bin/python and it resolves that name and find's the executable (and then retrieves the site-packages to search through). Which is somewhat esoteric and very unfriendly. Though it might became a little bit less unfriendly by allowing this to be a global option:

[~/.httpie]

{
    "plugins": {
        "interpreter": "C:\Users\Binary\python.exe"
    }
}

2 advantages over directly pointing to site-packages:

• Knowing where all 3rd party packages reside in is not a common knowledge, and it's location might change depending on the environment (distro/os).
• Having the interpreter allows doing more robust version checks (e.g ensuring interpreter.version_info == sys.version_info)

Advantages over #1108:

• Supports virtual envs
• Explicitl version checks
• Customization on the Python interpreter itself

The implementation would be simple; and works by firing the new python interpreter, retrieving the location for the site-packages and temporarily include them on the entry-point search.

httpie plugin (more user friendly)

A more user friendlier, but rather complex solution is to httpie to manage it's own plugins. It will be simply a wrapper around the pip, and manage a custom plugin-base on it's own. This can either install plugins globally to the httpie environment (so they be accessed outside of httpie, e.g requests-hawk) or it can install them to an isolated place so the normal site package loaders don't have access to them. (both are fine options IMO)

Being able to directly install plugins is something that this interface can delegate to pip under the hood (which is guaranteed to be bundled from 3.4+, but distros like debian is a bit problematic about it [can be solvable by explicitly depending on python-pip]).

Example scenerio;

$ httpie plugins install httpie-auth
Installed httpie-auth (2.3.0)
$ httpie plugins list
httpie-auth (2.3.0): <short desc>
httpie-xxx (2.3.0): <short desc>
$ httpie plugins upgrade --all
Upgraded httpie-auth (2.4.0)
Upgraded httpie-xxx (1.1.5)
$ httpie plugins uninstall httpie-auth
Removed httpie-auth

Advantages over the --interpreter:

• Much more easier for everyone to understand, has a better UI.
• Plugin loading errors (e.g due to python version mismatch, or any other bug) can be suppressed and delegated to user about what they can do (e.g try httpie plugins upgrade)
• Can be extended further to enable / disable specific plugins
• More isolated management

[matthew16550]

@isidentical I like the httpie plugins idea 👍.

Perhaps the default target dir should be next to the config.json file e.g. ~/.config/httpie/plugins/. And allow an option in config.json to specify a different dir.

If we are delegating to pip then -t might already be supported to specify a different dir on the command line. That seems like an advanced use case, but it would be available if anyone needed it.

Should it be plugins or --plugins (or --plugin!), what would the http --help look like?

[isidentical]

Perhaps the default target dir should be next to the config.json file e.g. ~/.config/httpie/plugins/. And allow an option in config.json to specify a different dir.

This would be a bit more complicated, considering there might be multiple httpie installations and they should not share the same set of plugins. One way to do this is ~/.config/httpie/plugins/<md5(python-identifier)>/..., so they would be separated accross different installations.

If we are delegating to pip then -t might already be supported to specify a different dir on the command line. That seems like an advanced use case, but it would be available if anyone needed it.

Indeed.

Should it be plugins or --plugins (or --plugin!), what would the http --help look like?

I do not believe --plugins make sense, since this has nothing to do with how the requests are sent or how stuff is loaded. It is also going to have custom actions, so I believe it should be done as a sub-command.

[jkbrzt]

Managing plugins through an HTTPie-provided command looks like the best solution all things considered.

---

httpie command

Currently, the HTTPie package doesn’t come with a command called httpie (only http and https).

At one point, I was going to add it for sessions management but then decided against it (5cc5b13) to keep the project simple.

Now, however, with this plugin management use case (and possible upcoming integration with our new companion web & desktop app that we’re working on) it makes sense to revisit the idea.

A new command is much cleaner than trying to enrich the existing ones, where we already have quite complex args parsing logic due to our "DSL", and as @isidentical pointed out, it doesn’t really belong there anyway.

The new command should only be used for management, not for making requests. When invoked without any args or with args that are invalid (or maybe even better — they look like the user is trying to make a request), we should instruct the user to use http [args] or https [args] (this will actually improve the onboarding experience as well because the project/command name mismatch can be confusing and some users will inevitably type httpie post-installation, so this will give them a useful hint).

---

So, with a custom plugin install command, we can continue to use pip for plugins + deps installation (under the hood) and pkg_resources for installed plugin lookup and loading.

Existing plugins won’t require any changes except for updating their installation instructions. That will be optional since I think we should continue to load plugins from the main site-packages as well.

I really like the idea of generating a site-packages-like directory for plugins for each interpreter in ~/.config/httpie/plugins/<md5(python-identifier)>/.... The isolation is important to keep httpie’s development simple, among other things.

But I think a Python installation's identity shouldn’t change when it’s just upgraded to a new version (e.g. Python 3.6.10 → 3.6.11). Because without that, the user will lose plugins after each Python upgrade. Not sure how to approach that, though.

We could also consider keeping the desired plugin list in the config file and allowing the user to quickly re-install the missing ones.

Some more thoughts:

• Many plugins depend on HTTPie itself. I think we should prevent its installation into the plugins directory, or uninstall it afterward. We should probably also validate if there’s some specific version requirement.
• As mentioned above, I think we should continue to support the existing global site-packages installation location.
• Then we should gracefully handle multiple plugins installation (define lookup order).
• The plugins directory should only be used to find the entry points for plugins (and to load them and their deps), but we should avoid anything else installed there to be imported by httpie (e.g., if requests are installed there it shouldn’t be touched by httpie).
• Later, we could also consider providing httpie plugins search, etc. (plugin names correspond to their PyPi names).
• Will this cause any issues for Snap and other installation methods?

[isidentical]

Existing plugins won’t require any changes except for updating their installation instructions. That will be optional since I think we should continue to load plugins from the main site-packages as well.

Indeed. For regular installations, both options should work (pip + httpie plugins install) with httpie plugins install being the preferred one on the documentation.

But I think a Python installation's identity shouldn’t change when it’s just upgraded to a new version (e.g. Python 3.6.10 → 3.6.11). Because without that, the user will lose plugins after each Python upgrade. Not sure how to approach that, though.

What I meant by md5(python-identifier) was not exactly the python version, but rather the md5() of the path to the site-packages, or to the python interpreter (sys.executable). There are 2 reasons for that:

• There might be multiple httpie's installed accross multiple virtual environments with the same python version but different set of plugins, this way since they will all have different paths to their executables the python-identifier will be different and the isolation will stay in place
• We will mirror the python upgrades as they are happening. For example if a minor version bump (3.6.10 -> 3.6.11) does preserve the site-packages, so do we. But for major bumps, we will change it. (This should happen rarely, even with PEP 602 it takes about 1.5/2 years for built packages to bump their interpreter's major version).

Many plugins depend on HTTPie itself. I think we should prevent its installation into the plugins directory, or uninstall it afterward. We should probably also validate if there’s some specific version requirement.

Yep, this will be taken care of. Since httpie itself is actually a dependency of httpie (as a package), we will already have it and I assume it should be possible for pip to recognize it as present so it won't bother installing it. Since it is also recognizing it, and it's metadata pip will find the correct version of the plugin that fits all dependency constraints (e.g httpie>=current_version).

As mentioned above, I think we should continue to support the existing global site-packages installation location.

Agreed.

Then we should gracefully handle multiple plugins installation (define lookup order).

What do you mean by define lookup order? If it is for the installation, pip should do the ordering of dependency graph, but if it is something else not sure.

The plugins directory should only be used to find the entry points for plugins (and to load them and their deps), but we should avoid anything else installed there to be imported by httpie (e.g., if requests are installed there it shouldn’t be touched by httpie).

As I mentioned in the proposal, we will temporarily make the pkg_resources to recognize the plugins directory as a location for site packages and remove it afterwards. It should not affect anything in terms of global state / other imports, only plugins.

Later, we could also consider providing httpie plugins search, etc. (plugin names correspond to their PyPi names).

Makes sense (as well as plugins enable/plugins disable).

Will this cause any issues for Snap and other installation methods?

One important thing to note here is that, we will need to have write access to the following httpie directory. If I am not mistaken, the default home interface for snap does not support dot-files (hidden):

The home interface allows access to non-hidden files owned by the user in the user’s home ($HOME) directory where a user normally stores their personal files and documents.

(from https://snapcraft.io/docs/home-interface)

There seems to be some discussions with workarounds, https://forum.snapcraft.io/t/accessing-xdg-config-home-of-the-host/9780/10, but I am not reliable are them (needs further research).

[jkbrzt]

• We will mirror the python upgrades as they are happening. For example if a minor version bump (3.6.10 -> 3.6.11) does preserve the site-packages, so do we. But for major bumps, we will change it. (This should happen rarely, even with PEP 602 it takes about 1.5/2 years for built packages to bump their interpreter's major version).

Okay, that’s great 👍🏻

[isidentical]

I focused on this today and seems like it is finally working (install/uninstall/list). Here is the draft branch to give an impression of the implementation, though it is nowhere near production-ready (in terms of UI): https://github.com/httpie/httpie/compare/master...isidentical:httpie-plugins?expand=1. There were a couple of caveats (e.g pip doesn't support uninstallation from a custom scheme etc) but they all seem to work now. The next steps would be creating a proper UI for this (especially in terms of error handling), and figuring out a way to write tests.

[isidentical]

We have finally landed httpie plugins interface, though the initial QA on brew is not yet completed. I'll probably try to complete it within today or tomorrow and then close this ticket.

[isidentical]

🎉 This feature is now publicly available in HTTPie for Terminal 3.0:

👉🏻 https://httpie.io/docs/cli/plugin-manager