Drupal Console installation fails frequently (at least on Drupal VM)

[geerlingguy]

Issue title

[ installation ] Installation frequently fails with 500 server error

Problem/Motivation

During routine CI tests for Drupal VM, as well as installations on individuals' computers quite frequently, I get a 500 error when calling https://drupalconsole.com/installer

How to reproduce

1. Kick off a few Drupal VM instance installations within a minute of each other.
2. Almost every time, the 3rd or 4th install will fail with:

TASK [geerlingguy.drupal-console : Install Drupal Console.] ********************
fatal: [localhost]: FAILED! => {"changed": false, "dest": "/usr/local/bin/drupal", "failed": true, "msg": "Request failed", "response": "HTTP Error 500: Internal Server Error", "state": "absent", "status_code": 500, "url": "https://drupalconsole.com/installer"}

More succinctly: HTTP Error 500: Internal Server Error

(From https://travis-ci.org/geerlingguy/drupal-vm/jobs/211786816#L1274)

The task in Ansible is basically calling a curl on that URL, trying to download the resultant output to /usr/local/bin/drupal.

Solution

It's possible we could maybe adapt the role Drupal VM uses to use Composer instead, but I wonder if we'd run into any similar issues.

Someone had mentioned, though, that Drupal Console might be hitting GitHub API rate limits?

[jmolivas]

@geerlingguy yes, we had issues with the github limit. I will investigate how to fix this issue.

[jmolivas]

@geerlingguy any hint how to avoid the github limit restriction? I will try to do some investigation this weekend and next week during MidCamp.

[geerlingguy]

@jmolivas - If you don't currently use OAuth to authenticate to the API, I think the rate limit is very low, like 60 requests per hour. If you use OAuth, then the limit is bumped to 5000/hour:

• https://developer.github.com/v3/rate_limit/
• https://developer.github.com/guides/getting-started/#authentication

[geerlingguy]

It looks like you may need to make some changes to:

• https://github.com/hechoendrupal/drupalconsole.com/blob/ef9648d5122e7d4aa143016eef7df2e23a0dab74/modules/custom/github_data/src/GitHubDataService.php
• https://github.com/hechoendrupal/drupalconsole.com/blob/ef9648d5122e7d4aa143016eef7df2e23a0dab74/modules/custom/github_data/src/Controller/InstallerController.php

Specifically, right now, it seems you query the GitHub API for the download, then fetch the download through drupalcomposer.org. Two options would be to either cache the entire download for something like an hour (though this could lead to annoying behavior when releasing new versions), or authenticate that request to the GitHub API. It doesn't look like there's any authentication currently.

[neclimdul]

Not familiar with how DrupalVM works to install stuff but Composer already does several things to work around the rate limiting. You might use it to pull it down or look at how it works to work around the problem.

composer/composer#4884

[geerlingguy]

Hitting this a lot today as well (possibly due to people setting up dev environments at DrupalCon...): when I visit the installer page manually, I get:

Github\Exception\RuntimeException: You have reached GitHub hour limit! Actual limit is: 60 in Github\HttpClient\HttpClient->request() (line 145 of vendor/knplabs/github-api/lib/Github/HttpClient/HttpClient.php).

[oxyc]

Another solution is to enable caching for \Github\Client. This should add If-Not-Modified-Since headers to the requests and according to this comment 304 responses do not count towards your rate limit.

[hjuarez20]

This is no longer present using v1.9.0

[bits-of-static]

Seems to be a topic again? I was getting a 500 server error since yesterday...

[gavett]

Seems to be a topic again? I was getting a 500 server error since yesterday...

I am also getting a 500 error since yesterday.

[neclimdul]

@gavett @bits-of-static I don't believe there was ever really any issue with drupal-console but with drupal-vm's travis jobs that was failing when it hit the drupal-console part.

If you're running into a similar problem I'd suggest looking into composer's token documentation
https://getcomposer.org/doc/06-config.md#github-oauth
PreviousNext also seems to have a pretty good blog post on how to set this up.
https://www.previousnext.com.au/blog/managing-composer-github-access-personal-access-tokens

[gavett]

@gavett @bits-of-static I don't believe there was ever really any issue with drupal-console but with drupal-vm's travis jobs that was failing when it hit the drupal-console part.

If you're running into a similar problem I'd suggest looking into composer's token documentation
https://getcomposer.org/doc/06-config.md#github-oauth
PreviousNext also seems to have a pretty good blog post on how to set this up.
https://www.previousnext.com.au/blog/managing-composer-github-access-personal-access-tokens

Thank you for the suggestion, I will try it. It does look like others are still having the issue as well: #4262

[bits-of-static]

I have followed the instructions, without success.

To make sure I have all the necessary rights, also created a full-access oauth key.
I think the playbook runs as root. so I also created /root/.composer/auth.json (had to manually, cp+chown from /home/vagrant/.composer since the auth.json file did not appear after sudo composer config -g github-oauth.github.com XXXX)

I was not surprised about the lack of success, since my https://github.com/settings/tokens page, reported the token as never used. Any suggestions here?

P.S.: on the https://www.previousnext.com.au/blog/managing-composer-github-access-personal-access-tokens blog, the following question was asked:

but how do you specify the username? after doing composer config, the composer update still ask for a username

Which could be a reason why the token does not get used. DrupalVM installation is non interactive after all.

[bits-of-static]

Neither does setting composer_github_oauth_token variable in the ansible configuration yaml lead to the token being actually used.

[bits-of-static]

My current workaround:
modify the corresponding provisioning\roles\geerlingguy.drupal-console\tasks\main.yml role in DrupalVM to

- name: Install Drupal Console.
  shell: >
    composer global require drupal/console && cp $(composer global config home)/vendor/bin/drupal  {{ drupal_console_path }}