PyUP fails to get file with '{' in the name

[browniebroke]

We use PyUP to keep cookiecutter-django up to date. It's a cookiecutter template to generate production-ready Django project.

As with many cookiecutter templates, it includes some folder which are placeholders for the generated project. That's the root of the source code, under which live the requirements files we want to update.

We've received less update recently, and when I checked our dashboard, I realised the files under the placeholder folder {{cookiecutter.project_slug}} where ignored by the bot.

I've installed the bot in a local virtual environment and ran it with debug logs enabled, which pointed at:

INFO:pyup.bot:Adding requirement file at {{cookiecutter.project_slug}}/requirements/local.txt
INFO:pyup.providers.github:Getting file at {{cookiecutter.project_slug}}/requirements/local.txt for branch master
DEBUG:urllib3.connectionpool:https://api.github.com:443 "GET /repos/pydanny/cookiecutter-django/contents/%257B%257Bcookiecutter.project_slug%257D%257D/requirements/local.txt?ref=master HTTP/1.1" 404 None
DEBUG:github.Requester:GET https://api.github.com/repos/pydanny/cookiecutter-django/contents/%257B%257Bcookiecutter.project_slug%257D%257D/requirements/local.txt?ref=master {'Authorization': 'token (oauth token removed)', 'User-Agent': 'PyGithub/Python'} None ==> 404 {'server': 'GitHub.com', 'date': 'Tue, 19 Mar 2019 11:24:41 GMT', 'content-type': 'application/json; charset=utf-8', 'transfer-encoding': 'chunked', 'status': '404 Not Found', 'x-ratelimit-limit': '5000', 'x-ratelimit-remaining': '4966', 'x-ratelimit-reset': '1552998281', 'x-oauth-scopes': 'repo, user:email', 'x-accepted-oauth-scopes': '', 'x-github-media-type': 'github.v3; format=json', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type', 'access-control-allow-origin': '*', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', 'x-frame-options': 'deny', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', 'content-security-policy': "default-src 'none'", 'content-encoding': 'gzip', 'x-github-request-id': 'C530:21BA:2A86CA3:56F6B6E:5C90D179'} {"message":"Not Found","documentation_url":"https://developer.github.com/v3/repos/contents/#get-contents"}
WARNING:pyup.providers.github:Unable to get {{cookiecutter.project_slug}}/requirements/local.txt on pydanny/cookiecutter-django

These loglines directed me at where the request is made:

pyup/pyup/providers/github.py

Lines 76 to 86 in b20fa88

	def get_file(self, repo, path, branch):
	logger.info("Getting file at {} for branch {}".format(path, branch))
	try:
	contentfile = repo.get_contents(quote(path), ref=branch)
	return contentfile.decoded_content.decode("utf-8"), contentfile
	except GithubException:
	logger.warning("Unable to get {path} on {repo}".format(
	path=path,
	repo=repo.full_name,
	))
	return None, None

I tried to do a more minimal test case by doing it manually, and it seems that some characters ({ and } at least) are quoted, causing a 404:

import requests
from urllib.parse import quote

base_url = 'https://api.github.com'
path = '/repos/pydanny/cookiecutter-django/contents/{{cookiecutter.project_slug}}/requirements/production.txt?ref=master'

requests.get(f'{base_url}{quote(path)}')  # Return a 404 not found

If I remove the call to quote(), it works:

requests.get(f'{base_url}{path}')  # Return a 200 response OK

I'm using Python 3.6 and PyUP installed from github master.

(When installing from PyPI version 1.0.2, I'm getting an extra / at the beginning of the the file path which causes a Github error, but this is now fixed in master)

[Jwomers]

@browniebroke interesting. I'm not sure if this is related, but PyUp updated the version of PyGitHub in the latest release, who in turn updated all their paths to remove the leading slashes. I had to update a bunch of places to remove leading slash being passed to get_file and a couple of other functions that use PyGithub.

[jayfk]

This could be the problem, yeah. I've quoted all file paths on pyup's side on purpose as it wasn't working when I initially tested it with Cookiecutter Django.

2fb7c01

[rafaelpivato]

Thanks for the excellent report, @browniebroke

[browniebroke]

Wow I nearly forgot about this one 😄