Fix a bug with url encoded parameters in url.parse

[erikdubbelboer]

Type of change

• Bugfix

Description of change

When you pass an url with url encoded parameters to Ajax.ajax but also pass extra parameters in the data argument, such as the atomx adapter does, it will break the url.

Ajax.ajax will parse the url and format it again:

Prebid.js/src/ajax.js

Lines 78 to 82 in 6956a56

	if (method === 'GET' && data) {
	let urlInfo = parseURL(url);
	Object.assign(urlInfo.search, data);
	url = formatURL(urlInfo);
	}

But url.parse decodes the whole url without url.format encoding it again:

Prebid.js/src/url.js

Line 29 in 6956a56

parsed.href = decodeURIComponent(url);

So Ajax.ajax('http://example.com?foo=foo%26foo%3Dxxx', null, {ignore: 123}) will load the url http://example.com?foo=foo&foo=xxx&ignore=123 instead of http://example.com?foo=foo%26foo%3Dxxx&ignore=123 which is just plain wrong.

This can be fixed by not just decoding the whole url but instead decoding and encoding the separate query parameters.

This pull request right now only removes the decodeURIComponent in url.parse as that doesn't seem to break any tests. Adding decodeURIComponent and encodeURIComponent to url.parseQS and url.formatQS seems to break some tests for adapters that already depend on this broken behavior.

[erikdubbelboer]

Any updates? Right now Prebid.js is impossible to use for us and I'm pretty sure other adapters suffer from this bug as well but just don't notice it because they might not always have url encoded values in the urls they use with Ajax.ajax.

[mkendall07]

@erikdubbelboer why can't you just pass the query params into the data argument? Is it because of needing to use the same key more than once (so you need an array of key/values instead of a just plain object? )

[erikdubbelboer]

@mkalafior the base url is build by external javascript code that isn't included in Prebid.js. The only way we could fix this at the moment is to encodeURIComponent all urls we pass to Ajax.ajax which would be a super ugly fix that will break things in the future for sure.

Don't you think url.parse is really broken by always doing decodeURIComponent on the whole url?

[mkendall07]

@erikdubbelboer yes I agree it's broken performing decodeURIComponent on the whole URL. The contract on all the url functions is not clear though, if we expect to decode/encode. So changing the behavior without any such clarification doesn't seem correct to me either. I think what we need is a param that says explicitly to perform encoding/decoding on each function tbh.

[erikdubbelboer]

@mkendall07 I can modify the pull request to add an options dictionary to url.parse to change this behavior. I'll make it enabled by default like it is now. This way everything keeps working the way it is now. The Ajax.ajax just needs to pass this option from its own options down to url.parse. Does that sound acceptable?

[mkendall07]

Sure that sounds ok in the short term. We discussed this a bit earlier and might end up pulling in a lightweight ajax library to be more standardized. That might not be right away tho.

[erikdubbelboer]

I'll modify the pull request next week.

All other Ajax libraries will obviously not run decodeURIComponent on the whole URL meaning all adapters that depend on this behavior will break.

[erikdubbelboer]

@mkendall07 I changed the pull request. I'm not sure if noDecodeWholeURL is a nice name for the option. If you have any other suggestions like me know and I'll change it.

[erikdubbelboer]

Not sure if passing the whole options object here is the best or if it should extract the options that apply to url.parse. This would result in a lot more code.

[erikdubbelboer]

I left this here to show that the current url.parse is quite broken with foo being set to xxx instead of the expected bar.

[erikdubbelboer]

@mkendall07 any update? We really need this as currently our adapter is broken because if this bug.

[erikdubbelboer]

@snapwich @ndhimehta @mkendall07 anyone?

[jaiminpanchal27]

LGTM