Correctly implement credentialed CORS for the server #4513

Azeirah · 2023-12-17T20:15:51Z

Refer to #4511 for details.

This PR implements CORS for credentialed requests for the server. A credentialed request is when you send the Authorization header along with a request, some clients send an empty or placeholder API key because OpenAI requires one, whereas this server doesn't necessarily require one.

You can test that it works with this extremely simple client that implements both a regular client as well as one that sends an API key.

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">
	<title>llama.cpp server CORS</title>
</head>
<body>
	<button class="without">Send without auth</button>
	<button class="with">Send with auth</button>

	<input type="text" class="url" placeholder="localhost:8080/v1/completion">

	<pre>
		<code class="out">
			
		</code>
	</pre>

	<script type="text/javascript">
		const $out = document.querySelector(".out");

		const $btn_without = document.querySelector(".without");
		const $btn_with = document.querySelector(".with");
		const $url = document.querySelector(".url");

		$btn_with.addEventListener("click", async function () {
			const res = await fetch($url.value, {
				method: "POST",
				headers: {
					Authorization: "no key",
					"Content-Type": "application/json",
				},
				body: JSON.stringify({
					messages: [{role: "user", content: "What is the color of the sky?"}]
				})
			});
			$out.innerText = JSON.stringify(JSON.parse(await res.text()), null, 4);
		});

		$btn_without.addEventListener("click", async function () {
			const res = await fetch($url.value, {
				method: "POST",
				headers: {
					"Content-Type": "application/json",
				},
				body: JSON.stringify({
					messages: [{role: "user", content: "What is the color of the sky?"}]
				})
			});
			$out.innerText = JSON.stringify(JSON.parse(await res.text()), null, 4);
		});
	</script>
</body>
</html>

The server should still work with any other client in all scenarios.

I think it's worth making a list of clients this code has been tested against to ensure there are no regressions.

The test client in this code
CodeGPT plugin for Jetbrains
llama.cpp built-in server

Fix eventsource format

Make fschat and flask-cors optional

zeyugao and others added 18 commits July 26, 2023 15:26

Use coversation template from fastchat for api proxy

ea5a7fb

Fix eventsource format

Add docs

bee2a3d

Make fschat and flask-cors optional

Merge remote-tracking branch 'origin/master'

59484c6

Use conv.set_system_message from upstream

712c2e9

Merge remote-tracking branch 'origin/master'

6ae3702

Fix when stop in request is null

ea73dac

Merge remote-tracking branch 'elsagranger/master'

2c8e920

Merge branch 'master' of github.com:ggerganov/llama.cpp

dbdb2c1

Merge branch 'master' of github.com:ggerganov/llama.cpp

ff441c9

Merge branch 'master' of github.com:ggerganov/llama.cpp

401bada

Merge branch 'master' of github.com:ggerganov/llama.cpp

cbdfa3c

Merge branch 'master' of github.com:ggerganov/llama.cpp

0ee4ff4

Merge branch 'master' of github.com:ggerganov/llama.cpp

ab54e65

Merge branch 'master' of github.com:ggerganov/llama.cpp

6b732a0

Merge branch 'master' of github.com:ggerganov/llama.cpp

1ec3f29

Merge branch 'master' of github.com:ggerganov/llama.cpp

ff38d7f

Merge branch 'master' of github.com:ggerganov/llama.cpp

bf57173

Implement credentialed CORS according to MDN

0876952

Azeirah closed this Dec 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Correctly implement credentialed CORS for the server #4513

Correctly implement credentialed CORS for the server #4513

Azeirah commented Dec 17, 2023

Correctly implement credentialed CORS for the server #4513

Correctly implement credentialed CORS for the server #4513

Conversation

Azeirah commented Dec 17, 2023