Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

take std.http in a different direction #18955

Merged
merged 59 commits into from
Feb 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
f58c59f
std.http.Server: don't emit Server HTTP header
andrewrk Feb 11, 2024
f1cf300
std.http.Server: fix error set
andrewrk Feb 11, 2024
90bd4f2
std.http: remove the ability to heap-allocate headers
andrewrk Feb 12, 2024
b47bd03
std.http.Server: protect against zero-length chunks
andrewrk Feb 12, 2024
06d0c58
std.mem: take advantage of length-based slicing
andrewrk Feb 12, 2024
50e2a5f
std.http: remove 'done' flag
andrewrk Feb 12, 2024
00acf8a
std.http.Server: remove source code from doc comments
andrewrk Feb 12, 2024
f46447e
std.http.Client.fetch: add redirect behavior to options
andrewrk Feb 12, 2024
4d401e6
std.http: remove Headers API
andrewrk Feb 13, 2024
b6ca89f
std.http.Client: disable zstd for now
andrewrk Feb 16, 2024
99a5de9
git fetching: fix redirect handling
andrewrk Feb 16, 2024
cf4a2c4
std.http.Client.Response.ParseError: remove OutOfMemory
andrewrk Feb 16, 2024
3d61890
std: convert http trailers test to unit test
andrewrk Feb 17, 2024
d574875
Revert "std.http: remove 'done' flag"
andrewrk Feb 17, 2024
7819263
std.http: parser fixes
andrewrk Feb 17, 2024
f9dff2f
std.http: fields at the top of the struct
andrewrk Feb 17, 2024
ae630b6
std.http.Client.connect: case insensitive host comparison
andrewrk Feb 17, 2024
ddb754f
std.http: fix parsing incorrect tokenization
andrewrk Feb 17, 2024
511acc1
std.http: remove format() method of Method
andrewrk Feb 17, 2024
7036644
std.http.Client: remove advisory file lock on fetch
andrewrk Feb 17, 2024
0ddcb83
std.http.Client.fetch: remove inappropriate seek
andrewrk Feb 17, 2024
743a0c9
std.http.Client: remove bad decisions from fetch()
andrewrk Feb 17, 2024
107992d
std.Uri: refactor std.mem.Allocator -> Allocator
andrewrk Feb 17, 2024
651aa5e
std.http.Client: eliminate arena allocator usage
andrewrk Feb 17, 2024
6de8748
std.http: skip tests on wasi and single-threaded
andrewrk Feb 17, 2024
e204b2c
std.http.Client.connectUnix: handle unsupported OS at compile time
andrewrk Feb 17, 2024
63fa151
std.compress.zstandard: fix buffer sizes
dweiller Feb 15, 2024
73f6d3a
std.compress.zstd: fix decompressStreamOptions
dweiller Feb 16, 2024
ac1b957
std.compress.zstd: remove allocation from DecompressStream
dweiller Feb 15, 2024
accbba3
std.compress.zstd: disable failing wasm32 tests
dweiller Feb 17, 2024
63acc85
std.http.Client: remove invalid use of refAllDecls
andrewrk Feb 18, 2024
5c12783
std.compress.zstd: make DecompressStream options runtime
dweiller Feb 18, 2024
c44a902
fix zstd compilation errors from previous commit
andrewrk Feb 18, 2024
f1565e3
std.http.Server.accept can no longer fail from OOM
andrewrk Feb 19, 2024
6129ecd
std.net, std.http: simplify
andrewrk Feb 19, 2024
c7fc2d7
std.http.Server: move closing bool
andrewrk Feb 19, 2024
968d08a
std.http.Server.Connection: remove dead code
andrewrk Feb 19, 2024
68d3e10
full send
andrewrk Feb 19, 2024
2df3de1
std.http.Server: no more dynamic allocation
andrewrk Feb 19, 2024
9129fb2
std.ArrayList: add writerAssumeCapacity
andrewrk Feb 20, 2024
6395ba8
std.http.Server: rework the API entirely
andrewrk Feb 20, 2024
12a9e0f
std.net.listen: fix Windows API use
andrewrk Feb 21, 2024
d943ce5
std.io.Reader: add discard function
andrewrk Feb 21, 2024
a8958c9
std.net: fix std lib test regression. fixup
andrewrk Feb 21, 2024
b4b9f6a
std.http.Server: reimplement chunked uploading
andrewrk Feb 21, 2024
2e7d806
std.http.Server: fix seeing phantom request
andrewrk Feb 21, 2024
17291e0
std.ArrayList: fixedWriter
andrewrk Feb 22, 2024
5145134
update standalone http test file to new API
andrewrk Feb 22, 2024
c0d8ac8
std.http.Server: fix handling of HEAD + chunked
andrewrk Feb 22, 2024
40ed3c4
std.http.Client: add keep_alive option to fetch
andrewrk Feb 22, 2024
380916c
std.http.Server.Request.Respond: support all transfer encodings
andrewrk Feb 22, 2024
abde76a
std.http.Server: handle expect: 100-continue requests
andrewrk Feb 22, 2024
737e7be
std.http: refactor unit tests
andrewrk Feb 23, 2024
d051b13
std.http.Server: implement respondStreaming with unknown len
andrewrk Feb 23, 2024
10beb19
std.http: assert against \r\n in headers
andrewrk Feb 23, 2024
483b63d
std.http: migrate remaining test/standalone/http.zig to std lib
andrewrk Feb 23, 2024
d7ac8c8
wasi: don't try to test http
andrewrk Feb 23, 2024
5b34a1b
std.http: disable the test that was never passing on windows
andrewrk Feb 23, 2024
653d415
std.http.Server: expose arbitrary HTTP headers
andrewrk Feb 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
231 changes: 133 additions & 98 deletions lib/std/Uri.zig
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
const Uri = @This();
const std = @import("std.zig");
const testing = std.testing;
const Allocator = std.mem.Allocator;

scheme: []const u8,
user: ?[]const u8 = null,
Expand All @@ -15,15 +16,15 @@ query: ?[]const u8 = null,
fragment: ?[]const u8 = null,

/// Applies URI encoding and replaces all reserved characters with their respective %XX code.
pub fn escapeString(allocator: std.mem.Allocator, input: []const u8) error{OutOfMemory}![]u8 {
pub fn escapeString(allocator: Allocator, input: []const u8) error{OutOfMemory}![]u8 {
return escapeStringWithFn(allocator, input, isUnreserved);
}

pub fn escapePath(allocator: std.mem.Allocator, input: []const u8) error{OutOfMemory}![]u8 {
pub fn escapePath(allocator: Allocator, input: []const u8) error{OutOfMemory}![]u8 {
return escapeStringWithFn(allocator, input, isPathChar);
}

pub fn escapeQuery(allocator: std.mem.Allocator, input: []const u8) error{OutOfMemory}![]u8 {
pub fn escapeQuery(allocator: Allocator, input: []const u8) error{OutOfMemory}![]u8 {
return escapeStringWithFn(allocator, input, isQueryChar);
}

Expand All @@ -39,7 +40,7 @@ pub fn writeEscapedQuery(writer: anytype, input: []const u8) !void {
return writeEscapedStringWithFn(writer, input, isQueryChar);
}

pub fn escapeStringWithFn(allocator: std.mem.Allocator, input: []const u8, comptime keepUnescaped: fn (c: u8) bool) std.mem.Allocator.Error![]u8 {
pub fn escapeStringWithFn(allocator: Allocator, input: []const u8, comptime keepUnescaped: fn (c: u8) bool) Allocator.Error![]u8 {
var outsize: usize = 0;
for (input) |c| {
outsize += if (keepUnescaped(c)) @as(usize, 1) else 3;
Expand Down Expand Up @@ -76,7 +77,7 @@ pub fn writeEscapedStringWithFn(writer: anytype, input: []const u8, comptime kee

/// Parses a URI string and unescapes all %XX where XX is a valid hex number. Otherwise, verbatim copies
/// them to the output.
pub fn unescapeString(allocator: std.mem.Allocator, input: []const u8) error{OutOfMemory}![]u8 {
pub fn unescapeString(allocator: Allocator, input: []const u8) error{OutOfMemory}![]u8 {
var outsize: usize = 0;
var inptr: usize = 0;
while (inptr < input.len) {
Expand Down Expand Up @@ -341,7 +342,7 @@ pub fn format(
/// The return value will contain unescaped strings pointing into the
/// original `text`. Each component that is provided, will be non-`null`.
pub fn parse(text: []const u8) ParseError!Uri {
var reader = SliceReader{ .slice = text };
var reader: SliceReader = .{ .slice = text };
const scheme = reader.readWhile(isSchemeChar);

// after the scheme, a ':' must appear
Expand All @@ -358,111 +359,145 @@ pub fn parse(text: []const u8) ParseError!Uri {
return uri;
}

/// Implementation of RFC 3986, Section 5.2.4. Removes dot segments from a URI path.
///
/// `std.fs.path.resolvePosix` is not sufficient here because it may return relative paths and does not preserve trailing slashes.
fn removeDotSegments(allocator: std.mem.Allocator, paths: []const []const u8) std.mem.Allocator.Error![]const u8 {
var result = std.ArrayList(u8).init(allocator);
defer result.deinit();

for (paths) |p| {
var it = std.mem.tokenizeScalar(u8, p, '/');
while (it.next()) |component| {
if (std.mem.eql(u8, component, ".")) {
continue;
} else if (std.mem.eql(u8, component, "..")) {
if (result.items.len == 0)
continue;
pub const ResolveInplaceError = ParseError || error{OutOfMemory};

while (true) {
const ends_with_slash = result.items[result.items.len - 1] == '/';
result.items.len -= 1;
if (ends_with_slash or result.items.len == 0) break;
}
} else {
try result.ensureUnusedCapacity(1 + component.len);
result.appendAssumeCapacity('/');
result.appendSliceAssumeCapacity(component);
}
}
}
/// Resolves a URI against a base URI, conforming to RFC 3986, Section 5.
/// Copies `new` to the beginning of `aux_buf`, allowing the slices to overlap,
/// then parses `new` as a URI, and then resolves the path in place.
/// If a merge needs to take place, the newly constructed path will be stored
/// in `aux_buf` just after the copied `new`.
pub fn resolve_inplace(base: Uri, new: []const u8, aux_buf: []u8) ResolveInplaceError!Uri {
std.mem.copyBackwards(u8, aux_buf, new);
// At this point, new is an invalid pointer.
const new_mut = aux_buf[0..new.len];

const new_parsed, const has_scheme = p: {
break :p .{
parse(new_mut) catch |first_err| {
break :p .{
parseWithoutScheme(new_mut) catch return first_err,
false,
};
},
true,
};
};

// ensure a trailing slash is kept
const last_path = paths[paths.len - 1];
if (last_path.len > 0 and last_path[last_path.len - 1] == '/') {
try result.append('/');
}
// As you can see above, `new_mut` is not a const pointer.
const new_path: []u8 = @constCast(new_parsed.path);

if (has_scheme) return .{
.scheme = new_parsed.scheme,
.user = new_parsed.user,
.host = new_parsed.host,
.port = new_parsed.port,
.path = remove_dot_segments(new_path),
.query = new_parsed.query,
.fragment = new_parsed.fragment,
};

return result.toOwnedSlice();
}
if (new_parsed.host) |host| return .{
.scheme = base.scheme,
.user = new_parsed.user,
.host = host,
.port = new_parsed.port,
.path = remove_dot_segments(new_path),
.query = new_parsed.query,
.fragment = new_parsed.fragment,
};

/// Resolves a URI against a base URI, conforming to RFC 3986, Section 5.
///
/// Assumes `arena` owns all memory in `base` and `ref`. `arena` will own all memory in the returned URI.
pub fn resolve(base: Uri, ref: Uri, strict: bool, arena: std.mem.Allocator) std.mem.Allocator.Error!Uri {
var target: Uri = Uri{
.scheme = "",
.user = null,
.password = null,
.host = null,
.port = null,
.path = "",
.query = null,
.fragment = null,
const path, const query = b: {
if (new_path.len == 0)
break :b .{
base.path,
new_parsed.query orelse base.query,
};

if (new_path[0] == '/')
break :b .{
remove_dot_segments(new_path),
new_parsed.query,
};

break :b .{
try merge_paths(base.path, new_path, aux_buf[new_mut.len..]),
new_parsed.query,
};
};

if (ref.scheme.len > 0 and (strict or !std.mem.eql(u8, ref.scheme, base.scheme))) {
target.scheme = ref.scheme;
target.user = ref.user;
target.host = ref.host;
target.port = ref.port;
target.path = try removeDotSegments(arena, &.{ref.path});
target.query = ref.query;
} else {
target.scheme = base.scheme;
if (ref.host) |host| {
target.user = ref.user;
target.host = host;
target.port = ref.port;
target.path = ref.path;
target.path = try removeDotSegments(arena, &.{ref.path});
target.query = ref.query;
return .{
.scheme = base.scheme,
.user = base.user,
.host = base.host,
.port = base.port,
.path = path,
.query = query,
.fragment = new_parsed.fragment,
};
}

/// In-place implementation of RFC 3986, Section 5.2.4.
fn remove_dot_segments(path: []u8) []u8 {
var in_i: usize = 0;
var out_i: usize = 0;
while (in_i < path.len) {
if (std.mem.startsWith(u8, path[in_i..], "./")) {
in_i += 2;
} else if (std.mem.startsWith(u8, path[in_i..], "../")) {
in_i += 3;
} else if (std.mem.startsWith(u8, path[in_i..], "/./")) {
in_i += 2;
} else if (std.mem.eql(u8, path[in_i..], "/.")) {
in_i += 1;
path[in_i] = '/';
} else if (std.mem.startsWith(u8, path[in_i..], "/../")) {
in_i += 3;
while (out_i > 0) {
out_i -= 1;
if (path[out_i] == '/') break;
}
} else if (std.mem.eql(u8, path[in_i..], "/..")) {
in_i += 2;
path[in_i] = '/';
while (out_i > 0) {
out_i -= 1;
if (path[out_i] == '/') break;
}
} else if (std.mem.eql(u8, path[in_i..], ".")) {
in_i += 1;
} else if (std.mem.eql(u8, path[in_i..], "..")) {
in_i += 2;
} else {
if (ref.path.len == 0) {
target.path = base.path;
target.query = ref.query orelse base.query;
} else {
if (ref.path[0] == '/') {
target.path = try removeDotSegments(arena, &.{ref.path});
} else {
target.path = try removeDotSegments(arena, &.{ std.fs.path.dirnamePosix(base.path) orelse "", ref.path });
}
target.query = ref.query;
while (true) {
path[out_i] = path[in_i];
out_i += 1;
in_i += 1;
if (in_i >= path.len or path[in_i] == '/') break;
}

target.user = base.user;
target.host = base.host;
target.port = base.port;
}
}

target.fragment = ref.fragment;

return target;
return path[0..out_i];
}

test resolve {
const base = try parse("http://a/b/c/d;p?q");

var arena = std.heap.ArenaAllocator.init(std.testing.allocator);
defer arena.deinit();
test remove_dot_segments {
{
var buffer = "/a/b/c/./../../g".*;
try std.testing.expectEqualStrings("/a/g", remove_dot_segments(&buffer));
}
}

try std.testing.expectEqualDeep(try parse("http://a/b/c/blog/"), try base.resolve(try parseWithoutScheme("blog/"), true, arena.allocator()));
try std.testing.expectEqualDeep(try parse("http://a/b/c/blog/?k"), try base.resolve(try parseWithoutScheme("blog/?k"), true, arena.allocator()));
try std.testing.expectEqualDeep(try parse("http://a/b/blog/"), try base.resolve(try parseWithoutScheme("../blog/"), true, arena.allocator()));
try std.testing.expectEqualDeep(try parse("http://a/b/blog"), try base.resolve(try parseWithoutScheme("../blog"), true, arena.allocator()));
try std.testing.expectEqualDeep(try parse("http://e"), try base.resolve(try parseWithoutScheme("//e"), true, arena.allocator()));
try std.testing.expectEqualDeep(try parse("https://a:1/"), try base.resolve(try parse("https://a:1/"), true, arena.allocator()));
/// 5.2.3. Merge Paths
fn merge_paths(base: []const u8, new: []u8, aux: []u8) error{OutOfMemory}![]u8 {
if (aux.len < base.len + 1 + new.len) return error.OutOfMemory;
if (base.len == 0) {
aux[0] = '/';
@memcpy(aux[1..][0..new.len], new);
return remove_dot_segments(aux[0 .. new.len + 1]);
}
const pos = std.mem.lastIndexOfScalar(u8, base, '/') orelse return remove_dot_segments(new);
@memcpy(aux[0 .. pos + 1], base[0 .. pos + 1]);
@memcpy(aux[pos + 1 ..][0..new.len], new);
return remove_dot_segments(aux[0 .. pos + 1 + new.len]);
}

const SliceReader = struct {
Expand Down
23 changes: 21 additions & 2 deletions lib/std/array_list.zig
Original file line number Diff line number Diff line change
Expand Up @@ -937,14 +937,33 @@ pub fn ArrayListAlignedUnmanaged(comptime T: type, comptime alignment: ?u29) typ
return .{ .context = .{ .self = self, .allocator = allocator } };
}

/// Same as `append` except it returns the number of bytes written, which is always the same
/// as `m.len`. The purpose of this function existing is to match `std.io.Writer` API.
/// Same as `append` except it returns the number of bytes written,
/// which is always the same as `m.len`. The purpose of this function
/// existing is to match `std.io.Writer` API.
/// Invalidates element pointers if additional memory is needed.
fn appendWrite(context: WriterContext, m: []const u8) Allocator.Error!usize {
try context.self.appendSlice(context.allocator, m);
return m.len;
}

pub const FixedWriter = std.io.Writer(*Self, Allocator.Error, appendWriteFixed);

/// Initializes a Writer which will append to the list but will return
/// `error.OutOfMemory` rather than increasing capacity.
pub fn fixedWriter(self: *Self) FixedWriter {
return .{ .context = self };
}

/// The purpose of this function existing is to match `std.io.Writer` API.
fn appendWriteFixed(self: *Self, m: []const u8) error{OutOfMemory}!usize {
const available_capacity = self.capacity - self.items.len;
if (m.len > available_capacity)
return error.OutOfMemory;

self.appendSliceAssumeCapacity(m);
return m.len;
}

/// Append a value to the list `n` times.
/// Allocates more memory as necessary.
/// Invalidates element pointers if additional memory is needed.
Expand Down
Loading
Loading