Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve schema limits #1157

Merged
merged 8 commits into from
Aug 24, 2023
Merged

Improve schema limits #1157

merged 8 commits into from
Aug 24, 2023

Conversation

jshearer
Copy link
Contributor

@jshearer jshearer commented Aug 22, 2023

Description:

  • Replace per-subschema field limit with a global complexity limit, currently set at 1000 total fields.
  • Add a --flow.enable-schema-inference flag to flowctl-go serve consumer to enable schema inference for capture tasks

This change is Reviewable

Copy link
Member

@jgraettinger jgraettinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments, looking good!

crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
Copy link
Member

@jgraettinger jgraettinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % final comments!

crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
for prop in slf.object.properties.iter_mut() {
enforce_field_count_limits(&mut prop.shape, loc.push_prop(&prop.name))
shape_to_squash.array.additional_items =
squash_addl_properties(shape_to_squash.array.additional_items);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't you just set additional_items and additional_properties to None ?

nit: if you can't and must keep this, then squash_addl_properties => squash_addl ? sometimes item, sometimes property

crates/doc/src/shape/limits.rs Outdated Show resolved Hide resolved
@jshearer jshearer merged commit ed8ee5f into master Aug 24, 2023
4 checks passed
@jgraettinger jgraettinger deleted the fix/improve_schema_limits branch August 24, 2023 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants