Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: partial @streaming blob support for Python #747

Open
wants to merge 715 commits into
base: main-1.x
Choose a base branch
from

Conversation

robin-aws
Copy link
Contributor

@robin-aws robin-aws commented Dec 12, 2024

Adds partial support for the @streaming Smithy trait on blobs, depending on a actions-and-streaming-stdlibs branch of Dafny that adds an Actions standard library, and only implemented when targeting Python so far. Event streaming is not technically supported yet because more codegen changes are necessary, but this will be a relatively small delta, as the Dafny types involved are already handled.

The motivation behind merging this incomplete work is largely to get the test model coverage into CI, so that it doesn't regress as the feature is completed, especially when Dafny verification is perturbed in future versions. The next priority will be to complete the Dafny branch and merge it to Dafny mainline.

Note that smithy-dafny will still emit a DANGER validation event saying that @streaming is not supported, which is intentional in order to guard against using the feature in production before it is fully completed: the test models exercising it explicitly suppress this event.

  • Added a dafny submodule under TestModels/dafny-dependencies/dafny tracking the actions-and-streaming-stdlibs branch, just to access the contents of Source/DafnyStandardLibraries/src/Std/Actions etc. This copy of Dafny is never built or used for any other building tasks.
  • Added a build task under StandardLibrary to build a .doo file from a subset of the Dafny standard library files above (using DafnyStandardLibraries-smithy-dafny-subset.toml). This is then used as a --library argument when verifying/translating polymorph libraries. This approach allows the standard library source to be built with different but compatible options such as --reads-clauses-on-methods.
    • This is guarded by a USE_DAFNY_STANDARD_LIBRARIES Makefile variable, since a newer Dafny is required for this to work.
  • Bumped the high watermark Dafny version tested in CI to 4.9.1, which is necessary since the standard library above uses newer features like @-attributes.
  • Modified Dafny and Python codegen to emit references to the ByteStream and RewindableByteStream types, and removed the previous workaround to first .read() all data in Python.
    • Because streams have mutable state, the Dafny code generation needs to take them into account when generating things like modifies clauses. I haven't yet implemented this, instead using a patch file to add the missing specification to the Streaming test model code.
  • Modified the aws-sdk/s3 test model and fleshed the Streaming test model out to actually test the feature.
  • Removes not-grep, since our globs are actually completely broken and the tool doesn't support our current code layout, and it was complaining about the contents of the dafny submodule. See License header scanning is broken #763.

Reviewer note: it is worth reading the contents of the pending Actions and Streams Dafny libraries, but they are still under construction and I'm not claiming they will not change before being merged. Interface feedback is welcome but is not intended to be a hard requirement in scope for this review.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@robin-aws robin-aws changed the title Robin aws/streaming test models python feat: @streaming support for Python (WIP) Jan 16, 2025
@robin-aws robin-aws changed the title feat: @streaming support for Python (WIP) feat: @streaming blob support for Python (WIP) Jan 21, 2025
@robin-aws robin-aws changed the title feat: @streaming blob support for Python (WIP) feat: partial @streaming blob support for Python Jan 21, 2025
@robin-aws robin-aws marked this pull request as ready for review January 21, 2025 18:22
@robin-aws robin-aws requested a review from a team as a code owner January 21, 2025 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants