-
Notifications
You must be signed in to change notification settings - Fork 520
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for the YAML 1.2 Core and JSON schemas [Take 2] #555
base: main
Are you sure you want to change the base?
Conversation
aba0cf5
to
11f107a
Compare
11f107a
to
ecf1ed4
Compare
ecf1ed4
to
33cdeed
Compare
33cdeed
to
31cd170
Compare
1b2f534
to
8f3f979
Compare
any updates on this? |
@perlpunk anything we can do to help push this along? |
I wonder when this will be merged... |
@ssbarnea I'll bring up the task of putting out a new pyyaml release, with the release team. I suspect this would be merged into the next release. |
@ingydotnet 6.0.1 is out, can we have an eta on the next major/minor release that'll merge this pr? |
We've discussed a November release. |
@ingydotnet Is that November target still realistic? Based on the progress so far I becoming quite pessimistic... Now working on replacing most of pyyaml use with ruamel.yaml in ansible-lint. What worries me the most is that the change was never merged as such code change is also expected to need some time to mature before the release, preferably involving some pre-releases. Still, until is merged to development branch, there is no hope that code would be improved, and instead more likely to be affected by code rot. |
@ssbarnea The first week of November is the time that we all agreed to look into putting out the next release. That is when we expect to look at tickets like this. |
@ingydotnet May I ask for clarification on the schedule?
|
b88bd44
to
9979971
Compare
9979971
to
346c1a4
Compare
so that other classes inheriting from it can use them * Move methods from SafeConstructor to BaseConstructor * Move methods from SafeRepresenter to BaseRepresenter
More and more YAML libraries are implementing YAML 1.2, either new ones simply starting with 1.2 or older ones adding support for it. While also the syntax was changed in YAML 1.2, this pull request is about the schema changes. As an example, in 1.1, Y, yes, NO, on etc. are resolved as booleans in 1.1. This sounds convenient, but also means that all these 22 different strings must be quoted if they are not meant as booleans. A very common obstacle is the country code for Norway, NO ("Norway Problem"). In YAML 1.2 this was improved by reducing the list of boolean representations. Also other types have been improved. The 1.1 regular expression for float allows . and ._ as floats, although there isn't a single digit in these strings. While the 1.2 Core Schema, the recommended default for 1.2, still allows a few variations (true, True and TRUE, etc.), the 1.2 JSON Schema is there to match JSON behaviour regarding types, so it allows only true and false. Note that this implementation of the YAML JSON Schema might not be exactly like the spec defines it (all plain scalars not resolving to numbers, null or booleans would be an error). Short usage example: class MyCoreLoader(yaml.BaseLoader): pass class MyCoreDumper(yaml.CommonDumper): pass MyCoreLoader.init_tags('core') MyCoreDumper.init_tags('core') data = yaml.load(input, Loader=MyCoreLoader) output = yaml.dump(data, Dumper=MyCoreDumper) Detailed example code to play with: import yaml class MyCoreLoader(yaml.BaseLoader): pass MyCoreLoader.init_tags('core') class MyJSONLoader(yaml.BaseLoader): pass MyJSONLoader.init_tags('json') class MyCoreDumper(yaml.CommonDumper): pass MyCoreDumper.init_tags('core') class MyJSONDumper(yaml.CommonDumper): pass MyJSONDumper.init_tags('json') input = """ - TRUE - yes - ~ - true #- .inf #- 23 #- #empty #- !!str #empty #- 010 #- 0o10 #- 0b100 #- 0x20 #- -0x20 #- 1_000 #- 3:14 #- 0011 #- +0 #- 0001.23 #- !!str +0.3e3 #- +0.3e3 #- &x foo #- *x #- 1e27 #- 1x+27 """ print('--------------------------------------------- BaseLoader') data = yaml.load(input, Loader=yaml.BaseLoader) print(data) print('--------------------------------------------- SafeLoader') data = yaml.load(input, Loader=yaml.SafeLoader) print(data) print('--------------------------------------------- CoreLoader') data = yaml.load(input, Loader=MyCoreLoader) print(data) print('--------------------------------------------- JSONLoader') data = yaml.load(input, Loader=MyJSONLoader) print(data) print('--------------------------------------------- SafeDumper') out = yaml.dump(data, Dumper=yaml.SafeDumper) print(out) print('--------------------------------------------- MyCoreDumper') out = yaml.dump(data, Dumper=MyCoreDumper) print(out) print('--------------------------------------------- MyJSONDumper') out = yaml.dump(data, Dumper=MyJSONDumper) print(out)
This way people can play with it, and we don't promise this wrapper will stay around forever, and newly created classes CommonDumper/CommonRepresenter aren't exposed. MyCoreLoader = yaml.experimental_12_Core_loader() data = yaml.load(input, Loader=MyCoreLoader) MyCoreDumper = yaml.experimental_12_Core_dumper() out = yaml.dump(data, Dumper=MyCoreDumper)
346c1a4
to
5e41e14
Compare
I just created a yamlcore package that allows you to use YAML 1.2 Core Tags on top of the PyYAML BaseLoader. As this PR is blocked on the API redesign, I decided to create something that users can use today already. |
Where can I find PyYAML 7.0.0a1 so that I can test it? |
Supersedes #512
This is a draft and subject to discussion.
See also #486
(For #512: Thanks to @SUSE for another hackweek! I had four days of work time dedicated to an open source project of my choice. https://hackweek.suse.com/20/projects/yaml-1-dot-2-schema-support-for-pyyaml)
Thanks to @SUSE for a volunteer day I used to make this continuation to my previous PR.
This PR depends on #483
Introduction
For a quick overview of the schema changes between YAML 1.1 and 1.2, look here: https://perlpunk.github.io/yaml-test-schema/schemas.html
While also the syntax was changed in YAML 1.2, this pull request is about the schema changes.
As an example, in 1.1,
Y
,yes
,NO
,on
etc. are resolved as booleans in 1.1.This sounds convenient, but also means that all these 22 different strings must be quoted if they are not meant as booleans. A very common obstacle is the country code for Norway,
NO
("Norway Problem").In YAML 1.2 this was improved by reducing the list of boolean representations.
Also other types have been improved. The 1.1 regular expression for float allows
.
and._
as floats, although there isn't a single digit in these strings.While the 1.2 Core Schema, the recommended default for 1.2, still allows a few variations (
true
,True
andTRUE
, etc.), the 1.2 JSON Schema is there to match JSON behaviour regarding types, so it allows onlytrue
andfalse
.Current State
PyYAML implements the 1.1 types (with a few changes like leaving out the single character booleans
y
,Y
etc.), and it was never updated to support one of the 1.2 Schemas.Problem
Besides the above mentioned problems with the 1.1 types, more and more libraries are created or updated for YAML 1.2, probably also thanks to the relatively new YAML Test Suite, and PyYAML should be able to read and write YAML files used or produced by other libraries.
This PR
The PyYAML Safeloader, which is currently the most recommended Loader if you don't need special behaviour, implements YAML 1.1 types.
I added tagsets for yaml11, json, core.
This way people can try out a YAML 1.2 Loader with little code:
Out of Scope
One problem is that PyYAML's callbacks are class based, and while I was able to make the code a bit more compact via a dictionary of types/callbacks, there are still method calls which must be in a certain class.
The
!!merge <<
key for example needs special handling.That way it's tedious to add custom Loaders. Turning the class based approach into an instance based is on our wishlist.
One example use case we have in mind is, that you want to use the 1.2 CoreLoader, but on top of that you want it to recognize timestamps and mergekeys.
Or you want a very basic loader that should treat everything as a string except booleans and null.
Example