v0.18 regression: colons in flow mapping values are no longer accepted

[sersorrel]

I have a YAML file which looks like this:

data:
  - src: {url: http://example.com}

If I use remarshal 0.17, I can convert this to JSON just fine:

$ nix run nixpkgs#remarshal_0_17 -- --of json test.yaml
{"data":[{"src":{"url":"http://example.com"}}]}

but with remarshal 0.20, parsing fails:

$ nix run nixpkgs#remarshal -- -t json test.yaml
Error: Cannot parse as YAML (found unexpected ':')

This appears to be related to an explicit exception in ruamel.yaml's scan_plain() for values like this:

# It's not clear what we should do with ':' in the flow context.
if (
    self.flow_level
    and ch == ':'
    and srp(length + 1) not in '\0 \t\r\n\x85\u2028\u2029,[]{}'
):
    srf(length)
    raise ScannerError(
        'while scanning a plain scalar',
        start_mark,
        "found unexpected ':'",
        self.reader.get_mark(),
        'Please check '
        'http://pyyaml.org/wiki/YAMLColonInFlowContext '
        'for details.',
    )

but this was fixed in PyYAML in 2017 with yaml/pyyaml#45, and I can't reproduce the same error installing ruamel.yaml from PyPI, so I'm not entirely sure what's going on here.

[dbohdan]

Thanks for a detailed bug report. What triggers the bug is passing typ="safe" to ruamel.yaml.YAML. This is why you can't reproduce it by default. The deeper cause is that apparently the C version of SafeLoader in ruamel.yaml has fallen behind the pure-Python version and doesn't behave identically. (It is worrying that despite this it's used when available. I'd prefer for the C version to be disabled by default until it caught up.) I have fixed the bug by forcing Remarshal to always use the pure-Python version and released 1.0.0rc3.

Note that in your use case pnpm generates YAML 1.2. It is very unlikely you are going to encounter a problem because pnpm lock files only use a subset of YAML, but Remarshal 0.17 is technically the wrong version for it. I recommend upgrading with the next non-rc release. (The big difference between Remarshal 0.17 and 0.18 was the switch from PyYAML to ruamel.yaml and therefore from YAML 1.1 to 1.2. This is why Nixpkgs has a separate version 0.17 for those who need YAML 1.1. I made a mistake by not declaring those Remarshal releases version 1.0 and 2.0 respectively.)

[sersorrel]

Awesome – thank you for the extremely rapid fix!

[dbohdan]

You're welcome!