yq -d removes '---' header

[030]

If a file contain a --- header and yq -d is issued, then the header gets removed.

[bquenin]

it's not only on -d, any write would remove the document header. It therefore prevents to use yq in combination with ytt which rely on the document header for metadata annotations.

#@data/values
---
key: value

becomes

#@data/values
key: value

and causes ytt to fail:

Expected YAML document to be annotated with data/values but was *yamlmeta.MapItem

[bquenin]

I had a look at the issue and it looks like this behavior comes from the gopkg.in/yaml.v3 library itself.
For some reason, it doesn't output document separators.

Strangely enough, the information about document separator is decoded and kept in the AST during parsing and later available to the encoder (in Node.Line). But since the behavior of the encoder is not configurable, there's no way to use this information.

The only thing I can think of is to rewrite an encoder supporting document separators or maybe submit a feature request to the official yaml library so that the encoder gets configurable.

[puffan]

It also happens when running yq r -I2 xxx.yaml, the output remove the '---'

[sebastian-philipp]

this is indeed some strange behavior. My original document is:

created: '2020-06-19T09:29:52.800086'
daemon_id: ubuntu
daemon_type: crash
hostname: ubuntu
last_refresh: '2020-06-19T09:33:50.400609'
started: '2020-06-19T09:29:53.002004'
status: 1
status_desc: running
version: 16.0.0-901-g713ef3c
---
container_image_name: foobar
created: '2020-06-19T09:33:32.265147'
daemon_id: ubuntu
daemon_type: mon
hostname: ubuntu
last_refresh: '2020-06-19T09:33:50.400690'
status: -1
status_desc: unknown

running cat document.yaml | yq -C r -d'*' - writes:

container_id: e8cdc5d68c36
container_image_id: 74803e884bea289d2d2d3ebdf6d37cd560499e955595695b1390a89800f4e37a
container_image_name: foobar
created: '2020-06-19T09:29:52.800086'
daemon_id: ubuntu
daemon_type: crash
hostname: ubuntu
last_refresh: '2020-06-19T09:33:50.400609'
started: '2020-06-19T09:29:53.002004'
status: 1
status_desc: running
version: 16.0.0-901-g713ef3c
container_image_name: foorbar
created: '2020-06-19T09:33:32.265147'
daemon_id: ubuntu
daemon_type: mon
hostname: ubuntu
last_refresh: '2020-06-19T09:33:50.400690'
status: -1
status_desc: unknown

which makes ya pretty much useless for me.

[Fixmetal]

+1 I faced this issue too. I've got some Ansible var yaml files which get wrote by yq.
In my case I do a write to update a value on a key like this:
yq write -i "v${VERSION}"

It always wipes out the '---' header.

[mikefarah]

This is fixed in v4 of yq

[jackyzy823]

sorry for bothering you， but i think this has not be fixed in V4.

yq -V
yq version 4.2.0

input aaa.yml

#@data/values
---
key: value

command:

yq eval -i '.key=111' aaa.yml

result

#@data/values
key: 111

[jrmyr]

I can report the same for version 4.3.1: it seems not to be fixed.

[TheKoguryo]

I have the same issue on ver 3 and 4.

[Diaoul]

I can reproduce this as well with an assign operation:

$ cat test.yaml
---
a: 123
b:
  c: 456
$ yq e '.b.c = 789' test.yaml
a: 123
b:
  c: 789

[ProfessorManhattan]

This issue is still occurring with 4.9.1

[mikefarah]

It is too - sorry

[mikefarah]

I've raised an issue against the underlying yaml library: go-yaml/yaml#749

[mikefarah]

Ok now its fixed (4.11.0)

[mikefarah]

Sorry, on closer examination, @Diaoul's example works - not the comment above the doc like in jackyzy823's example

[mikefarah]

Ok I think I've covered all scenarios in the 4.11.1 release @jackyzy823

[jrmyr]

I could successfully test @jackyzy823's example as well.
This was also my use case and I am really happy it is fixed. Thanks a lot!

[mikefarah]

yay!

[heww]

Hi @mikefarah , i tested with yq 1.11.2, seems that it doesn't work like excepted, --- is still removed when it's not at the first line of doc.

ubuntu$ yq -V
yq (https://github.com/mikefarah/yq/) version 4.11.2
ubuntu$ cat <<EOF | yq e '.password = "random password"' -
> #@data/values
> #@overlay/match-child-defaults missing_ok=True
> 
> ---
> #! The namespace to install Harbor
> namespace: harbor
> hostname: example.com
> password:
> EOF
#@data/values
#@overlay/match-child-defaults missing_ok=True
#! The namespace to install Harbor
namespace: harbor
hostname: example.com
password: random password
ubuntu$