Additional Colon After Scalar Values

[mwaeckerlin]

Hi all, I have issues generating proper scalars (null, numbers, sometimes strings) in YAML with the library.

When I add scalars in JSON objects, they get an additional colon :, as if they were mapping-keys instead of simple scalars:

Background: I want to automate addition of projects to docker-compose.yaml definitions.

Simple reproduction of the problem:

const fs = require('fs')
const yaml = require('yaml')
let compose = yaml.parseDocument('networks:  xxx:\n')
let networks = ['test']
networks.forEach(network => compose.addIn(['networks'], {
  [network]: {
    driver_opts: {
      encrypted: 1
    }
  }
}))
console.log(yaml.stringify(compose, {
  nullStr: '',
  simpleKeys: true,
  lineWidth: 0
}))

Expected Output:

networks:
  xxx:
  test:
      driver_opts:
        encrypted: 1

Actual output:

networks:
  xxx:
  test:
      driver_opts:
        encrypted: 1:

I have the same issue with:

  compose.hasIn(['volumes', 'test-db-volume']) || compose.addIn(['volumes'], {
    ['test-db-volume']: null
  })

Expected:

volumes:
  test-db-volume:

Actual output:

volumes:
  test-db-volume::

Is this a bug or a feature?
And how doe I get what I want?

[mwaeckerlin]

Hmm, this works:

compose.addIn(['volumes', `${project}-db-volume`], null)
compose.addIn(['networks', network, 'driver_opts', 'encrypted'], 1)

→ Why?
→ Is that a bug in the library?

[eemeli]

Huh, yeah, the stringifier for that should fail with simpleKeys: true.

The problem is caused by the addIn() ultimately calling

YAMLMap.prototype.add({ test: { driver_opts: { encrypted: 1 } } })

That function is expecting to either get a Pair instance, or a { key, value } object, and when that doesn't happen, it constructs a pair from the input. It might make more sense to fail at this point, tbh.

So you end up with the object being assigned as a mapping key, with an empty value. Without any stringifier options, your doc formats as

networks:
  xxx:
  ? test:
      driver_opts:
        encrypted: 1

where the ? is an explicit-key indicator as the mapping value is complex.

As complex keys aren't allowed with simpleKeys, formatting that should fail, but the check for that isn't catching a non-node value being used as a key, and so we end up with bad output.

[mwaeckerlin]

The question-mark was another issue that costed me hours to solve. There is no reason for a question-mark!

Finally, I could help with that here: Stack Overflow - How can i prevent yaml parser to put question mark in its output?

Somehow I don't really get the concepts of that library, and I'm loosing hours with searching for the strange behaviour and don't really find it in the documentation… :-(

Why is the key complex? test is a simple string key, no reason for a question mark!
Why does it expect a key/value pair and not simply anything?!?

And I don't understand this description:

add(value), addIn(path, value) 	void 	Adds a value to the collection. For !!map and !!omap the value must be a Pair instance or a { key, value } object, which may not have a key that already exists in the map.

Could you translate that to English, or to Javascript, please:

For !!map and !!omap the value must be a Pair instance or a { key, value } object, which may not have a key that already exists in the map.

What is a "Pair instance or a { key, value } object" in Javascript?
What is a "!!map" or a "!!omap", and what are other available options than those?

What exactly is expected as a value?!?

The documentation is not really written in a language I understand…

[eemeli]

The question-mark was another issue that costed me hours to solve. There is no reason for a question-mark!

As mentioned in the Stack Overflow answer, the ? is a YAML explicit mapping key.

Why is the key complex? test is a simple string key, no reason for a question mark!

With your code, the key you're setting is not the string test, but the object { test: { driver_opts: { encrypted: 1 } } }.

Why does it expect a key/value pair and not simply anything?!?

Because you're setting a value in a mapping, where you need a key and a value.

Could you translate that to English, or to Javascript, please:

For !!map and !!omap the value must be a Pair instance or a { key, value } object, which may not have a key that already exists in the map.

What is a "Pair instance or a { key, value } object" in Javascript? What is a "!!map" or a "!!omap", and what are other available options than those?

A "Pair instance" is an instance of the Pair class. A "{ key, value } object" is a plain object with key and value properties. A !!map is a YAML mapping; !!omap is a YAML 1.1 specific thing.

[mwaeckerlin]

Does this mean, "value" must be a scalar?!? Why? And in fact, it is not necessary, it works well!

How should I the add complex values (that represent objects/maps and arrays)?

What I'm doing: update docker-compose.yaml files by adding new projects, and now it works well like this:

#!/usr/bin/env node
const fs = require('fs')
const yaml = require('yaml')

var argv = require('yargs')(process.argv.slice(2))
  .usage('$0 [OPTIONS] <file> <project>')
  .version('1.0.0')
  .options({
    'd': {>
      alias: 'db',
      type: 'boolean',
      description: 'include generation of a database',
    },
    'g': {
      alias: 'global',
      type: 'boolean',
      description: 'generate global compose (not in infrastructure)',
    }
  })
  .demandCommand(2, 'you need to specify both <file> and <project>')
  .positional('file', {
    describe: 'the docker-compose.yaml file to update',
    type: 'string'
  })
  .positional('project', {
    describe: 'the project name',
    type: 'string'
  })
  .help('h')
  .alias('h', 'help')
  .argv
const [file, project] = argv._
const PROJECT = project.toUpperCase().replace(/-/, '_')

const compose = yaml.parseDocument(fs.readFileSync(file, 'utf-8'), {merge: true})
const networks = [
  ...(argv.db ? [`${project}-db`] : []),
  `ingress-${project}`,
  `kafka-${project}`,
  `redis-${project}`,
  `socket-gateway-${project}`
]
const all = compose.createAlias(compose.get('x-all-services'), 'all-services')
const config = argv.global
  ? {
    extends: {
      file: `backends/${project}/docker-compose.yaml`,
      service: project
    },
    environment: {
      '<<': all,
    }
  } : {
    image: `\${DOCKER_REPOSITORY}/${project}:develop`,
    environment: {
      '<<': all,
      [`${PROJECT}_DB_TYPE`]: 'postgresql',
      [`${PROJECT}_DB_NAME`]: `\${${PROJECT}_DB_NAME}\${ENVIRONMENT}`,
      [`${PROJECT}_DB_HOST`]: `${project}-db`,
      [`${PROJECT}_DB_USER`]: null,
      [`${PROJECT}_DB_PASSWORD`]: null,
      [`${PROJECT}_DB_PORT`]: 5432
    }
  }

const configDb = argv.global ?
  {
    extends: {
      file: `backends/${project}/docker-compose.yaml`,
      service: `${project}-db`
    }
  } : {
    image: 'postgres:16',
    environment: {
      ['POSTGRES_PASSWORD']: `\${${PROJECT}_DB_PASSWORD}`,
      ['POSTGRES_USER']: `\${${PROJECT}_DB_USER}`,
      ['POSTGRES_DB']: `\${${PROJECT}_DB_NAME}\${ENVIRONMENT}`
    }
  }

// Add new service to the compose document
compose.hasIn(['services', project]) || compose.addIn(['services', project], {
  ...config,
  networks
})

// Add db with volume if not already present
if (argv.db) {
  compose.hasIn(['services', `${project}-db`]) || compose.addIn(['services', `${project}-db`], {
    ...configDb,
    volumes: {
      [`${project}-db-volume`]: null
    },
    networks: {
      [`${project}-db`]: null
    }
  })
  compose.hasIn(['volumes', `${project}-db-volume`]) || compose.addIn(['volumes', `${project}-db-volume`], null)
}

// Add networks if not already present
networks.forEach(network => compose.hasIn(['networks', network]) || compose.addIn(['networks', network, 'driver_opts', 'encrypted'], 1))

fs.writeFileSync(file, yaml.stringify(compose, {
  nullStr: '',
  simpleKeys: true,
  lineWidth: 0
}))

It adds to the docker-compose.yaml file (project name test, with flag -d, without flag -d):

services:
  ...
  test:
    image: ${DOCKER_REPOSITORY}/test:develop
    environment:
      <<: *all-services
      TEST_DB_TYPE: postgresql
      TEST_DB_NAME: ${TEST_DB_NAME}${ENVIRONMENT}
      TEST_DB_HOST: test-db
      TEST_DB_USER:
      TEST_DB_PASSWORD:
      TEST_DB_PORT: 5432
    networks:
      - test-db
      - ingress-test
      - kafka-test
      - redis-test
      - socket-gateway-test
  test-db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: ${TEST_DB_PASSWORD}
      POSTGRES_USER: ${TEST_DB_USER}
      POSTGRES_DB: ${TEST_DB_NAME}${ENVIRONMENT}
    volumes:
      test-db-volume:
    networks:
      test-db:
volumes:
  ...
  test-db-volume:
networks:
  ...
  test-db:
    driver_opts:
      encrypted: 1
  ingress-test:
    driver_opts:
      encrypted: 1
  kafka-test:
    driver_opts:
      encrypted: 1
  redis-test:
    driver_opts:
      encrypted: 1
  socket-gateway-test:
    driver_opts:
      encrypted: 1