Duplicate Key Error

[lordsoffallen]

Describe the bug
Omega conf can't parse the valid yaml file with anchors defined. The following yaml throws duplicate key error for the section:

anthropic-cached: &anthropic-cache
    <<: *anthropic

When I replace that it works. That's a valid yaml format and key merging should happen

To Reproduce

_models:
  anthropic: &anthropic
    type: &LLM LargeModel
    backend: anthropic
    model: claude-3-5-sonnet-20240620
    credentials: anthropic-api-key
    model_params: &model-params
      temperature: !!float 0.5
  anthropic-cached: &anthropic-cache
    <<: *anthropic
    model_params:
      <<: *model-params
      extra_headers:
        anthropic-beta: prompt-caching-2024-07-31
  google: &google
    type: *LLM
    backend: google
    model: google/gemini-1.5-pro
    model_params: *model-params
    backend_kwargs:
      region: europe-west1
      project_id: 546565-n0
  llama-v3p1: &llama-v3p1
    type: *LLM
    backend: fireworks
    model: accounts/fireworks/models/llama-v3p1-70b-instruct
    credentials: fireworks-api-key
    model_params:
      <<: *model-params
      max_tokens: 16384

summarizer#llm:
  prompt: test
  <<: *anthropic-cache

Expected behavior
I expect omegaconf to read the file without errors.

Additional context

• OmegaConf version: 2.3.0
• Python version: Python 3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]
• Operating system: Ubuntu 22.04

[astrojuanlu]

I cannot reproduce:

from yaml import safe_load
from omegaconf import OmegaConf
from ruamel.yaml import YAML
yaml = YAML()
with open("conf.yaml") as fh:
    content = safe_load(fh)
    
content_oc = OmegaConf.load("conf.yaml")
with open("conf.yaml") as fh:
    content_ru = yaml.load(fh.read())

In [20]: OmegaConf.to_container(content_oc) == content
Out[20]: True

In [21]: OmegaConf.to_container(content_oc) == content_ru
Out[21]: True

[lordsoffallen]

I cannot reproduce:

from yaml import safe_load
from omegaconf import OmegaConf
from ruamel.yaml import YAML
yaml = YAML()
with open("conf.yaml") as fh:
    content = safe_load(fh)
    
content_oc = OmegaConf.load("conf.yaml")
with open("conf.yaml") as fh:
    content_ru = yaml.load(fh.read())

In [20]: OmegaConf.to_container(content_oc) == content
Out[20]: True

In [21]: OmegaConf.to_container(content_oc) == content_ru
Out[21]: True

@astrojuanlu now you should be able to replicate. It throws error when I have the entries like summarizer. However once the anchor is fixed it works as expected.

[astrojuanlu]

Now I can reproduce:

In [16]: import yaml

In [17]: with open("example.yaml") as fh:
    ...:     content = yaml.load(fh, Loader=yaml.SafeLoader)
    ...: 

In [18]: content
Out[18]: 
{'_models': {'anthropic': {'type': 'LargeModel',
   'backend': 'anthropic',
   'model': 'claude-3-5-sonnet-20240620',
   'credentials': 'anthropic-api-key',
   'model_params': {'temperature': 0.5}},
  'anthropic-cached': {'type': 'LargeModel',
   'backend': 'anthropic',
   'model': 'claude-3-5-sonnet-20240620',
   'credentials': 'anthropic-api-key',
   'model_params': {'temperature': 0.5,
    'extra_headers': {'anthropic-beta': 'prompt-caching-2024-07-31'}}},
  'google': {'type': 'LargeModel',
   'backend': 'google',
   'model': 'google/gemini-1.5-pro',
   'model_params': {'temperature': 0.5},
   'backend_kwargs': {'region': 'europe-west1', 'project_id': '546565-n0'}},
  'llama-v3p1': {'type': 'LargeModel',
   'backend': 'fireworks',
   'model': 'accounts/fireworks/models/llama-v3p1-70b-instruct',
   'credentials': 'fireworks-api-key',
   'model_params': {'temperature': 0.5, 'max_tokens': 16384}}},
 'summarizer#llm': {'type': 'LargeModel',
  'backend': 'anthropic',
  'model': 'claude-3-5-sonnet-20240620',
  'credentials': 'anthropic-api-key',
  'model_params': {'temperature': 0.5,
   'extra_headers': {'anthropic-beta': 'prompt-caching-2024-07-31'}},
  'prompt': 'test'}}

In [19]: with open("example.yaml") as fh:
    ...:     content = yaml.load(fh, Loader=get_yaml_loader())
    ...: 
---------------------------------------------------------------------------
ConstructorError                          Traceback (most recent call last)
Cell In[19], line 2
      1 with open("example.yaml") as fh:
----> 2     content = yaml.load(fh, Loader=get_yaml_loader())

File /private/tmp/yaml-playground/.venv/lib/python3.11/site-packages/yaml/__init__.py:81, in load(stream, Loader)
     79 loader = Loader(stream)
     80 try:
---> 81     return loader.get_single_data()
     82 finally:
     83     loader.dispose()

File /private/tmp/yaml-playground/.venv/lib/python3.11/site-packages/yaml/constructor.py:51, in BaseConstructor.get_single_data(self)
     49 node = self.get_single_node()
     50 if node is not None:
---> 51     return self.construct_document(node)
     52 return None

File /private/tmp/yaml-playground/.venv/lib/python3.11/site-packages/yaml/constructor.py:60, in BaseConstructor.construct_document(self, node)
     58     self.state_generators = []
     59     for generator in state_generators:
---> 60         for dummy in generator:
     61             pass
     62 self.constructed_objects = {}

File /private/tmp/yaml-playground/.venv/lib/python3.11/site-packages/yaml/constructor.py:413, in SafeConstructor.construct_yaml_map(self, node)
    411 data = {}
    412 yield data
--> 413 value = self.construct_mapping(node)
    414 data.update(value)

File /private/tmp/yaml-playground/.venv/lib/python3.11/site-packages/omegaconf/_utils.py:144, in get_yaml_loader.<locals>.OmegaConfLoader.construct_mapping(self, node, deep)
    142         continue
    143     if key_node.value in keys:
--> 144         raise yaml.constructor.ConstructorError(
    145             "while constructing a mapping",
    146             node.start_mark,
    147             f"found duplicate key {key_node.value}",
    148             key_node.start_mark,
    149         )
    150     keys.add(key_node.value)
    151 return super().construct_mapping(node, deep=deep)

ConstructorError: while constructing a mapping
  in "example.yaml", line 9, column 21
found duplicate key model_params
  in "example.yaml", line 11, column 5