Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ERROR] hydra.errors.ConfigCompositionException: Error merging override corpus_path=psgs_w100.tsv.gz #2

Closed
VincentXWD opened this issue Jan 5, 2025 · 2 comments
Closed

[ERROR] hydra.errors.ConfigCompositionException: Error merging override corpus_path=psgs_w100.tsv.gz #2

VincentXWD opened this issue Jan 5, 2025 · 2 comments

Comments

@VincentXWD
Copy link
Contributor

Hello developers,

Thanks for contributing such a great RAG open source project! I'm building index and the dense retriever via script provided by flexrag on my own server:

CORPUS_PATH=psgs_w100.tsv.gz
CORPUS_FIELDS='[title,text]'
DB_PATH=./

HYDRA_FULL_ERROR=1 python -m flexrag.entrypoints.prepare_index \
    corpus_path=$CORPUS_PATH \
    saving_fields=$CORPUS_FIELDS \
    retriever_type=dense \
    dense_config.database_path=$DB_PATH \
    dense_config.encode_fields='[text]' \
    dense_config.passage_encoder_config.encoder_type=hf \
    dense_config.passage_encoder_config.hf_config.model_path='facebook/contriever' \
    dense_config.passage_encoder_config.hf_config.device_id=[0,1,2,3] \
    dense_config.index_type=faiss \
    dense_config.faiss_config.batch_size=4096 \
    dense_config.faiss_config.log_interval=100000 \
    dense_config.batch_size=4096 \
    dense_config.log_interval=100000 \
    reinit=True

and meet errors below:

Traceback (most recent call last):
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/config_loader_impl.py", line 390, in _apply_overrides_to_config
    OmegaConf.update(cfg, key, value, merge=True)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 741, in update
    root.__setattr__(last_key, value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/dictconfig.py", line 338, in __setattr__
    self._format_and_raise(key=key, value=value, cause=e)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/dictconfig.py", line 334, in __setattr__
    self.__set_impl(key, value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/dictconfig.py", line 318, in __set_impl
    self._set_item_impl(key, value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 604, in _set_item_impl
    self.__dict__["_content"][key]._set_value(value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/listconfig.py", line 618, in _set_value
    raise e
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/listconfig.py", line 614, in _set_value
    self._set_value_impl(value, flags)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/listconfig.py", line 646, in _set_value_impl
    raise ValidationError(msg)
omegaconf.errors.ValidationError: Invalid value assigned: str is not a ListConfig, list or tuple.
    full_key: corpus_path
    object_type=Config

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/flexrag/entrypoints/prepare_index.py", line 95, in <module>
    main()
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
           ^^^^^^
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
            ^^^^^^^^^^
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/hydra.py", line 105, in run
    cfg = self.compose_config(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/config_loader_impl.py", line 276, in _load_configuration_impl
    ConfigLoaderImpl._apply_overrides_to_config(config_overrides, cfg)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/config_loader_impl.py", line 397, in _apply_overrides_to_config
    raise ConfigCompositionException(
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/hydra/_internal/config_loader_impl.py", line 390, in _apply_overrides_to_config
    OmegaConf.update(cfg, key, value, merge=True)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 741, in update
    root.__setattr__(last_key, value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/dictconfig.py", line 338, in __setattr__
    self._format_and_raise(key=key, value=value, cause=e)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/dictconfig.py", line 334, in __setattr__
    self.__set_impl(key, value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/dictconfig.py", line 318, in __set_impl
    self._set_item_impl(key, value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/basecontainer.py", line 604, in _set_item_impl
    self.__dict__["_content"][key]._set_value(value)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/listconfig.py", line 618, in _set_value
    raise e
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/listconfig.py", line 614, in _set_value
    self._set_value_impl(value, flags)
  File "/home/wdxu/miniconda3/envs/py311/lib/python3.11/site-packages/omegaconf/listconfig.py", line 646, in _set_value_impl
    raise ValidationError(msg)
hydra.errors.ConfigCompositionException: Error merging override corpus_path=psgs_w100.tsv.gz

Since I'm new in RAG and not familiar with hydra. I'm wondering how to solve this problem. Thanks!
@VincentXWD
Copy link
Contributor Author

VincentXWD commented Jan 5, 2025 •
edited
Loading

Solved, seems we should use [psgs_w100.tsv] rather than 'psgs_w100.tsv.gz'

@ZhuochengZhang98
Copy link
Collaborator

Thank you for your feedback. The issue was solved by #3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@VincentXWD @ZhuochengZhang98