Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trust_remote_code error in simple evaluate for hellaswag #2222

Closed
sidhantls opened this issue Aug 18, 2024 · 1 comment
Closed

trust_remote_code error in simple evaluate for hellaswag #2222

sidhantls opened this issue Aug 18, 2024 · 1 comment

Comments

@sidhantls
Copy link

sidhantls commented Aug 18, 2024

Getting a remote code error but doesn't seem to be an argument in simple_evaluate that can set trust_remote_code=True. Anyone facing a similar error?

Error:

2024-08-18:12:13:13,353 INFO     [task.py:411] Building contexts for hellaswag on rank 0...
 29%|██▉       | 2903/10042 [00:01<00:02, 2888.73it/s]
  File "python3.8/site-packages/datasets/load.py", line 102, in _raise_timeout_error
    raise ValueError(
ValueError: Loading this dataset requires you to execute custom code contained in the dataset repository on your local machine. Please set the option `trust_remote_code=True` to permit the loading of this dataset.

Script:

  results1 = simple_evaluate( 
           model=lm_obj,
           tasks=["hellaswag", "winogrande", "arc_easy", "arc_challenge", "piqa"],
           num_fewshot=0,
           batch_size=batch_size,
           cache_requests=None,
           log_samples=False,
           bootstrap_iters=0,
           gen_kwargs="max_new_tokens=40",
       )
@sidhantls
Copy link
Author

Fixed it by adding datasets.config.HF_DATASETS_TRUST_REMOTE_CODE = True (from #1467 (comment))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant