-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hydra override error when running evaluations after non-editable installation #12
Comments
Where did you run |
I ran |
Low priority - things are working when we are executing |
Hi. I am having the same issue when running toxicity assessment. I tried to run |
In the DecodingTrust directory, dt-run +toxicity=realtoxicityprompts-toxic ++dry_run=True ++model=openai/gpt-3.5-turbo-0301 ++toxicity.n=25 ++toxicity.template=0 |
@danielz02 You mentioned that you could run without errors. Do you think this is caused by our environment settings ? |
Same issue here. I |
Hi, we are working on a new version that integrates everything more smoothly. Could you try editable install? We also have a newer version in the |
Awesome! Editable install solved this issue. Also looking forward to the new version! |
Hi, it seems that the
when I print the config in
May I ask how to solve this issue? |
Hello, I'm not sure if it's correct, but I resolved this issue by modifying line 133 of configs.py to |
Hi! |
Hi! I'm still facing this same issue.
returns:
|
Hi! Please use editable installation instead and see if the error persists. |
Describe the bug
If I install DecodingTrust using the second method under '(Conda +) Pip' section, and then I run an evaluation using the provided script, I'll get an Hydra override error.
To Reproduce
Steps to reproduce the behavior:
Note: there are two installation methods you can use to reproduce this error
conda create --name dt-test python=3.9 pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia conda activate dt-test pip install "decoding-trust @ git+https://github.com/AI-secure/DecodingTrust.git"
omegaconf.errors.ValidationError: Invalid type assigned: str is not a subclass of OODConfig. value: knowledge_2020_5shot full_key: ood object_type=BaseConfig
Expected behavior
There shouldn't be any error in getting the output config. Specifically in my case, the key 'ood' in the output config should have a dict value acquired from the config file of which the name is the command line arg for 'ood'.
Screenshots
Environment:
Additional context
The text was updated successfully, but these errors were encountered: