Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance gap in reproducing results #12

Open
ShangGL123 opened this issue Jul 23, 2023 · 3 comments
Open

Performance gap in reproducing results #12

ShangGL123 opened this issue Jul 23, 2023 · 3 comments

Comments

@ShangGL123
Copy link

image
The reproduced results show that DS is much lower than the numerical value in the paper. Is this result jitter normal?

@jiaxiaosong1002
Copy link
Collaborator

Hi, thanks for your interests.

This codebase has encountered a major refactor and it is mixed with some implementations in our next work (accepted by ICCV 2023 and this time we will release both the original 189K and 2M checkpoints, please stay tuned!). without thorough inspection and tuning due to limited time and computational resource. The model is also retrained.

Thus, there might be some bugs. I check out one of our old team_agent files and write a similar one for this codebase:
old_team_agent.py
Could you please try this one? (Just replace the original_team_agent) In the meantime, I will cross-check with the private codebase when having time.

Thanks for your time.

@Naive-Bayes
Copy link

image The reproduced results show that DS is much lower than the numerical value in the paper. Is this result jitter normal?

Hello, which scenarios file do you use, just the "all_towns_traffic_scenarios_no256.json" provided in the codebase?

@raozhongyu
Copy link

Hi, I'd like to ask can you provide the inference time when you evaluate this code. How long it takes ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants