Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Questions about the data #7

Open
renxwang opened this issue Jan 24, 2024 · 4 comments
Open

Questions about the data #7

renxwang opened this issue Jan 24, 2024 · 4 comments

Comments

@renxwang
Copy link

Hello, very nice and inspiring paper.
I would like to try your proposed two metrics on my own dataset and well-trained model.

However, I have a problem about this codes in the file "quantify.py":
data = np.load(Path(args.feature_dir, 'data.npz'))
y_p, z_p, y_q, z_q = data['y_p'], data['z_p'], data['y_q'], data['z_q']

How should I calculate y_p, z_p and y_q, z_q based on my own dataset?
Thank you very much.

@m-Just
Copy link
Owner

m-Just commented Jan 24, 2024

Hi, thank you for your interest in our work!

To quantify the shifts, you must train an environment classifier first, and then use the classifier to extract the requested features. These codes are provided in the repository (see train.py and extract.py in the same folder where quantify.py is located). There is also main.py that handles all the procedures for you. Please see README on how to launch the main script.

@renxwang
Copy link
Author

Thanks for your reply.
Just a simple question, if I don't have the ground truth in my test data because it is in vivo, could I still quantify the domain shift of the test dataset using your method?
Thank you very much!

@m-Just
Copy link
Owner

m-Just commented Jan 25, 2024

Without labels, only diversity shift can be quantified. You need labels for both training and test data to quantify correlation shift.

If you have reasons to believe that your model is accurate on the test data, maybe you can generate some pseudo labels for them.

@renxwang
Copy link
Author

Thank you very much and I will try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants