Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regarding the inconsistency between the results of testing with the checkpoint you provided and in the paper #8

Open
heikeyuhuajia opened this issue May 17, 2024 · 1 comment

Comments

@heikeyuhuajia
Copy link

heikeyuhuajia commented May 17, 2024

Hello, I downloaded the LEVIR and WHU datasets through your Baidu.com link and got the BestLEVIR and BestWHU checkpoints through your link, but when I tested them, I found that there was a slight fluctuation in the test results with yours, according to my intrinsic thought that the test shouldn't fluctuate, right? Also if I put my thesis into your test should the results be consistent with your thesis or with my test results. I didn't retrain, I just got your trained checkpoints.

# my test in LEVIR
Begin evaluation...
Is_training: False. [1,256],  running_mf1: 0.96526
Is_training: False. [101,256],  running_mf1: 0.96637
Is_training: False. [201,256],  running_mf1: 0.94838
acc: 0.99020 miou: 0.90490 mf1: 0.94799 iou_0: 0.98974 iou_1: 0.82007 F1_0: 0.99484 F1_1: 0.90114 precision_0: 0.99342 precision_1: 0.92660 recall_0: 0.99627 recall_1: 0.87704 
# your paper in LEVIR
Begin evaluation...
Is_training: False. [1,256],  running_mf1: 0.94652
Is_training: False. [101,256],  running_mf1: 0.95923
Is_training: False. [201,256],  running_mf1: 0.94998
acc: 0.99012 miou: 0.90427 mf1: 0.94761 iou_0: 0.98966 iou_1: 0.81887 F1_0: 0.99480 F1_1: 0.90042 precision_0: 0.99339 precision_1: 0.92574 recall_0: 0.99623 recall_1: 0.87645
# my test in WHU
Begin evaluation...
Is_training: False. [1,95],  running_mf1: 0.93710
acc: 0.99156 miou: 0.89873 mf1: 0.94416 iou_0: 0.99126 iou_1: 0.80620 F1_0: 0.99561 F1_1: 0.89270 precision_0: 0.99578 precision_1: 0.88894 recall_0: 0.99544 recall_1: 0.89649
# your paper in WHU
Begin evaluation...
Is_training: False. [1,95],  running_mf1: 0.93640
acc: 0.99182 miou: 0.90135 mf1: 0.94575 iou_0: 0.99152 iou_1: 0.81119 F1_0: 0.99574 F1_1: 0.89575 precision_0: 0.99583 precision_1: 0.89385 recall_0: 0.99566 recall_1: 0.89766 
@chengtianxiu
Copy link
Collaborator

Hello, I downloaded the LEVIR and WHU datasets through your Baidu.com link and got the BestLEVIR and BestWHU checkpoints through your link, but when I tested them, I found that there was a slight fluctuation in the test results with yours, according to my intrinsic thought that the test shouldn't fluctuate, right? Also if I put my thesis into your test should the results be consistent with your thesis or with my test results. I didn't retrain, I just got your trained checkpoints.

# my test in LEVIR
Begin evaluation...
Is_training: False. [1,256],  running_mf1: 0.96526
Is_training: False. [101,256],  running_mf1: 0.96637
Is_training: False. [201,256],  running_mf1: 0.94838
acc: 0.99020 miou: 0.90490 mf1: 0.94799 iou_0: 0.98974 iou_1: 0.82007 F1_0: 0.99484 F1_1: 0.90114 precision_0: 0.99342 precision_1: 0.92660 recall_0: 0.99627 recall_1: 0.87704 
# your paper in LEVIR
Begin evaluation...
Is_training: False. [1,256],  running_mf1: 0.94652
Is_training: False. [101,256],  running_mf1: 0.95923
Is_training: False. [201,256],  running_mf1: 0.94998
acc: 0.99012 miou: 0.90427 mf1: 0.94761 iou_0: 0.98966 iou_1: 0.81887 F1_0: 0.99480 F1_1: 0.90042 precision_0: 0.99339 precision_1: 0.92574 recall_0: 0.99623 recall_1: 0.87645
# my test in WHU
Begin evaluation...
Is_training: False. [1,95],  running_mf1: 0.93710
acc: 0.99156 miou: 0.89873 mf1: 0.94416 iou_0: 0.99126 iou_1: 0.80620 F1_0: 0.99561 F1_1: 0.89270 precision_0: 0.99578 precision_1: 0.88894 recall_0: 0.99544 recall_1: 0.89649
# your paper in WHU
Begin evaluation...
Is_training: False. [1,95],  running_mf1: 0.93640
acc: 0.99182 miou: 0.90135 mf1: 0.94575 iou_0: 0.99152 iou_1: 0.81119 F1_0: 0.99574 F1_1: 0.89575 precision_0: 0.99583 precision_1: 0.89385 recall_0: 0.99566 recall_1: 0.89766 

Indeed, the experimental results should not fluctuate, perhaps inconsistencies in the version of the environment can cause this problem, make sure the version is the same as in the Requirements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants