Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Strange points appear when re-train data #21

Closed
thotd21 opened this issue Jul 8, 2020 · 27 comments
Closed

Strange points appear when re-train data #21

thotd21 opened this issue Jul 8, 2020 · 27 comments

Comments

@thotd21
Copy link

thotd21 commented Jul 8, 2020

Hi, I intend to add a new image to the dataset and train it again, but after 20 epochs, the test image has a strange color in the neck:
tinh 20 epoch 3
linh 20 epoch 1
I use a Google colab for training with Pytorch 1.15.
Do you know this error?
And can you give more information about the python environment which runs out the same as the result in the article?
@roman22275
Copy link

Hmm.. Looks weird. Can u give me your weights? I will try to examine it

@LZQhardworker
Copy link
Collaborator

It might be some problem with the version of the code, we will check it.

@roman22275
Copy link

if u can, please contact me zaryanov_r@mail.ru I have something to discuss

@thotd21
Copy link
Author

thotd21 commented Jul 23, 2020

Hmm.. Looks weird. Can u give me your weights? I will try to examine it

Hi, this is the model used when running inference with an error:
https://drive.google.com/drive/folders/1Qrvn-oRJ7jGE93fOvm-VbNct2d49-w5X?usp=sharing

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Jul 23, 2020 via email

@thotd21
Copy link
Author

thotd21 commented Jul 23, 2020

@HAN-YANG-HIT 👍 I'm retraining a 1000-image data set to shorten the time to reaching 200 epoch, and will check the results later.
Thanks for the feedback.

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Jul 23, 2020 via email

@thotd21
Copy link
Author

thotd21 commented Jul 23, 2020 •
edited
Loading

Why does batchsize = 8 affect the quality of generated images?
How many cards do you use when training?
with 2 2080 TI cards I only reached the batch size = 4.

And I want to ask a little more about training details:

model G and G2 should be trained with 200 epoches, while model G1 and U net should be trained with 20 epoches

Does it mean I must train all models to 200 epochs, then use G1 and U net at epoch 20th combined with G and G2 at epoch 200th when running inference? Or train to 20 epochs then use G and G2 models as pretrained models for the remain epochs?
Thanks you.

@thotd21
Copy link
Author

thotd21 commented Jul 23, 2020

@HAN-YANG-HIT
When I use a dataset with only 1000 images for training, will it reduce quality when running tests?
In case of reducing the number of images in training, should I need to increase the number of epochs?

@Learningchen
Copy link

Hi, I intend to add a new image to the dataset and train it again, but after 20 epochs, the test image has a strange color in the neck:
tinh 20 epoch 3
linh 20 epoch 1
I use a Google colab for training with Pytorch 1.15.
Do you know this error?
And can you give more information about the python environment which runs out the same as the result in the article?

could you please tell me how to train my own data? for example ,how to only train G1,G2?

@PLXIV
Copy link

PLXIV commented Mar 22, 2021

I would like to know this as well,

Would you be so kind to give some indications please?

@subinst1999
Copy link

Hi
can you help in training the model in colab . Can you share the configuration file

@amandazw
Copy link

@HAN-YANG-HIT
When I use a dataset with only 1000 images for training, will it reduce quality when running tests?
In case of reducing the number of images in training, should I need to increase the number of epochs?

Hi,did you solve this problem by re-train more times? I got the same problem

@LZQhardworker
Copy link
Collaborator

I proposed a solution to this problem. I posted on the update.

@anwarelahi
Copy link

@LZQhardworker Where is the update. Can you please share it with me as I am trying to retrain model.

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Dec 21, 2021 via email

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Dec 22, 2021 via email

@anwarelahi
Copy link

@LZQhardworker Thanks for the updates. Is it possible to execute each module/step separate? like only execute semantic generation module then clothes warping module.

If possible can you please share script for each module or let me know if it can not be.

@LZQhardworker
Copy link
Collaborator

You can simply separate these modules by commenting the codes

@LZQhardworker
Copy link
Collaborator

@anwarelahi Hi, how is your training. Is the light point artifact solved in your training?

@anwarelahi
Copy link

@anwarelahi Hi, how is your training. Is the light point artifact solved in your training?

Hi, Yes my person image looks fine on sample result but when I use latest .pth files from checkpoints for test, try on is not working on test.py. Can you please suggest what .pth files should be use for testing try on.

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Jan 3, 2022 via email

@anwarelahi
Copy link

Yes right tried by default latest and others like 200 and 20 too but no luck. I put all those .pth files on checkpoints\label2city folder of both train and inference projects.

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Jan 3, 2022 via email

@anwarelahi
Copy link

No, not any type of error, instead try on clothes are not appearing on my model image somehow.

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Jan 3, 2022 via email

@LZQhardworker
Copy link
Collaborator

LZQhardworker commented Jan 3, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@amandazw @PLXIV @thotd21 @Learningchen @subinst1999 @roman22275 @anwarelahi @LZQhardworker