Format of 'flist' #15

AterLuna · 2018-04-18T02:23:18Z

I want to train network with new dataset.
For training, I tried to modify inpaint.yml file, but I'm not sure about how to set dataset path.
It seems that I have to add new dataset as 'flist' file to DATA_FLIST, but I cannot find how to make appropriate 'flist' files.
Is there any reference for flist file format, or some examples of them?

TrinhQuocNguyen · 2018-04-18T02:26:08Z

Hi AterLuna,
You have to write code for yourself to generate the flist file. Here is my code:

#!/usr/bin/python

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to shuffle')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories
    dirs = os.listdir(args.folder_path)
    dirs_name_list = []

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # print all directory names
    for dir_item in dirs:
        # modify to full path -> directory
        dir_item = args.folder_path + "/" + dir_item
        # print(dir_item)

        training_folder = os.listdir(dir_item + "/training")
        for training_item in training_folder:
            training_item = dir_item + "/training" + "/" + training_item
            training_file_names.append(training_item)

        validation_folder = os.listdir(dir_item + "/validation")
        for validation_item in validation_folder:
            validation_item = dir_item + "/validation" + "/" + validation_item
            validation_file_names.append(validation_item)
    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # This would print all the files and directories

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

JiahuiYu · 2018-04-18T02:29:33Z

@TrinhQuocNguyen Thanks for you response. @AterLuna In addition, the format of file lists are attached. You can either use absolute path or relative path for training.

/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00027049.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00017547.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00023248.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00029613.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00007055.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00021404.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00008928.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00003579.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00010811.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00014556.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015131.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015634.png
...
...

AterLuna · 2018-04-18T02:32:10Z

@TrinhQuocNguyen Thank you for sharing your codes. I'll try it with my dataset.
@JiahuiYu Thank you for your example.

TrinhQuocNguyen · 2018-04-18T02:37:39Z

@AterLuna , Supposed you want to use my code, you need to make a new directory named training_data. In that directory, you make 2 more new directories: training and validation respectively and put your images in those.
@JiahuiYu You are super welcome, thank you for your contribution.

javanli · 2018-06-26T09:34:14Z

@TrinhQuocNguyen
In your code ,line 41,for validation_item in training_folder:
maybe it should be for validation_item in validation_folder?

TrinhQuocNguyen · 2018-07-05T06:52:37Z

I have modified a little bit for anyone needs it. Moreover, you may apply "data augmentation" and put them into other folders ( folder1, folder2,...) without being messy 😄
The directory tree should be looked like:

- model_logs
- neuralgym_logs
- training_data
  -- training
    --- <folder1>
    --- <folder2>
    --- .....
  -- validation
    --- <val_folder1>
    --- <val_folder2>
    --- .....
- <this_file.py>

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The train filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The validation filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to be shuffled')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories and separate them into 2 types: training and validation
    training_dirs = os.listdir(args.folder_path + "/training")
    validation_dirs = os.listdir(args.folder_path + "/validation")

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # append all files into 2 lists
    for training_dir in training_dirs:
        # append each file into the list file names
        training_folder = os.listdir(args.folder_path + "/training" + "/" + training_dir)
        for training_item in training_folder:
            # modify to full path -> directory
            training_item = args.folder_path + "/training" + "/" + training_dir + "/" + training_item
            training_file_names.append(training_item)

    # append all files into 2 lists
    for validation_dir in validation_dirs:
        # append each file into the list file names
        validation_folder = os.listdir(args.folder_path + "/validation" + "/" + validation_dir)
        for validation_item in validation_folder:
            # modify to full path -> directory
            validation_item = args.folder_path + "/validation" + "/" + validation_dir + "/" + validation_item
            validation_file_names.append(validation_item)

    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

ChiediVenia · 2018-10-08T22:52:46Z

I have modified a little bit for anyone needs it. Moreover, you may apply "data augmentation" and put them into other folders ( folder1, folder2,...) without being messy smile
The directory tree should be looked like:

- model_logs
- neuralgym_logs
- training_data
  -- training
    --- <folder1>
    --- <folder2>
    --- .....
  -- validation
    --- <val_folder1>
    --- <val_folder2>
    --- .....
- <this_file.py>

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The train filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The validation filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to be shuffled')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories and separate them into 2 types: training and validation
    training_dirs = os.listdir(args.folder_path + "/training")
    validation_dirs = os.listdir(args.folder_path + "/validation")

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # append all files into 2 lists
    for training_dir in training_dirs:
        # append each file into the list file names
        training_folder = os.listdir(args.folder_path + "/training" + "/" + training_dir)
        for training_item in training_folder:
            # modify to full path -> directory
            training_item = args.folder_path + "/training" + "/" + training_dir + "/" + training_item
            training_file_names.append(training_item)

    # append all files into 2 lists
    for validation_dir in validation_dirs:
        # append each file into the list file names
        validation_folder = os.listdir(args.folder_path + "/validation" + "/" + validation_dir)
        for validation_item in validation_folder:
            # modify to full path -> directory
            validation_item = args.folder_path + "/validation" + "/" + validation_dir + "/" + validation_item
            validation_file_names.append(validation_item)

    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

I created the folder tree, I put the images in the folders, I launch the file with your code and I return this:

./training_data/training/folder1/asd.jpg
./training_data/training/folder6/vdikkj.jpeg
./training_data/training/folder4/vdb.jpeg
./training_data/training/folder2/images.jpeg
./training_data/training/folder3/scv.jpeg
./training_data/training/folder5/waq.jpeg
./training_data/training/folder8/123.jpeg
./training_data/training/folder9/das.jpeg
./training_data/training/folder7/index.jpeg
./training_data/validation/val_folder6/vdikkj.jpeg
./training_data/validation/val_folder9/das.jpeg
./training_data/validation/val_folder1/asd.jpg
./training_data/validation/val_folder2/images.jpeg
./training_data/validation/val_folder7/index.jpeg
./training_data/validation/val_folder3/scv.jpeg
./training_data/validation/val_folder5/waq.jpeg
./training_data/validation/val_folder8/123.jpeg
./training_data/validation/val_folder4/vdb.jpeg
Traceback (most recent call last):
File "this_file.py", line 58, in
os.mknod(args.train_filename)
FileNotFoundError: [Errno 2] No such file or directory

Can you help me?please.

JiahuiYu · 2018-10-09T00:14:21Z

@ChiediVenia Please make sure that your input file exists. Normally it can be fixed by a careful check.

ChiediVenia · 2018-10-09T23:34:02Z

@ChiediVenia Please make sure that your input file exists. Normally it can be fixed by a careful check.

congratulations for the job ... and thanks for the reply.
What should be the input file?
I have created only the folders following the outline and the images of the sub folders. I havent created any other files or folders

JiahuiYu · 2018-10-09T23:57:15Z

@ChiediVenia I wonder if you already have the following files and folders.

parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')

ChiediVenia · 2018-10-14T12:36:45Z

@ChiediVenia I wonder if you already have the following files and folders.

parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')

Thank you for your help

TerminatorSd · 2019-01-31T02:51:56Z

os.mknod is not available to users on Mac OS, the solution can be found there Replaces os.mknod with portable equivalent

In short, replace mknod with open, like this:

os.open(args.train_filename, os.O_CREAT)
# os.mknod(args.train_filename)

JiahuiYu · 2019-01-31T17:14:29Z

@TerminatorSd I appreciate your information to this issue!

YangSN0719 · 2019-03-09T01:42:30Z

I shall be very grateful to you

JiahuiYu · 2019-03-09T01:52:10Z

@YangSN0719 Hi, most likely that your file list is incorrect. The error means there is no image file with the file list you provided. Please check carefully. One suggestion is to use absolute path of file list.

YangSN0719 · 2019-03-12T12:49:17Z

Thank you very much for your reply. As a beginner, I am very sorry to disturb you. I will check your source file again @JiahuiYu

YangSN0719 · 2019-03-12T12:50:38Z

@TrinhQuocNguyen
First of all, thank you very much for your source code, but the file I generated is unusually large, and there are not many pictures. I only selected 5000 for training and 300 for testing. The generated file screenshots are as follows，I would appreciate your reply

YangSN0719 · 2019-03-12T12:58:44Z

@TrinhQuocNguyen Sorry, as a beginner, I am very sorry to disturb you, I will look at your source file again, I hope you can send me your inpaint.yml file for my reference, thank you very much

JiahuiYu · 2019-03-12T16:07:51Z

@YangSN0719 I think you need line break for different image files, right? Each image file occupies one line.

TrinhQuocNguyen · 2019-03-13T02:17:40Z

@TrinhQuocNguyen Sorry, as a beginner, I am very sorry to disturb you, I will look at your source file again, I hope you can send me your inpaint.yml file for my reference, thank you very much

@YangSN0719 I think you need line break for different image files, right? Each image file occupies one line.

That is what I think: you need line break like the author has provided as follows:

/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00027049.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00017547.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00023248.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00029613.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00007055.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00021404.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00008928.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00003579.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00010811.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00014556.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015131.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015634.png
...
...

YangSN0719 · 2019-03-13T05:30:29Z

@TrinhQuocNguyen @JiahuiYu
Thank you very much for your reply.My problem has been solved and is under training,
please refer to the following link
https://blog.csdn.net/Gavinmiaoc/article/details/81250782

lx120 · 2019-11-21T07:36:12Z

Hi, should I use how many valid images in training process? Is it have any influence to training？

JiahuiYu · 2019-11-21T17:50:42Z

@lx120 The number of validation images does not affect training. They are for "validation" purpose.

redaghanem · 2020-01-25T00:37:59Z

Please i need help to train your model

I get error while training

decajcd · 2020-02-25T03:19:09Z

What is the number of training image channels？3 or 4?

minushuang · 2020-06-03T06:53:38Z

@TrinhQuocNguyen Thanks for you response. @AterLuna In addition, the format of file lists are attached. You can either use absolute path or relative path for training.

/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00027049.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00017547.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00023248.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00029613.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00007055.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00021404.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00008928.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00003579.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00010811.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00014556.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015131.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015634.png
...
...

thank u very much for your source code.my question is that the input for training need three pair images, it contains a raw image with mask, a mask and a inpainted image. but it seems that the file list you offered only have one type? what should i do if want to train with three-pairs input

minushuang · 2020-06-04T08:01:10Z

It's OK, the mask is auto generated in the code,thanks

@TrinhQuocNguyen Thanks for you response. @AterLuna In addition, the format of file lists are attached. You can either use absolute path or relative path for training.

/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00027049.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00017547.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00023248.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00029613.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00007055.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00021404.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00008928.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00003579.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00010811.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00014556.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015131.png                                                                                                                                                                                                               
/home/jiahui.yu/data/celeba_hq/celeba_hq_images/img00015634.png
...
...

thank u very much for your source code.my question is that the input for training need three pair images, it contains a raw image with mask, a mask and a inpainted image. but it seems that the file list you offered only have one type? what should i do if want to train with three-pairs input

Mortyzhang · 2020-11-06T11:53:41Z

@TrinhQuocNguyen Thank you for sharing your codes. I'll try it with my dataset.
@JiahuiYu Thank you for your example.

Dear author, I want to ask if I need to prepare mask images for training? Thank you.

HarveyYesan · 2020-12-07T09:25:54Z

@TrinhQuocNguyen
Shell commands can achieve the same function to generate file lists. as following:
find folder/ -name "*.png" | sort > filepath.txt

Hi AterLuna,
You have to write code for yourself to generate the flist file. Here is my code:

#!/usr/bin/python

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to shuffle')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories
    dirs = os.listdir(args.folder_path)
    dirs_name_list = []

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # print all directory names
    for dir_item in dirs:
        # modify to full path -> directory
        dir_item = args.folder_path + "/" + dir_item
        # print(dir_item)

        training_folder = os.listdir(dir_item + "/training")
        for training_item in training_folder:
            training_item = dir_item + "/training" + "/" + training_item
            training_file_names.append(training_item)

        validation_folder = os.listdir(dir_item + "/validation")
        for validation_item in validation_folder:
            validation_item = dir_item + "/validation" + "/" + validation_item
            validation_file_names.append(validation_item)
    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # This would print all the files and directories

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

win-ner · 2021-01-02T12:42:56Z

Must it be divided into training set and validation set?

sdulyq · 2021-04-13T01:18:00Z

I write something easy to modify.

from random import shuffle
import os

train_filename = 'G:/image_inpainting/train_shuffled.flist'
validation_filename = 'G:/image_inpainting/validation_shuffled.flist'
training_file_names = []
validation_file_names = []
training_path = 'G:/image_inpainting/training'
validation_path = 'G:/image_inpainting/validation'
training_items = os.listdir(training_path)
validation_items = os.listdir(validation_path)

# loop
for training_item in training_items:
    training_item = training_path + training_item
    training_file_names.append(training_item)

for validation_item in validation_items:
    validation_item = validation_path + validation_item
    validation_file_names.append(validation_item)


shuffle(training_file_names)
shuffle(validation_file_names)
# write
fo = open(train_filename, "w")
fo.write("\n".join(training_file_names))
fo.close()

fo = open(validation_filename, "w")
fo.write("\n".join(validation_file_names))
fo.close()

hashimhafeez · 2021-05-10T02:58:06Z

Hi AterLuna,
You have to write code for yourself to generate the flist file. Here is my code:

#!/usr/bin/python

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to shuffle')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories
    dirs = os.listdir(args.folder_path)
    dirs_name_list = []

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # print all directory names
    for dir_item in dirs:
        # modify to full path -> directory
        dir_item = args.folder_path + "/" + dir_item
        # print(dir_item)

        training_folder = os.listdir(dir_item + "/training")
        for training_item in training_folder:
            training_item = dir_item + "/training" + "/" + training_item
            training_file_names.append(training_item)

        validation_folder = os.listdir(dir_item + "/validation")
        for validation_item in validation_folder:
            validation_item = dir_item + "/validation" + "/" + validation_item
            validation_file_names.append(validation_item)
    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # This would print all the files and directories

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

Hello i am newbe
Where should i place this code same inpait.yml file or to creat new one?

eduardatmadenn · 2021-05-19T11:17:19Z

Small detail here about the folder structure, maybe it helps: The code works as is, if the folder hierarchy is like this:

project_root/
├─ training_data/
│  ├─ name_of_db/
│  │  ├─ training/
│  │  ├─ validation/

mowei951010 · 2021-11-12T09:47:09Z

I want to fix my photos, but running inpaint_ops.py doesn't work,
Hope the author can more detailed information I want to learn

AnhPC03 · 2023-03-12T10:46:20Z

Hi @JiahuiYu , @TrinhQuocNguyen
The flist.txt file only contains path to normal images? Where do I put mask of that images? Thank you

fittiing · 2023-08-28T13:29:09Z

嗨 AterLuna，您必须自己编写代码来生成 flist 文件。这是我的代码：

#!/usr/bin/python

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to shuffle')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories
    dirs = os.listdir(args.folder_path)
    dirs_name_list = []

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # print all directory names
    for dir_item in dirs:
        # modify to full path -> directory
        dir_item = args.folder_path + "/" + dir_item
        # print(dir_item)

        training_folder = os.listdir(dir_item + "/training")
        for training_item in training_folder:
            training_item = dir_item + "/training" + "/" + training_item
            training_file_names.append(training_item)

        validation_folder = os.listdir(dir_item + "/validation")
        for validation_item in validation_folder:
            validation_item = dir_item + "/validation" + "/" + validation_item
            validation_file_names.append(validation_item)
    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # This would print all the files and directories

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

May I ask which directory and folder to place this. flist file in after writing?

fittiing · 2023-08-29T02:14:52Z

你好 AterLuna，您必须自己编写代码来生成 flist 文件。这是我的代码：

#!/usr/bin/python

import argparse
import os
from random import shuffle

parser = argparse.ArgumentParser()
parser.add_argument('--folder_path', default='./training_data', type=str,
                    help='The folder path')
parser.add_argument('--train_filename', default='./data_flist/train_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--validation_filename', default='./data_flist/validation_shuffled.flist', type=str,
                    help='The output filename.')
parser.add_argument('--is_shuffled', default='1', type=int,
                    help='Needed to shuffle')

if __name__ == "__main__":

    args = parser.parse_args()

    # get the list of directories
    dirs = os.listdir(args.folder_path)
    dirs_name_list = []

    # make 2 lists to save file paths
    training_file_names = []
    validation_file_names = []

    # print all directory names
    for dir_item in dirs:
        # modify to full path -> directory
        dir_item = args.folder_path + "/" + dir_item
        # print(dir_item)

        training_folder = os.listdir(dir_item + "/training")
        for training_item in training_folder:
            training_item = dir_item + "/training" + "/" + training_item
            training_file_names.append(training_item)

        validation_folder = os.listdir(dir_item + "/validation")
        for validation_item in validation_folder:
            validation_item = dir_item + "/validation" + "/" + validation_item
            validation_file_names.append(validation_item)
    # print all file paths
    for i in training_file_names:
        print(i)
    for i in validation_file_names:
        print(i)

    # This would print all the files and directories

    # shuffle file names if set
    if args.is_shuffled == 1:
        shuffle(training_file_names)
        shuffle(validation_file_names)

    # make output file if not existed
    if not os.path.exists(args.train_filename):
        os.mknod(args.train_filename)

    if not os.path.exists(args.validation_filename):
        os.mknod(args.validation_filename)

    # write to file
    fo = open(args.train_filename, "w")
    fo.write("\n".join(training_file_names))
    fo.close()

    fo = open(args.validation_filename, "w")
    fo.write("\n".join(validation_file_names))
    fo.close()

    # print process
    print("Written file is: ", args.train_filename, ", is_shuffle: ", args.is_shuffled)

Hello, I ran according to your code and encountered this error. How can I solve it? Isn't this code creating a . flit file?Traceback (most recent call last):
File ".\1.py", line 59, in
with open(args.train_filename, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: './data_flist/train_shuffled.flist'

JiahuiYu closed this as completed Apr 18, 2018

JiahuiYu mentioned this issue Apr 26, 2018

How should the filelist look like? #20

Closed

Hunterwolf88 mentioned this issue Apr 30, 2018

training with my own dataset #23

Closed

jhyume mentioned this issue May 5, 2018

how to create a flist file? #27

Closed

JiahuiYu mentioned this issue May 11, 2018

File preparation #32

Closed

jiyoungAn mentioned this issue May 25, 2018

'Image is None' Problem #48

Closed

JiahuiYu mentioned this issue Jun 14, 2018

Zero epoch for two days #60

Closed

This comment has been minimized.

Sign in to view

JiahuiYu added the good first issue Good for newcomers label Aug 9, 2019

JiahuiYu reopened this Aug 9, 2019

LeonCurry mentioned this issue Aug 29, 2019

Train data konstantg/dem-fill#2

Closed

JiahuiYu mentioned this issue Dec 23, 2019

About the dimensions of inputs #123

Open

JiahuiYu closed this as completed Jan 25, 2020

Repository owner deleted a comment from redaghanem Jan 25, 2020

JiahuiYu reopened this Jan 25, 2020

This was referenced May 17, 2021

How to generate mask_flist.txt and how to turn my mask.png to mask.pkl? avalonstrel/GatedConvolution_pytorch#16

Open

How to generate the val_ff_mask_flist.txt/val_rect_mask_flist.txt avalonstrel/GatedConvolution_pytorch#13

Open

Tiuuuuu mentioned this issue Jun 27, 2023

I write something easy to modify. #543

Open

Format of 'flist' #15

Format of 'flist' #15

Comments

AterLuna commented Apr 18, 2018 • edited Loading

TrinhQuocNguyen commented Apr 18, 2018 • edited Loading

JiahuiYu commented Apr 18, 2018

AterLuna commented Apr 18, 2018

TrinhQuocNguyen commented Apr 18, 2018 • edited Loading

javanli commented Jun 26, 2018

TrinhQuocNguyen commented Jul 5, 2018 • edited Loading

ChiediVenia commented Oct 8, 2018

JiahuiYu commented Oct 9, 2018

ChiediVenia commented Oct 9, 2018

JiahuiYu commented Oct 9, 2018

ChiediVenia commented Oct 14, 2018

TerminatorSd commented Jan 31, 2019 • edited Loading

JiahuiYu commented Jan 31, 2019

This comment has been minimized.

YangSN0719 commented Mar 9, 2019

JiahuiYu commented Mar 9, 2019

YangSN0719 commented Mar 12, 2019

YangSN0719 commented Mar 12, 2019

YangSN0719 commented Mar 12, 2019

JiahuiYu commented Mar 12, 2019

TrinhQuocNguyen commented Mar 13, 2019 • edited Loading

YangSN0719 commented Mar 13, 2019

lx120 commented Nov 21, 2019

JiahuiYu commented Nov 21, 2019

redaghanem commented Jan 25, 2020

decajcd commented Feb 25, 2020

minushuang commented Jun 3, 2020 • edited Loading

minushuang commented Jun 4, 2020

Mortyzhang commented Nov 6, 2020

HarveyYesan commented Dec 7, 2020

win-ner commented Jan 2, 2021

sdulyq commented Apr 13, 2021

hashimhafeez commented May 10, 2021

eduardatmadenn commented May 19, 2021

mowei951010 commented Nov 12, 2021

AnhPC03 commented Mar 12, 2023

fittiing commented Aug 28, 2023

fittiing commented Aug 29, 2023

AterLuna commented Apr 18, 2018 •

edited

Loading

TrinhQuocNguyen commented Apr 18, 2018 •

edited

Loading

TrinhQuocNguyen commented Apr 18, 2018 •

edited

Loading

TrinhQuocNguyen commented Jul 5, 2018 •

edited

Loading

TerminatorSd commented Jan 31, 2019 •

edited

Loading

TrinhQuocNguyen commented Mar 13, 2019 •

edited

Loading

minushuang commented Jun 3, 2020 •

edited

Loading