How can I split this dataset into train, validation, and test set?

0
votes

I have defined a dataset with my own data, following the instructions from https://www.tensorflow.org/tutorials/load_data/images, as below:

list_ds = tf.data.Dataset.list_files(str(data_dir/'*/*'))

I have looked through the methods of tf.data.Dataset, but couldn't figure out how to split this dataset into three parts(train, validation, test) like tfds.Split.

How can I split this dataset into three parts? I hope the size of train/validation/test set to be 80%, 10%, 10% of list_ds each.

1
pythontensorflow
Did do labeled those? and which format is that data? - Govinda Malavipathirana
The data are jpg/png img data, labeled and split into 7 folders in the original directory. - 이호영

1 Answers

0
votes

This can be achieved in multiple ways:

1) Put your train, test and validation data into three separate folders and call tf.data.Dataset.list_files(...) 3 times with appropriate file path.

2) Make use of Dataset.skip() and Dataset.take(). You will have to manually count the actual number of entries to skip/take based on your dataset size.

More information about dataset maneuvers can be found in TF Docs: https://www.tensorflow.org/guide/data

Hope this helped!