Importing own dataset in TensorFlow

Question

I am making a neural network using TensorFlow. I now want to import my own images as dataset, to train the neural network on these images.

For this, I at first get a list of filenames and their corresponding label. I am doing this with the following code:

locations = {"chimpanzee":["G:\Profielwerkstuk\chimpanzee"],
           "gorilla":["G:\Profielwerkstuk\gorilla"],
           "howler monkey":["G:\Profielwerkstuk\howler_monkey"]}

files = []
labels = []
numberlabel = []

for label, folders in locations.items():
    numberlabel.append(label)
    filesperlabel = []
    for folder in folders:
        filesperlabel.extend([join(folder, f) for f in listdir(folder) if isfile(join(folder, f))])
    print(len(filesperlabel))
    files.extend(filesperlabel)
    labels.extend([len(numberlabel) - 1] * len(filesperlabel))

locations is here a dictionary with the label as key and their folder as value.

This works fine: I get a list of the file locations, a list of the labels and a list of which turns the output neuron number in a monkey type.

I am then trying to makes from this a TensorFlow dataset. I at first convert the list of the filenames and the labelnames in a TensorFlow constant:

filenames = tf.constant(files)
labelnames = tf.constant(labels)

I first tried the code on the Tensorflow website. However, I get the error that image contains no shape after decoding an image. How to correctly import images?

Further, after making an dataset, I want to loop through it in batches. The MNIST dataset contains a very handy function for it, but is it also possible for custom datasets? Searching for this problems doesn't give me a lot of useful tips.

Manh Khôi Duong Manh Khôi Duong · Accepted Answer · 2018-03-13T18:53:31

For the import: Use OpenCV to import images

import opencv as cv
imgDataset = []
for i in range(len(files)):
    imgDataset.append(cv2.imread(files[i]))
imgDataset = np.asarray(imgDataset)

You will get an numpy array of the shape (num_files, height, width, color_channels)

Accessing the i-th image should be imgDataset[i].

The problem here can be that if your dataset is too big, you will later get problems to feed it into placeholders. Batching the whole dataset can be difficult.

If you are getting difficulties with the size then take a look at this link: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly.html

This is a step closer to batch the dataset.

Importing own dataset in TensorFlow

1 Answers