I want to train a model to detect eye prescription data from pictures/pdfs. Ex. prescription
I'm thinking of splitting the task into 2:
- Define/train a model that will detect where the table on picture is.
- Define/train a model that will predict data from the result of previous model.
And I'm stuck at the first model. I started by creating 100k images (ex. https://i.stack.imgur.com/CQJXE.png, https://i.stack.imgur.com/iJHuk.png) with tables in them and tables' coordinates as label. Example of label:
27.38 // top coordinate of 1 table in picture
172.00 // left coordinate of 1 table
459.62 // right coordinate of 1 table
311.00 // bottom coordinate of 1 table
4.00 // number of columns of 1 table
5.00 // number of rows of 1 table
25.00 // top coordinate of 2 table in picture
424.50 // left coordinate of 2 table
462.00 // right coordinate of 2 table
544.50 // bottom coordinate of 2 table
7.00 // number of columns of 2 table
2.00 // number of rows of 2 table
Preparing data
X:
image = tf.image.decode_png(image_string, channels=3)
image = tf.image.resize(image,(337, 238), method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
image = tf.image.rgb_to_grayscale(image,name=None)
image = tf.image.convert_image_dtype(image, dtype=tf.uint16, saturate=False)
image_string = tf.image.encode_png( image , compression=-1, name=None)
image_string = image_string.numpy()
image_shape = image.shape
Y:
I just multiply coordinate by 0.4 (because I'm resizing image to 40%), other data from generated label leaving as 0 (at least for now, expect to predict only coordinates) repeated 5 times, because there might be only 5 tables in picture.
Ex:
0.6120000000000001, 24.400000000000002, 193.284, 65.60000000000001, 0, 0, 2.8920000000000003, 97.60000000000001, 189.784, 144.0, 0, 0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0
Training:
Creating dataset:
def parseExample(example):
features = {
"image": tf.io.FixedLenFeature([], tf.string),
"output": tf.io.VarLenFeature(dtype=tf.float32),
# "output": tf.io.FixedLenFeature([], tf.float32),
'height': tf.io.FixedLenFeature([], tf.int64),
'width': tf.io.FixedLenFeature([], tf.int64),
'depth': tf.io.FixedLenFeature([], tf.int64),
'name': tf.io.FixedLenFeature([], tf.string)
}
parsed = tf.io.parse_single_example(example, features=features)
image = tf.image.decode_png(parsed['image'], channels=1)
image = tf.reshape(image, (parsed['height'], parsed['width'], parsed['depth']))
label = parsed["output"]
return image, label.values
def make_dataset(dir, dtype, dataSetType, parse_fn):
dataset = tf.data.TFRecordDataset(tf.data.Dataset.list_files("{}\\{}_{}_*.tfrecord".format(args.records_dir,dataSetType, dtype )))
dataset = dataset.shuffle(buffer_size=1000)
dataset = dataset.map(parse_fn)
dataset = dataset.batch(batch_size=32)
dataset.cache('E:\\trainingcache')
return dataset
Model: After a lot of playing around with different layers/activation/optimizers/losses, this is what working out the best.
def getTableModel(shape):
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(256,3,3, padding='valid',activation='relu', dilation_rate=(1, 1), use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', input_shape= shape),
tf.keras.layers.Conv2D(128,3,3, activation='relu'),
tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(30, activation='relu', kernel_initializer='normal')
])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01, amsgrad=True)
model.compile(loss='categorical_crossentropy',
optimizer=optimizer,
metrics=['mse', 'mae','acc'])
return model
Training:
def train(model, train, cppath):
EPOCHS = 1000
cbs = [PrintDot()]
checkpointcb = tf.keras.callbacks.ModelCheckpoint(filepath=cppath,
save_weights_only=True,
verbose=1, period=100)
print(model.summary())
# The patience parameter is the amount of epochs to check for improvement
early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=10)
history = model.fit(train, epochs=EPOCHS, verbose=0, callbacks=[PrintDot(), checkpointcb])
What am i missing or doing wrong ? Because the best accuracy that I get is ~0.7
Output after training and testing:
625/625 [==============================] - 20s 32ms/step - loss: 1620.7100 - mse: 110371661623007769526272.0000 - mae: 133217427456.0000 - acc: 0.7244 test loss, test acc: [1620.709959765625, 1.1037166e+23, 133217430000.0, 0.7244]
Testing 1 prediction:
Prediction:
[8.52919688e+12 2.94164575e+13 1.72934342e+14 6.60441483e+13 0.00000000e+00 0.00000000e+00 0.00000000e+00 8.95260683e+13 1.73240895e+14 1.32049206e+14 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]
Actual label
[-0.6120000000000001, 24.400000000000002, 193.284, 65.60000000000001, 0, 0, 2.8920000000000003, 97.60000000000001, 189.784, 144.0, 0, 0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
