I'm training a CNN using for image classification. Due to the limited size of my data set I'm using transfer learning. Basically, I'm using the pre-trained network Google is proving in its retrain example (https://www.tensorflow.org/tutorials/image_retraining).
The model works great and gives a very good accuracy. But my dataset is highly imbalance which mean accuracy is not the best metric to judge the performance of the model.
By looking into different solutions, some suggested changing the sampling method or the performance metric used. I'm choosing to go with the later.
Tensorflow provides a good verity of metrics including, AUC, precision, recall, etc.
Now, here is the code of the retraing model: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/image_retraining/retrain.py
I'm adding the following to add_evaluation_step(result_tensor, ground_truth_tensor) function:
with tf.name_scope('AUC'):
with tf.name_scope('prediction'):
prediction = tf.argmax(result_tensor, 1)
with tf.name_scope('AUC'):
auc_value = tf.metrics.auc(tf.argmax(ground_truth_tensor, 1), prediction, curve='ROC')
tf.summary.scalar('accuracy', evaluation_step)
tf.summary.scalar('AUC', auc_value)
But I'm getting this error:
Traceback (most recent call last): File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/examples/image_retraining/retrain.py", line 1135, in tf.app.run(main=main, argv=[sys.argv[0]] + unparsed) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 44, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/examples/image_retraining/retrain.py", line 911, in main ground_truth_input: train_ground_truth}) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 767, in run run_metadata_ptr) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 965, in _run feed_dict_string, options, run_metadata) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1015, in _do_run target_list, options, run_metadata) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/client/session.py", line 1035, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value AUC/AUC/auc/false_positives
[[Node: AUC/AUC/auc/false_positives/read = IdentityT=DT_FLOAT, _class=["loc:@AUC/AUC/auc/false_positives"], _device="/job:localhost/replica:0/task:0/cpu:0"]]Caused by op u'AUC/AUC/auc/false_positives/read', defined at: File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/examples/image_retraining/retrain.py", line 1135, in tf.app.run(main=main, argv=[sys.argv[0]] + unparsed) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/platform/app.py", line 44, in run _sys.exit(main(_sys.argv[:1] + flags_passthrough)) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/examples/image_retraining/retrain.py", line 874, in main final_tensor, ground_truth_input) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/examples/image_retraining/retrain.py", line 806, in add_evaluation_step auc_value, update_op = tf.metrics.auc(tf.argmax(ground_truth_tensor, 1), prediction, curve='ROC') File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/ops/metrics_impl.py", line 555, in auc labels, predictions, thresholds, weights) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/ops/metrics_impl.py", line 473, in _confusion_matrix_at_thresholds false_p = _create_local('false_positives', shape=[num_thresholds]) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/ops/metrics_impl.py", line 177, in _create_local validate_shape=validate_shape) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/ops/variables.py", line 226, in init expected_shape=expected_shape) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/ops/variables.py", line 344, in _init_from_args self._snapshot = array_ops.identity(self._variable, name="read") File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/ops/gen_array_ops.py", line 1490, in identity result = _op_def_lib.apply_op("Identity", input=input, name=name) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/framework/op_def_library.py", line 768, in apply_op op_def=op_def) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 2402, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/user_2/tensorflow/bazel-bin/tensorflow/examples/image_retraining/retrain.runfiles/org_tensorflow/tensorflow/python/framework/ops.py", line 1264, in init self._traceback = _extract_stack()
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value AUC/AUC/auc/false_positives [[Node: AUC/AUC/auc/false_positives/read = IdentityT=DT_FLOAT, _class=["loc:@AUC/AUC/auc/false_positives"], _device="/job:localhost/replica:0/task:0/cpu:0"]]
But I don't understand why is this because in the main I have this:
init = tf.global_variables_initializer()
sess.run(init)
