Key dnn/hiddenlayer_0/bias not found in checkpoint error

0
votes

I am trying to implement a DNN Classifier with an estimator as shown in this tutorial: http://nbviewer.jupyter.org/gist/yufengg/7dd4170cc730ce418a1dcd141532ec6e

However, whenever I run the code, I am getting this lengthy error message:

Traceback (most recent call last): File "/Users/vivekreddy/Desktop/NN/IRis Practice Estimators.py", line 48, in steps=1000) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 352, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 888, in _train_model log_step_count_steps=self._config.log_step_count_steps) as mon_sess: File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 384, in MonitoredTrainingSession stop_grace_period_secs=stop_grace_period_secs) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 795, in init stop_grace_period_secs=stop_grace_period_secs) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 518, in init self._sess = _RecoverableSession(self._coordinated_creator) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 981, in init _WrappedSession.init(self, self._create_session()) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 986, in _create_session return self._sess_creator.create_session() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 675, in create_session self.tf_sess = self._session_creator.create_session() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 446, in create_session init_fn=self._scaffold.init_fn) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 275, in prepare_session config=config) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 207, in _restore_checkpoint saver.restore(sess, ckpt.model_checkpoint_path) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1755, in restore {self.saver_def.filename_tensor_name: save_path}) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 905, in run run_metadata_ptr) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1137, in _run feed_dict_tensor, options, run_metadata) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run options, run_metadata) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call raise type(e)(node_def, op, message) NotFoundError: Key dnn/hiddenlayer_0/bias not found in checkpoint [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op u'save/RestoreV2', defined at: File "", line 1, in File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/idlelib/run.py", line 115, in main ret = method(*args, **kwargs) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/idlelib/run.py", line 325, in runcode exec code in self.locals File "/Users/vivekreddy/Desktop/NN/IRis Practice Estimators.py", line 48, in steps=1000) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 352, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 888, in _train_model log_step_count_steps=self._config.log_step_count_steps) as mon_sess: File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 384, in MonitoredTrainingSession stop_grace_period_secs=stop_grace_period_secs) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 795, in init stop_grace_period_secs=stop_grace_period_secs) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 518, in init self._sess = _RecoverableSession(self._coordinated_creator) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 981, in init _WrappedSession.init(self, self._create_session()) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 986, in _create_session return self._sess_creator.create_session() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 675, in create_session self.tf_sess = self._session_creator.create_session() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 437, in create_session self._scaffold.finalize() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 214, in finalize self._saver.build() File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1302, in build self._build(self._filename, build_save=True, build_restore=True) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1339, in _build build_save=build_save, build_restore=build_restore) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 790, in _build_internal restore_sequentially, reshape) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 502, in _AddShardedRestoreOps name="restore_shard")) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 449, in _AddRestoreOps restore_sequentially) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 847, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1030, in restore_v2 shape_and_slices=shape_and_slices, dtypes=dtypes, name=name) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op op_def=op_def) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1650, in init self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

NotFoundError (see above for traceback): Key dnn/hiddenlayer_0/bias not found in checkpoint [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

After doing some print statements, I am sure the error is occurring on this line of code: classifier.train(input_fn=input_fn(training_set),steps=1000)

I wonder if it has something to do with my Tensorflow Version, which is 1.6.
Any help appreciated. Thanks.

2
pythonpython-2.7tensorflowdeep-learning
A wild guess: the print() statements in the tutorial imply Python 3 whereas you are using the old Python 2. – cdarke
I don't think that's the case because LinearClassifier works. When I change LinearClassifier to DNNClassifier and add in hidden units, that's when I get the error. – Vivek Reddy
How can I try it out on python 3 though? When I type python into my Terminal I see that Python 2.7.13 is installed. When I type python3 into the terminal, I see that Python 3.6.2 is installed. How do I choose to run Python 3.6.2 specifically when running the program in IDLE? – Vivek Reddy
It depends which IDLE you are running. When you start IDLE it should give the Python version on the first line. – cdarke

2 Answers

5
votes

I think I know the answer. Just delete all the files (including checkpoints) under your model directory. There would be files there if you executed a previous model and did not remove the files after execution.

1
votes

This happens because the same directory contains components of another model. In my case, I had components of a previous linear model and was trying to create a DNN classifier in the same folder. I removed all of them (/temp/iris_model/), and tried again. It worked.

OS: Ubuntu 16.04 LTS
tensorflow: 1.11.0
IDE: jupytor notebook