I am new to TPU and I encounter an error when I try to train a CNN (on TPU) on my own dataset. I am able to run MNIST example and run my code when --use_tpu=False. But when I set --use_tpu=Ture, I will have
InvalidArgumentError
.
I System
The tf-1-7 image, as mentioned in the tutorial.
II Error
2018-04-17 06:35:31.602915: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1327, in _do_call return fn(*args) File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1310, in _run_fn self._extend_graph() File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1358, in _extend_graph graph_def.SerializeToString(), status)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'input_pipeline_task0/IteratorToStringHandle': Operation was explicitly assigned to /job:tpu_worker/task:0/device:CPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
[[Node: input_pipeline_task0/IteratorToStringHandle = IteratorToStringHandle_device="/job:tpu_worker/task:0/device:CPU:0"]]
My thought is that it is caused by the way I input the data. Now, I use pickle to load the data in the local machine and then use from_tensor_slices to import data. I tried to save data in google cloud storage but it will report "the file cannot be found."
Any solutions? Or better way to import data?
Many thanks!
