Colab+Keras+TensorBoard FailedPreconditionError

2
votes

I'm trying to run a simple Keras script and use Google Colab with TensorBoard. Here's my code: 

import tensorflow as tf
import tensorflow.keras as keras
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.applications.mobilenet import MobileNet
from tensorboardcolab import TensorBoardColab, TensorBoardColabCallback

# Settings
num_classes = 10
batch_size = 16
epochs = 1

# Data setup
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

# Select model
model = MobileNet(weights=None, input_shape=x_train.shape[1:], classes=num_classes)

# Select loss, optimizer, metric
model.compile(loss='categorical_crossentropy',
                            optimizer=tf.train.AdamOptimizer(0.001),
                            metrics=['accuracy'])    
# Train
tbc=TensorBoardColab()
model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test, y_test), 
                    callbacks=[TensorBoardColabCallback(tbc)])

This is a suggestion I saw to use TensorBoard with Colab as referenced here: Can I use Tensorboard with Google Colab?

However, when adding the callback I get the error:

FailedPreconditionError: Error while reading resource variable conv_dw_8_2/depthwise_kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/conv_dw_8_2/depthwise_kernel/N10tensorflow3VarE does not exist. [[Node: conv_dw_8_2/depthwise/ReadVariableOp = ReadVariableOpdtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]] [[Node: loss_2/mul/_147 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6752_loss_2/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Does anybody know what I'm doing wrong? This seems like a very useful way to run TensorBoard on Colab if I could get it working.

1
python-3.xkerastensorboardgoogle-colaboratory

1 Answers

1
votes

This is caused by conflicting versions of Keras. Tensorboardcolab uses the full keras library while you import the tf.keras implementation of the Keras API. So when you fit the model you end up using two different versions of keras.

You have a few options:

Use Keras libary and change your imports

import tensorflow as tf
import keras
from keras.datasets import cifar10
from keras.applications.mobilenet import MobileNet
from tensorboardcolab import TensorBoardColab, TensorBoardColabCallback

Although the code runs fine with these changes, you might consider using Keras's version of the Adam optimizer, so you don't need to import tensorflow explicitly anymore.

model.compile(loss='categorical_crossentropy', 
                    optimizer=keras.optimizers.Adam(lr=0.001), 
                    metrics=['accuracy'])`

Use tf.keras and patch TensorBoardColab

Your code runs fine, if you'd patch callbacks.py and core.py and fix the imports there:

from keras.callbacks import TensorBoard from tensorflow.keras.callbacks import TensorBoard

You could also use this fork where I made these changes.