2GB limit error when training Keras sequential model using Tensorflow dataset

Question

I'm using tf.data.experimental.make_csv_dataset function to create the input to a Keras sequential model. My first layer is a DenseFeature that receives a list of tf.feature_column (indicator, bucketized, numeric etc). The following layers are Dense using relu activation. When I run the fit function I get the error: "Cannot create a tensor proto whose content is larger than 2GB.". What do I need to change to make this model train?

The below is the main part of the code:

train_input = tf.data.experimental.make_csv_dataset(["df_train.csv"], batch_size=64, label_name="loss_rate", num_epochs=1)
eval_input = tf.data.experimental.make_csv_dataset(["df_val.csv"], batch_size=64, label_name="loss_rate", shuffle=False, num_epochs=1)

#all_features is generated by a function (it has 87 tf.feature_column objects)
feature_layer = layers.DenseFeatures(all_features)

def deep_sequential_model():
    model = tf.keras.Sequential([
        feature_layer,
        layers.Dense(64, activation='relu'),
        layers.Dense(32, activation='relu'),
        layers.Dense(1, activation='sigmoid')
    ])

    optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

    model.compile(loss='mse',
                optimizer=optimizer,
                metrics=['mae', 'mse'])

    return model

model = deep_sequential_model()
model.fit(train_input,
          validation_data=eval_input,
          epochs=10)

I'm getting the error:

/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py in __init__(self, node_def, g, inputs, output_types, control_inputs, input_types, original_op, op_def)
   1696             "Cannot create a tensor proto whose content is larger than 2GB.")
   1697       if not _VALID_OP_NAME_REGEX.match(node_def.name):
-> 1698         raise ValueError("'%s' is not a valid node name" % node_def.name)
   1699       c_op = None
   1700     elif type(node_def).__name__ == "SwigPyObject":

ValueError: '_5' is not a valid node name```

Erm... Your error is not "Cannot create a tensor proto whose content is larger than 2GB." Instead, your error is "'_5' is not a valid node name" — sebrockm

user3047030 user3047030 · Accepted Answer · 2019-11-07T00:19:49

I've just found the problem. The csv that I was loading had an index column that was creating a Tensor without any name and I believe this was causing issues. I just removed the index from the csv and it worked.

2GB limit error when training Keras sequential model using Tensorflow dataset

1 Answers