ValueError: Cannot reshape a tensor (BERT - transfer learning)

Question

I'm building a multiclass text classification model using HuggingFace's transformers library, using Keras and BERT.

To convert my inputs to the required bert format, I'm using the encode_plus method found in the BertTokenizer class found here

The data is a paragraph of sentences per feature, and has a single label (of 45 labels in total)

The code to convert the inputs is :

def create_input_array(df, tokenizer):
sentences = df.text.values
labels = df.label.values

input_ids = []
attention_masks = []
token_type_ids = []

# For every sentence...
for sent in sentences:
    # `encode_plus` will:
    #   (1) Tokenize the sentence.
    #   (2) Prepend the `[CLS]` token to the start.
    #   (3) Append the `[SEP]` token to the end.
    #   (4) Map tokens to their IDs.
    #   (5) Pad or truncate the sentence to `max_length`
    #   (6) Create attention masks for [PAD] tokens.

    encoded_dict = tokenizer.encode_plus(
        sent,  # Sentence to encode.
        add_special_tokens=True,  # Add '[CLS]' and '[SEP]'
        max_length=128,  # Pad & truncate all sentences.
        pad_to_max_length=True,
        return_attention_mask=True,  # Construct attn. masks.
        return_tensors='tf',  # Return tf tensors.
    )

    # Add the encoded sentence to the list.
    input_ids.append(encoded_dict['input_ids'])

    # And its attention mask (simply differentiates padding from non-padding).
    attention_masks.append(encoded_dict['attention_mask'])

    token_type_ids.append(encoded_dict['token_type_ids'])

return [np.asarray(input_ids, dtype=np.int32),
        np.asarray(attention_masks, dtype=np.int32),
        np.asarray(token_type_ids, dtype=np.int32)]

The model in it's most basic form which still reproduces the error:

model = TFBertForSequenceClassification.from_pretrained(
"bert-base-uncased",
num_labels = labellen,
output_attentions = False, 
output_hidden_states = False
)

Compile and fit:

optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
model.fit(x_train, y[:100], epochs=1, batch_size=3)

The error when I run this :

ValueError: Cannot reshape a tensor with 768 elements to shape [1,1,128,1] (128 elements) for '{{node tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape}} = Reshape[T=DT_FLOAT, Tshape=DT_INT32](tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape/ReadVariableOp, tf_bert_for_sequence_classification_3/bert/embeddings/LayerNorm/Reshape/shape)' with input shapes: [768], [4] and with input tensors computed as partial shapes: input1 = [1,1,128,1].

I understand that BERT converts every token into a 768 value array, but that is the only knowledge I have of that particular number, so I'm stuck on how to proceed.

If anyone has experience with the HuggingFace library, I would also appreciate your thoughts on whether TFBertForSequenceClassification is appropriate for paragraph classification.

Many thanks.

ML_Engine ML_Engine · Accepted Answer · 2020-04-14T09:54:07

In case anybody else needs help with this, it was quite a complex fix but here is what I did:

Changed from using numpy arrays to tf datasets

I don't think this is entirely necessary, so if you're using numpy arrays still then ignore this paragraph and alter the reshape functions below accordingly (from tf.reshape to np reshape methods)

From:

return [np.asarray(input_ids, dtype=np.int32),
         np.asarray(attention_masks, dtype=np.int32),
         np.asarray(token_type_ids, dtype=np.int32)]

To:

input_ids = tf.convert_to_tensor(input_ids)
attention_masks = tf.convert_to_tensor(attention_masks)

 return input_ids, attention_masks

(So the lists are being converted to a tensor)

Call convert inputs function (note omission of the token_type_ids)

From the documentation, the attention masks and token type ids are optional for BERT. In this example I'm only using input_ids and attention_masks

 train_ids, train_masks = create_input_array(df[:], tokenizer=tokenizer)

Reshape inputs

train_ids = tf.reshape(train_ids, (-1, 128, 1) )
train_masks = tf.reshape(train_masks, (-1, 128, 1) )

Convert labels to tensor

labels = tf.convert_to_tensor(y[:])
n_classes = np.unique(y).max() + 1

Import all tensors to a tf dataset

dataset = tf.data.Dataset.from_tensors(( (train_ids, train_masks), labels ))

Load BERT model and add layers

Before all I had was the one-liner model = TFBert... now I'm creating an input layer for each of the input_ids and masks, returning only the first output from the bert layer,flatten, and then add a dense layer.

model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased', trainable=False)

# Input layers
input_layer = Input(shape=(128, ), dtype=np.int32)
input_mask_layer = Input(shape=(128, ), dtype=np.int32)

# Bert layer, return first output
bert_layer = model([input_layer, input_mask_layer])[0]

# Flatten layer
flat_layer = Flatten() (bert_layer)

# Dense layer
dense_output = Dense(n_classes, activation='softmax') (flat_layer)

model_ = Model(inputs=[input_layer, input_mask_layer], outputs=dense_output)

Compile model

optimizer = tf.keras.optimizers.Adam(learning_rate=1e-3, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model_.compile(optimizer=optimizer, loss=loss, metrics=[metric])

Here the entire dataset is passed as the first argument, which also contain the labels.

model_.fit(dataset, epochs=4, batch_size=4, verbose=1)

Hope this helps.

ValueError: Cannot reshape a tensor (BERT - transfer learning)

1 Answers