Sparse matrix in LSTM

Question

I am building a small LSTM model for binary classification with tf-idf transformation. And I am getting this warning and it's taking a long time to train:

UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "

My code looks like this

xtrain_tfv = tfv.transform(xtrain)
xvalid_tfv = tfv.transform(xvalid)

model = Sequential()
model.add(Embedding(xtrain_tfv.shape[0], 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))

# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

print('Train...')
model.fit(xtrain_tfv, ytrain,
          batch_size=32,
          epochs=15,
          validation_data=(xvalid_tfv, yvalid))
score, acc = model.evaluate(xtest_tfv, ytest,
                            batch_size=32)

xtrain_tfv has a shape of (6851, 9122) . How to handle this?

Can you share sample data? And how large is your vocabulary? — thushv89
Also, why are you using tf-idf when you have an Embedding layer? What are you planning to achieve? — thushv89
Thank you for pointing that out @thushv89 . I think I have mistakenly used tf-idf instead of 'one hot'. I am pretty beginner in the field. Sorry for posting with wrong codes. — mmrbulbul
No, that's okay you can leave it as it is. Someone might find it useful. — thushv89
@mmrbulbul: Can you please confirm if using tf-idf instead of one hot has resolved your issue? Thanks. — Tensorflow Support

Tensorflow Warrior Tensorflow Warrior · Accepted Answer · 2020-06-05T11:07:07

You can ignore the warning if the process which you are doing is not causing the slowness of the model.

The warning can be explained as below.

# dense
array = ['a', None, None, 'c']

# sparse
array = [(0, 'a'), (3, 'c')]

So as you can see if you have a lot of empty entries a sparse array will be much more efficient than a dense one. But if all entries are filled in, dense is far more efficient. In your case somewhere in the tensor flow graph a sparse array is being converted to a dense one of indeterminate size. The warning is just saying it is possible that you can waste a lot of memory like this. But it might not be a problem at all if the sparse array is not too big/already quite dense.

You have mentioned that you have solved the issue using one-hot encoding. But if your vocabulary size is large, one-hot encoding is not suggested since it creates sparse matrix of large dimension.

You can follow the below method to convert your text data into sequences of same length.

import tensorflow as tf
from tf.keras.preprocessing.text import Tokenizer
from tf.keras.preprocessing.sequence import pad_sequences
# prepare tokenizer
t = Tokenizer()
t.fit_on_texts(token_list) 
vocab_size = 10000 # depends on your data
# integer encode the documents
encoded_docs = t.texts_to_sequences(sentences)
max_length = max_len
X = pad_sequences(encoded_docs, maxlen=max_length, padding='post')

For embedding layer, you can use pre-trained word embeddings which will create a desnse vector.You can follow this Tensorflow's documentation for the same.

Sparse matrix in LSTM

1 Answers