Keras extending embedding layer input

Question

A keras sequential model with embedding needs to be retrained starting from the currently known weights.

A Keras sequential model is trained on the provided (text) training data. The training data is tokenized by a (custom made) tokenizer. The input dimension for the first layer in the model, a embedding layer, is the number of words known by the tokenizer.

After a few days additional training data becomes available. The tokenizer needs to be refitted on this new data as it may contain additional words. That means that the input dimension of the embedding layer changes, so the previously trained model is not usable anymore.

self.model = Sequential()
self.model.add(Embedding(tokenizer.totalDistinctWords + 1, 
    hiddenSize + 1, batch_size=1, 
    input_length=int(self.config['numWords'])))
self.model.add(LSTM(hiddenSize, return_sequences=True, 
    stateful=True, activation='tanh', dropout = dropout))
self.model.add(LSTM(hiddenSize, return_sequences=True, 
    stateful=True, activation='tanh', dropout = dropout))

self.model.add(TimeDistributed(Dense(
    len(self.controlSupervisionConfig.predictableOptionsAsList))))
self.model.add(Activation('softmax'))

I want to use the previously trained model as initializer for the new training session. For the new words in the tokenizer, the embedding layer should just use a random initialization. For the words already known by the tokenizer, it should use the previously trained embedding.

aragaer aragaer · Accepted Answer · 2019-12-11T18:38:02

You can access (get and set) the layer's weights directly as numpy arrays with code like weights = model.layers[0].get_weights() and model.layers[0].set_weighs(weights, where model.layers[0] is your Embedding layer. This way you can store embeddings separately and set known embeddings by copying them from stored data.

Keras extending embedding layer input

1 Answers