How 2 keras model outputs can be integrated?

Question

I've been thinking if I were to build multiple keras models that have different objectives, I will illustrate with an example to explain what I mean. Let's assume the main objective is to predict the price for a stock sign, let's say there will be 2 models, the first model is for sentiment analysis and its main objective is to process news articles and predict whether the stock price will go up or down. The second model would be an LSTM that takes as input the historical prices data and predicts the next period (which can be a minute, a day, a month ...) price. Let's say I want the LSTM model to consider as well the outcome of sentiment analysis and possibly the output of other models that predict different metrics in the same fashion, is this possible? and how is it usually done? I think the same logic might apply on many other examples (recommender systems, retail inventory prediction ...)

You can put all those predicted features like any other feature to a model. Also, you can check up 'ensemble learning', it's usually for combining models that predict the same outcome, but you can use some principles from there — user2827262
Thanks, can you point me to some documentation you think is relevant? — sK500
towardsdatascience.com/… ; scikit-learn.org/stable/modules/ensemble.html . For the first thing I was saying; you would have to combine all these features into some x-axis, like time. — user2827262

mb0850 mb0850 · Accepted Answer · 2020-10-03T22:29:34

The Keras Functional API helps you to create more flexible models including the multiple input pipelines.

Per your requirement, You have a RNN (LSTM) model which would do sequence processing of stock prices (numerical data) and another CNN or RNN model doing Text processing of news articles (text data). With the Keras Functional API, you could create a Model that will have these 2 modules in separate Input Pipelines which you could then merge to predict stock price.

Example of a Shoe price predictor Model:

In the figure you have 3 input pipelines (say specifications of a shoe, its description and the picture(s)), which would then be merged by layers.concatenate() to form the model to predict the price of the shoe.

TF Keras Functional API Guide

Example implementation of a 2-input question-answering model:

from tensorflow.keras.models import Model
from tensorflow.keras import layers
from tensorflow.keras import Input

text_vocabulary_size = 10000
question_vocabulary_size = 10000
answer_vocabulary_size = 500

text_input = Input(shape=(None,), dtype='int32', name='text')

embedded_text = layers.Embedding(64, text_vocabulary_size)(text_input)

encoded_text = layers.LSTM(32)(embedded_text)

question_input = Input(shape=(None,),
   dtype='int32',
   name='question')

embedded_question = layers.Embedding(
    32, question_vocabulary_size)(question_input)

encoded_question = layers.LSTM(16)(embedded_question)

concatenated = layers.concatenate([encoded_text, encoded_question],
   axis=-1)
answer = layers.Dense(answer_vocabulary_size,
   activation='softmax')(concatenated)

model = Model([text_input, question_input], answer)

model.compile(optimizer='rmsprop',
  loss='categorical_crossentropy',
  metrics=['acc'])

Refer Deep Learning with Python Book by François Chollet Chap 17.1

How 2 keras model outputs can be integrated?

1 Answers