Neural network with linear activation output. Calculate output range for each of the output neurons

1
votes

Let's assume I have a neural network like the following:

model = keras.models.Sequential()
model.add(keras.layers.Dense(10, input_shape=(5,), activation='relu'))
model.add(keras.layers.Dense(4, activation='linear'))

With n output neurons with a linear activation function.

The training process is not important here, so we can take a look at the random weights that keras initialized using:

model.weights

Of course, in a real example, these weights should be adjusted in the training process.

Depending on these model.weights, each of the output neurons returns values in a range.

I would like to calculate this exact range.

Does keras offer any function to calculate it?

I built a flawed piece of code to make an approximation of it, using a loop and predicting random inputs. But this would not be really useful in a real example with much more inputs/neurons/weights.

Here a few examples trying to clarify my question (All of them assume that the input values are between and 1):

model = keras.models.Sequential()
model.add(keras.layers.Dense(1, input_shape=(2,), 
          activation='linear', use_bias=False))

model.set_weights([np.array([1, 1]).reshape(2, 1)])

For the previous example the output neuron results would be between 0 and 2

model.set_weights([np.array([-0.5, 1]).reshape(2, 1)])

For the previous example the output neuron results would be between -0.5 and 1

model = keras.models.Sequential()
model.add(keras.layers.Dense(2, input_shape=(2,), activation='linear', use_bias=False))
model.add(keras.layers.Dense(1, activation='linear', use_bias=False))

model.set_weights([np.array([1, 1, 1, 1]).reshape(2,2), np.array([1, 1]).reshape(2,1)])

For the previous example, the output neuron results would be between 0 and 4

These are simplified examples. In a real scenario with a much complex network structure, activation functions, bias..... these ranges are not obvious to calculate.

2
pythontensorflowkerasdeep-learningneural-network
sure it does. just needed you to define multiple outputs anywhere you want in your NN and then compile the model. after training, if you fed one sample, you can get the output of layers you chose. – SoheilStar
"Depending on these model.weights, each of the output neurons returns values in a range." is this true, given you're using ReLU activation (which has no upper bound)? Or do you mean practically, as in "the computer can't process values larger than X, which gives us a theoretical upper bound of Y"? – Oso
In regression outputs do not have an upper bound, that's why output layer is linear. You are trying to predict continuous values. Maybe you can add your approximation code so the question will be more clear. – Frightera
Some theoretical insights: math.stackexchange.com/questions/1395404/… – hafiz031
@Frightera I update my question trying to clarify a bit more – Alfonso_MA

2 Answers

1
votes

It sounds like you are roughly interested in what is referred to as neural network verification. This field broadly consists of answering the question: given a range of possible inputs, what is the range of possible outputs from a neural network with a given set of weights? A few things to note:

  1. A neural network is essentially a complex, non-linear function. That is, it maps the input space to the output space. Defining an output range does not make sense except with respect to an input range. In your question you make no reference to the inputs, so your examples are flawed/incomplete.

  2. In general, neural network verification is an emerging field with most published works being fairly recent (last 5-7 years). That being said, there are exact and approximate methods for fully connected networks with a variety of activation functions. I'll list a few such methods here:

https://arxiv.org/abs/2004.05519 - MATLAB toolbox, but you could export your neural network in ONNX format and then use MATLAB for the verification/output range analysis.

https://arxiv.org/abs/1804.10829 - specifically for ReLU activation function.

https://anwu1219.github.io/download/Marabou.pdf with python API available here: https://github.com/NeuralNetworkVerification/Marabou

The field is still evolving so you may have to do some of the codings yourself rather than using pre-existing libraries in some cases, but these papers/ a search query for neural network verification should at least give you some ideas of where to start.

0
votes

IMO, there is no such a function, as far as I know, to estimate the output value's range( without imposing your restriction).
For example, a dense function without bias is just a plain linear function of a=bx, in your case, you are restricting x to 0-1 range and explicitly setting b to your desired values.
You will always get the value in those ranges you`ve cited in your question. A hypothetical example is to choose b randomly and the range in your questions would not hold the ground.

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
model = keras.models.Sequential()
model.add(keras.layers.Dense(1, input_shape=(2,), activation='linear', use_bias=False))
import matplotlib.pyplot as plt
#model.set_weights([np.array([1, 1]).reshape(2, 1)]) 
eval_func = keras.backend.function([model.input], model.layers[-1].output)
outputs = eval_func(np.array([[2,1]]))
counts, bins = np.histogram(outputs)
plt.hist(bins[:-1], bins, weights=counts)

enter image description here