python - Sparse Tensor (matrix) from a dense Tensor Tensorflow

9
votes

I am creating a convolutional sparse autoencoder and I need to convert a 4D matrix full of values (whose shape is [samples, N, N, D]) into a sparse matrix.

For each sample, I have D NxN feature maps. I want to convert each NxN feature map to a sparse matrix, with the maximum value mapped to 1 and all the others to 0.

I do not want to do this at run time but during the Graph declaration (because I need to use the resulting sparse matrix as an input to other graph operations), but I do not understand how to get the indices to build the sparse matrix.

5
pythongraphtensorflowsparse-matrixautoencoder
Do you want to do this conversion in Tensorflow or in python? If in python This function can help you convert from dense to sparse matrix (docs.scipy.org/doc/scipy/reference/generated/…) And you can use tf.SparseTensor (which uses coo format) to store each feature map, and use a list to store all sparse tensors. - Yao Zhang
Specifically, nonzero() (docs.scipy.org/doc/scipy/reference/generated/…) can give you the indices for nonzero elements. Not sure if this is considered the runtime approach. This could be some data preprocessing before the graph declaration. Is the 4D dense matrix generated at runtime or simply some given input data? - Yao Zhang
I don't want to do that at runtime (I know how to do that with numpy) but during the graph declaration (so with Tensorflow) - user4706825

5 Answers

16
votes

You can use tf.where and tf.gather_nd to do that:

import numpy as np
import tensorflow as tf

# Make a tensor from a constant
a = np.reshape(np.arange(24), (3, 4, 2))
a_t = tf.constant(a)
# Find indices where the tensor is not zero
idx = tf.where(tf.not_equal(a_t, 0))
# Make the sparse tensor
# Use tf.shape(a_t, out_type=tf.int64) instead of a_t.get_shape()
# if tensor shape is dynamic
sparse = tf.SparseTensor(idx, tf.gather_nd(a_t, idx), a_t.get_shape())
# Make a dense tensor back from the sparse one, only to check result is correct
dense = tf.sparse_tensor_to_dense(sparse)
# Check result
with tf.Session() as sess:
    b = sess.run(dense)
np.all(a == b)
>>> True
3
votes

Simple code to convert dense numpy array to tf.SparseTensor:

def denseNDArrayToSparseTensor(arr):
  idx  = np.where(arr != 0.0)
  return tf.SparseTensor(np.vstack(idx).T, arr[idx], arr.shape)
1
votes

Tensorflow has tf.sparse.from_dense since 1.15. Example:

In [1]: import tensorflow as tf

In [2]: x = tf.eye(3) * 5

In [3]: x
Out[3]: 
<tf.Tensor: shape=(3, 3), dtype=float32, numpy=
array([[5., 0., 0.],
       [0., 5., 0.],
       [0., 0., 5.]], dtype=float32)>

Applying tf.sparse.from_dense:

In [4]: y = tf.sparse.from_dense(x)

In [5]: y.values
Out[5]: <tf.Tensor: shape=(3,), dtype=float32, numpy=array([5., 5., 5.], dtype=float32)>

In [6]: y.indices
Out[6]: 
<tf.Tensor: shape=(3, 2), dtype=int64, numpy=
array([[0, 0],
       [1, 1],
       [2, 2]])>

Verify identity by applying tf.sparse.to_dense:

In [7]: tf.sparse.to_dense(y) == x
Out[7]: 
<tf.Tensor: shape=(3, 3), dtype=bool, numpy=
array([[ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True]])>
0
votes

Note there is a built in function in the contrib (taken from )

0
votes

In TF 2.3 Tensorflow Probability has a function for this:

import tensorflow_probability as tfp

tfp.math.dense_to_sparse(x, ignore_value=None, name=None)