'ValueError: Encountered a SparseTensorValue that cannot be decoded by ListColumnRepresentation' when working with Tensorflow Transform / Beam

Question

I am running a tensorflow transform / beam pipeline to load/preprocess and save as TFRecords. Those records are then loaded. During preprocessing in Tensorflow Transform, I want to pad a sparse tensor. Thus I want to convert it to dense, pad it and transform it back to sparse.
The code looks somewhat like this:

import tensorflow_transform as tft
import tensorflow as tf

#...
def preprocess_fn(input_features):
    output_features = {}
    output_features[CATEGORICAL_FEATURE_NAMES] = tft.compute_and_apply_vocabulary(...)
    #dense = tf.sparse.to_dense(output_features[CATEGORICAL_FEATURE_NAMES])
    ## do something with dense
    #output_features[CATEGORICAL_FEATURE_NAMES] = tf.contrib.layers.dense_to_sparse(dense)

    return output_features

To load the TFRecords I use the following funciton:

def tfrecords_input_fn(files_name_pattern, transformed_metadata,
                       mode=tf.estimator.ModeKeys.EVAL,
                       num_epochs=1,
                       batch_size=64):
    dataset = tf.data.experimental.make_batched_features_dataset(
        file_pattern=files_name_pattern,
        batch_size=batch_size,
        features=transformed_metadata.schema.as_feature_spec(),
        reader=tf.data.TFRecordDataset,
        num_epochs=num_epochs,
        shuffle=True if mode == tf.estimator.ModeKeys.TRAIN else False,
        shuffle_buffer_size=1 + (batch_size * 2),
        prefetch_buffer_size=1,
        drop_final_batch=True
    )

    iterator = dataset.make_one_shot_iterator()
    features = iterator.get_next()
    target = features.pop(TARGET_FEATURE_NAME)

    return features, target

Running the whole pipeline (loading raw data, transforming, saving TFRecords, then loading those to just print them to the screen) works fine but uncommenting the 2 lines in "preprocess_fn" leads to the following errors:

File ".../lib/python3.6/site-packages/tensorflow_transform/impl_helper.py", line 262, in to_instance_dicts raise ValueError('Encountered a SparseTensorValue that cannot be ' ValueError: Encountered a SparseTensorValue that cannot be decoded by ListColumnRepresentation.

...

ValueError: Encountered a SparseTensorValue that cannot be decoded by ListColumnRepresentation. [while running '%s - Transform/ConvertAndUnbatch']

Does anyone have advice on this code or any hints on what I missed? Any help is very much appreciated!

Best, Dominik

Zohar Zohar · Accepted Answer · 2019-06-15T17:02:40

tf.Transform supports sparse tensors that are ragged, so each row should be left aligned for both the input and output.

'ValueError: Encountered a SparseTensorValue that cannot be decoded by ListColumnRepresentation' when working with Tensorflow Transform / Beam

1 Answers