TFX Transform Rank Mismatch While Loading/Applying TFX Beam Transform Graph

2
votes

I've already successfully fit a TFTransformOutput to some data (in this case, the Census dataset from UCI common amongst the TF and TFX examples.) I try to apply the transformer with the method transform_raw_features(raw_features) but keep getting the error:

ValueError: Node 'transform/transform/inputs/workclass_copy' has an _output_shapes attribute inconsistent with the GraphDef for output #0: Shapes must be equal rank, but are 0 and 1

Digging into the source code, it seems the error originates in saved_transform_io in the method _partially_apply_saved_transform_impl while doing:

saver = tf_saver.import_meta_graph(meta_graph_def, import_scope=import_scope,
input_map=input_map)

I examined the meta_graph_def produced by TFX TFTransform and Beam and notice that the graph indeed has a series of copied variables with input/output rank differences. However, that is nothing I have control over.

The column in the error message is "workclass" which is a simple categorical column. What might I be doing incorrectly? What is the best way to debug this? At this point, I've already dug deep into the TF source code but the error seems to originate with how the TFTransform graph was written, not sure what levers I have to change/fix that.

enter image description here

This is using TF Transform v0.9 and the corresponding TF v1.9

Traceback (most recent call last): File "/home/sahmed/workspace/ml_playground/TFX-TFT/trainers.py", line 449, in parse_csv transformed_stuff=xformer.transform_raw_features(raw_features) File "/home/sahmed/miniconda3/envs/kml2/lib/python2.7/site-packages/tensorflow_transform/output_wrapper.py", line 122, in transform_raw_features self.transform_savedmodel_dir, raw_features)) File "/home/sahmed/miniconda3/envs/kml2/lib/python2.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 360, in partially_apply_saved_transform_internal saved_model_dir, logical_input_map, tensor_replacement_map) File "/home/sahmed/miniconda3/envs/kml2/lib/python2.7/site-packages/tensorflow_transform/saved/saved_transform_io.py", line 218, in _partially_apply_saved_transform_impl input_map=input_map) File "/home/sahmed/miniconda3/envs/kml2/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1960, in import_meta_graph **kwargs) File "/home/sahmed/miniconda3/envs/kml2/lib/python2.7/site-packages/tensorflow/python/framework/meta_graph.py", line 744, in import_scoped_meta_graph producer_op_list=producer_op_list) File "/home/sahmed/miniconda3/envs/kml2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 432, in new_func return func(*args, **kwargs) File "/home/sahmed/miniconda3/envs/kml2/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 422, in import_graph_def raise ValueError(str(e)) ValueError: Node 'transform/transform/inputs/workclass_copy' has an _output_shapes attribute inconsistent with the GraphDef for output #0: Shapes must be equal rank, but are 0 and 1

1
tensorflowapache-beamtensorflow-transform

1 Answers

1
votes

The issue is likely that the shape of the workclass tensor is incompatible with what transform_raw_features expects.

TFTransformOutput.transform_raw_features() expects these features to have the same characteristics as described in the metadata given to tft.AnalyzeDataset() similarly to how it's done in this example: https://github.com/tensorflow/transform/blob/master/examples/simple_example.py#L63

Could you take a look at the metadata used in your pipeline and see that it is compatible with the data fed into TFTransformOutput.transform_raw_features()?