'BertEmbeddings' object has no attribute 'bias' while converting tf checkpoint

Question

When trying to convert the checkpoint of a self pre-trained tensorflow BERT model (using the create-pretraining.py script from google) into a pytorch model using the convert_bert_original_tf_checkpoint_to_pytorch.py from Huggingface.

I always end up with the following error: AttributeError: 'BertEmbeddings' object has no attribute 'bias'

The init_vars names (just the first ones) look like this:

['bert/embeddings/layer_normalization/beta', 'bert/embeddings/layer_normalization/beta/adam_m', 'bert/embeddings/layer_normalization/beta/adam_v', 'bert/embeddings/layer_normalization/gamma', 'bert/embeddings/layer_normalization/gamma/adam_m', 'bert/embeddings/layer_normalization/gamma/adam_v']

Code that produces the error looks like this:

for m_name in name:                                                     
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):                          
                scope_names = re.split(r"_(\d+)", m_name)                       
            else:                                                               
                scope_names = [m_name]                                          
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":         
                pointer = getattr(pointer, "weight")                            
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":   
                print(scope_names)                                              
                pointer = getattr(pointer, "bias")                              
            elif scope_names[0] == "output_weights":                            
                pointer = getattr(pointer, "weight")                            
            elif scope_names[0] == "squad":                                     
                pointer = getattr(pointer, "classifier")                        
            else:                                                               
                try:                                                            
                    pointer = getattr(pointer, scope_names[0])                  
                except AttributeError:                                          
                    logger.info("Skipping {}".format("/".join(name)))

Going through all the names and getting the right attributes from the model. When it comes to the Layer Normalization in the BertEmbeddings the script produces an error. Did anyone else encouter that error before? How did you fix this?

Here again the whole stacktrace:

Traceback (most recent call last):
  File "convert_bert_original_tf_checkpoint_to_pytorch.py", line 62, in <module>
    convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path)
  File "convert_bert_original_tf_checkpoint_to_pytorch.py", line 37, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/modeling_bert.py", line 136, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "module.py", line 594, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertEmbeddings' object has no attribute 'bias'

Bert Config is the following:

Building PyTorch model from configuration: BertConfig {
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 512,
  "initializer_range": 0.02,
  "intermediate_size": 2048,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 8,
  "num_hidden_layers": 8,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

blueberry-cake666 blueberry-cake666 · Accepted Answer · 2020-09-02T08:27:38

Turns out it's just the naming of "layer_normalization" instead of "LayerNorm". I just changed the script and now it works.

'BertEmbeddings' object has no attribute 'bias' while converting tf checkpoint

2 Answers