How to parse the data after Bert embedding?

Question

I am doing binary classification for title sentences in news. (To determinate whether the new is political biased) I am using the Bert embedding from https://pypi.org/project/bert-embedding/ to embedding training sentences (one raw one title sentence) in Dataframes then feed vectorised Data into logistic regression, but the output data shape from the Bert embedding doesn't support logistic regression model. How can I parse this to make it fit logistic regression model?

Before I used tifdVectorizer it works perfectly and the output is numpy array like

[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]]

each row is vectorised data for one sentence and It's an array with size of 1903 And I have 516 titles in training data. The output shapes are:

train_x.shape: (516, 1903) test_x.shape (129, 1903)
train_y.shape: (516,) test_y.shape (129,)

But after I switched into Bert_Embedding the output vector for ONE row is numpy array list like

[list([array([ 9.79349554e-01, -7.06475616e-01 ...... ]dtype=float32),
 array([ ........ ],dtype=float32), ......................
 array([ ........ ],dtype=float32)]

the output shape is like: train_x.shape: (516, 1) test_x.shape (129, 1) train_y.shape: (516,) test_y.shape (129,)

 def transform_to_Bert(articles_file: str, classified_articles_file: str):
    df = get_df_from_articles_file(articles_file, classified_articles_file)
    df_train, df_test, _, _ = train_test_split(df, df.label, stratify=df.label, test_size=0.2)
    bert_embedding = BertEmbedding()
    df_titles_values=df_train.title.values.tolist()
    result_train = bert_embedding(df_titles_values)
    result_test = bert_embedding(df_test.title.values.tolist())
    train_x = pd.DataFrame(result_train, columns=['A', 'Vector'])
    train_x = train_x.drop(columns=['A'])

    test_x = pd.DataFrame(result_test, columns=['A', 'Vector'])
    test_x=test_x.drop(columns=['A'])
    test_x=test_x.values
    train_x=train_x.values
    print(test_x)
    print(train_x)
    train_y = df_train.label.values
    test_y = df_test.label.values
    return {'train_x': train_x, 'test_x': test_x, 'train_y': train_y, 'test_y': test_y, 'input_length': train_x.shape[1], 'vocab_size': train_x.shape[1]}

Column A is the original title string in the result. So I just drop it.

Below is the code where I use tifd vectoriser which works for logistical model.

def transform_to_tfid(articles_file: str, classified_articles_file: str):
    df = get_df_from_articles_file(articles_file, classified_articles_file)
    df_train, df_test, _, _ = train_test_split(df, df.label, stratify=df.label, test_size=0.2)
    vectorizer = TfidfVectorizer(stop_words='english', )
    vectorizer.fit(df_train.title)
    train_x= vectorizer.transform(df_train.title)
    train_x=train_x.toarray()
    print(type(train_x))
    print(train_x)
    test_x= vectorizer.transform(df_test.title)
    test_x=test_x.toarray()
    print(test_x)
    train_y = df_train.label.values
    test_y = df_test.label.values
    return {'train_x': train_x, 'test_x': test_x, 'train_y': train_y, 'test_y': test_y, 'input_length': train_x.shape[1], 'vocab_size': train_x.shape[1]}


model=LogisticRegression(solver='lbfgs')
model.fit(train_x, train_y)

the error is ValueError: setting an array element with a sequence. I expected output shape from Bert: train_x.shape: (516, 1) test_x.shape (129, 1) is like that from tifd: train_x.shape: (516, 1903) test_x.shape (129, 1903)so that it fits the logistic model

Jianhao Luo Jianhao Luo · Accepted Answer · 2019-10-27T14:46:04

ok that was my mistake or maybe it's bad convention from the library author:

[list([array([ 9.79349554e-01, -7.06475616e-01 ...... ]dtype=float32),
 array([ ........ ],dtype=float32), ......................
 array([ ........ ],dtype=float32)]

actually it's :

[[list([array([ 9.79349554e-01, -7.06475616e-01 ...... ]dtype=float32),
     array([ ........ ],dtype=float32), ......................
     array([ ........ ],dtype=float32)]]

So you gotta index 0 to reach that

How to parse the data after Bert embedding?

1 Answers