Training SVM with word2vector data

Question

Input text data is tokenised:

data['tokenised'] ['hasan', 'minhaj', 'responds', 'netflix', 'pulling', 'episode', 'comedy', 'show', 'saudi', 'arab']

Data is padded with 0s to make all the tokenised texts of equal length(51 in this case):

len(data['tokenised'][0])
51

Word vectors of 100 dimensions are called: embeddings_index = dict() f = open('glove.6B.100d.txt') for line in f: values = line.split() word = values[0] coefs = np.asarray(values[1:], dtype='float32') embeddings_index[word] = coefs f.close()

Input data tokens are converted to their vector form:

def word2vec(tokens,max_size,dim):
    print(tokens)
    vec = np.zeros((max_size,dim))
    for ind,tok in enumerate(tokens):
        if(tok==0):
            vec[ind] = vec[ind]
        else:
            try:
                print(ind)
                vec[ind] = embeddings_index[tok]
            except KeyError:
                continue
return vec


data['w2v'][0]
array([[-0.41133001, -0.20108999, -0.54119998, ..., -0.67202002,
     0.14799   , -0.055051  ],
   [ 0.049478  ,  0.26212001, -0.78268999, ..., -0.14226   ,
    -0.32286   ,  0.13525   ],
   [-0.14078   ,  0.6573    ,  0.44602001, ..., -0.55290002,
     0.19839001,  0.39563   ],
   ...,
   [ 0.        ,  0.        ,  0.        , ...,  0.        ,
     0.        ,  0.        ],
   [ 0.        ,  0.        ,  0.        , ...,  0.        ,
     0.        ,  0.        ],
   [ 0.        ,  0.        ,  0.        , ...,  0.        ,
     0.        ,  0.        ]])

Now i have each text with length 51 words and each word is represented by 100 dimensions vector(as shown in data[‘w2v’]). data['w2v'][0].shape (51, 100) All the arrays are of same dimension i.e. (51,100) and all the array elements are of float type.

data[‘w2v’] is a column of dataframe.

Split data in train-test : x_train,x_test,y_train,y_test =

train_test_split(data['w2v'],data['class'],test_size=0.2,stratify=data['class'])

x_train.shape    #series data type
(10248,)

x_train[7].shape   #2D array
(51, 100)

On fitting SVM model with 2D numpy array i get below error: model = LinearSVC(C=0.3) model.fit(x_train,y_train) ValueError: setting an array element with a sequence.

Note: All the numpy array are of same shape (51,100).

Please suggest how this error should be handled? How should i modify x_train so that model can be trained?

Proyag Proyag · Accepted Answer · 2019-02-19T11:01:28

Your x_train is currently three-dimensional. What you've done so far results in each training example x_train[i] being a (51,100) array, i.e. the shape of x_train is (n_samples, 51, 100).

When you're calling the fit method, x_train needs to have the shape (n_samples, n_features) (straight from the docs). So you need to reduce the 51x100 array for each input into a 1-d array/vector. You can do this by -

Simply reshaping your (51,100) shaped arrays into (5100)-sized vectors, so that your x_train is shaped (n_samples, 5100), or
By combining the embeddings in some way, like averaging them or something to get a smaller number of features. For example, you could sum/average over each (51,100)-shaped array to get (51)-sized feature vectors. I mention this possibility because averaging word embeddings to get a sentence embedding is a very rough but commonly-used baseline technique (this is of course not generally applicable to convert input shapes).

Training SVM with word2vector data

1 Answers