How to use pytorch's cat function for K-Fold Validation (i.e. concatenate a list of pytorch chunks together)

Question

I'm working on splitting up a data set for k-fold cross validation but having trouble with concatenating a list of tensors using Pytorch's stack/cat functions.

First, I split the training and test set into chunks using the .chunk method as follows

  x_train_folds = torch.chunk(x_train, num_folds)
  y_train_folds = torch.chunk(y_train, num_folds)

Where x_train is a tensor of torch.Size([5000, 3, 32, 32]) and y_train is a tensor of torch.Size([5000])

x_train_folds and y_train_folds are now tuples of num_folds tensors

Then, I need to setup a series of nested loops to iterate through the different values for K, and the various folds, while always excluding one fold from the training set to be used at test/validation time:

  for k in k_choices:
    k_to_accuracies[k] = [] # create empty space to append for a given k-value
    for fold in range(num_folds):
      # create training sets by excluding the current loop index fold and using that as the test set
      x_train_cross_val = torch.cat((x_train_folds[:fold], x_train_folds[fold+1:]), 0)
      y_train_cross_val = torch.cat((y_train_folds[:fold], y_train_folds[fold+1:]), 0)
      classifier = KnnClassifier(x_train_cross_val, y_train_cross_val)
      k_to_accuracies[k].append(classifier.check_accuracy(x_train_folds[fold], y_train_folds[fold], k=k))

As you can see, I'm always skipping one fold from the original training set to be used for validation. This is standard K-fold cross validation.

Unfortunately, I am getting the following error which I cannot seem to figure out: TypeError: expected Tensor as element 0 in argument 0, but got tuple

As you can see in the API listing, .cat seems to need a tuple of tensors which is what I have. https://pytorch.org/docs/stable/torch.html#torch.cat

Does anyone have any suggestions?

Much appreciated -Drew

one one · Accepted Answer · 2020-06-24T00:10:54

try:

x_train_cross_val = torch.cat((*x_train_folds[:fold], *x_train_folds[fold+1:]), 0)
y_train_cross_val = torch.cat((*y_train_folds[:fold], *y_train_folds[fold+1:]), 0)

torch.cat receives a tuple whose elements are torch.Tensor type. However, the elements in your tuple x_train_folds[:fold] are still tuple. So, you need to remove the tuple 'decorator' of your tensors.

How to use pytorch's cat function for K-Fold Validation (i.e. concatenate a list of pytorch chunks together)

1 Answers