Getting memory error in Dictionary Learning using scikit learn

Question

I have 50 images of size 1028x1028. I was trying to make a dictionary by taking few patches randomly from 50 images.

here is my code =>

from os import listdir
from time import time
import matplotlib.pyplot as plt
import numpy as np
import scipy as sp
from sklearn.decomposition import MiniBatchDictionaryLearning
from sklearn.feature_extraction.image import extract_patches_2d
from sklearn.feature_extraction.image import reconstruct_from_patches_2d
from sklearn.utils.fixes import sp_version
from sklearn.datasets import load_sample_image
from scipy import ndimage
from skimage import color
from skimage import io
from PIL import Image
from sklearn.decomposition import SparseCoder
from sklearn.decomposition import sparse_encode
from skimage import data,restoration
from scipy.misc import imfilter, imread
from scipy.signal import convolve2d as conv2
import sys
from sklearn.feature_extraction import image


x = []

path = 'resize/'
c=0
for e in listdir(path):
    matrix = np.asarray(Image.open(path+e).convert('L')) 
    x.append(matrix)    

images = np.array(x)

input = np.asarray(Image.open('H03.bmp').convert('L')) 
height , width = input.shape

patchsize = (7,14)
patches = image.PatchExtractor((7,14),10000,10).transform(images)
print(patches.shape)

data = patches.reshape(patches.shape[0], -1)
n_iter = 1000
dico = MiniBatchDictionaryLearning(n_components=100,alpha=3,n_iter=n_iter)

V = dico.fit_transform(data).components_

but in the last line I am getting MemoryError. This is the error =>

(480000, 7, 14) Traceback (most recent call last): File "new.py", line 63, in

V = dico.fit_transform(data).components_ File "/usr/local/lib/python3.4/dist-packages/sklearn/base.py", line 494, in fit_transform

return self.fit(X, **fit_params).transform(X) File "/usr/local/lib/python3.4/dist-packages/sklearn/decomposition/dict_learning.py", line 1238, in fit

return_n_iter=True) File "/usr/local/lib/python3.4/dist-packages/sklearn/decomposition/dict_learning.py", line 677, in dict_learning_online

random_state=random_state) File "/usr/local/lib/python3.4/dist-packages/sklearn/utils/extmath.py", line 364, in randomized_svd

power_iteration_normalizer, random_state) File "/usr/local/lib/python3.4/dist-packages/sklearn/utils/extmath.py", line 258, in randomized_range_finder

Q, _ = linalg.lu(safe_sparse_dot(A, Q), permute_l=True) File "/usr/local/lib/python3.4/dist-packages/sklearn/utils/extmath.py", line 189, in safe_sparse_dot

return fast_dot(a, b) MemoryError

I don't know why I am getting this error??

MemoryError happens when you run out of memory basically. You don't have enough RAM. What is your system configuration? — Vikash Singh
ok Mr. @VikashSingh I am taking only 10 image. let see what happen wait a min... — Sudip Das

Vikash Singh Vikash Singh · Accepted Answer · 2017-02-25T09:53:28

As discussed in comments.

MemoryError happens when you run out of memory basically. You don't have enough RAM. What is your system configuration?

OP replied:

I think, you are r8.... if i take few images around 10 images. then its working fine thanks to help. –

Basically user is running out of RAM space.

Getting memory error in Dictionary Learning using scikit learn

1 Answers