I am unable to read any CSV file using Pandas on PyCharm, despite having installed pandas

Question

import pandas as pd
import numpy as np
data = pd.read_csv('dataset.csv')
print(data.head())

Traceback (most recent call last): File "C:/Users/Shantanu Shubham/PycharmProjects/dash/dashboard.py", line 6, in data = pd.read_csv('gapminder.csv') File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\io\parsers.py", line 685, in parser_f return _read(filepath_or_buffer, kwds) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\io\parsers.py", line 463, in _read data = parser.read(nrows) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\io\parsers.py", line 1169, in read df = DataFrame(col_dict, columns=columns, index=index) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\core\frame.py", line 411, in init mgr = init_dict(data, index, columns, dtype=dtype) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\core\internals\construction.py", line 257, in init_dict return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\core\internals\construction.py", line 87, in arrays_to_mgr return create_block_manager_from_arrays(arrays, arr_names, axes) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\core\internals\managers.py", line 1694, in create_block_manager_from_arrays blocks = form_blocks(arrays, names, axes) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\core\internals\managers.py", line 1752, in form_blocks float_blocks = _multi_blockify(items_dict["FloatBlock"]) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\core\internals\managers.py", line 1846, in _multi_blockify values, placement = _stack_arrays(list(tup_block), dtype) File "C:\Users\Shantanu Shubham\PycharmProjects\dash\venv\lib\site-packages\pandas\core\internals\managers.py", line 1874, in _stack_arrays stacked = np.empty(shape, dtype=dtype) MemoryError: Unable to allocate array with shape (535, 79309) and data type float64

Process finished with exit code 1

Answer 1

This means that the file you are reading is larger than the amount of RAM you have available. There are ways to split the file into "chunks" using generators which you can read up on.

Answer 2

Are you sure the file you're importing has correct content/structure?

The error message 'Unable to allocate array with shape (535, 79309)' makes me think there's something wrong with the content of the file.

Here's how the shape function works:

from numpy import array
data = [[11, 22], [33, 44], [55, 66]]
data = array(data)
print(data.shape)

end you end up with:

(3, 2)

Now, the shape of the array you're trying to read is (535, 79309) - quite big, huh?