UnicodeDecodeError when reading a dataset from a webpage into a Pandas dataframe

Question

I am trying to copy the MovieLens dataset into a Pandas dataframe in Python.

movies = pd.read_csv('http://files.grouplens.org/datasets/movielens/ml-100k/u.item', sep='|', names = ['movie_id', 'title'], usecols = range(2))
movies.head()

However, I get the following error when I do the above.

UnicodeDecodeError Traceback (most recent call last) pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_tokens()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_with_dtype()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._string_convert()

pandas/_libs/parsers.pyx in pandas._libs.parsers._string_box_utf8()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

During handling of the above exception, another exception occurred:

UnicodeDecodeError Traceback (most recent call last) in () 8 9 ---> 10 movies = pd.read_csv('http://files.grouplens.org/datasets/movielens/ml-100k/u.item', sep='|', names = ['movie_id', 'title'], usecols = range(2)) 11 movies.head()

/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace, as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map, float_precision) 703 skip_blank_lines=skip_blank_lines) 704 --> 705 return _read(filepath_or_buffer, kwds) 706 707 parser_f.name = name

/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py in _read(filepath_or_buffer, kwds) 449 450 try: --> 451 data = parser.read(nrows) 452 finally: 453 parser.close()

/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py in read(self, nrows) 1063 raise ValueError('skipfooter not supported for iteration') 1064 -> 1065 ret = self._engine.read(nrows) 1066 1067 if self.options.get('as_recarray'):

/usr/local/lib/python3.5/dist-packages/pandas/io/parsers.py in read(self, nrows) 1826 def read(self, nrows=None): 1827 try: -> 1828 data = self._reader.read(nrows) 1829 except StopIteration: 1830 if self._first_chunk:

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.read()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_rows()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_column_data()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_tokens()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_with_dtype()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._string_convert()

pandas/_libs/parsers.pyx in pandas._libs.parsers._string_box_utf8()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

I am working in a Jupyter notebook in Python 3. How can I resolve the error?

\xe9 is the é character encoding for native windows text encodings, not UTF-8. Using movies = pd.read_csv('http://files.grouplens.org/datasets/movielens/ml-100k/u.item', sep='|', names = ['movie_id', 'title'], usecols = range(2), encoding='cp1250') might work. — Abdou

DSM DSM · Accepted Answer · 2017-12-22T17:44:52

That file isn't UTF-8 encoded. We can make a reasonable guess, though, and we can ask the chardet third-party library to help us out:

In [23]: import urllib, chardet

In [24]: url = 'http://files.grouplens.org/datasets/movielens/ml-100k/u.item'

In [25]: u = chardet.UniversalDetector()

In [26]: u.feed(urllib.request.urlopen(url).read())

In [27]: u.close()
Out[27]: {'confidence': 0.73, 'encoding': 'ISO-8859-1', 'language': ''}

In [28]: movies = pd.read_csv(url, sep='|', names = ['movie_id', 'title'], 
                              usecols = range(2), encoding='iso-8859-1')

In [29]: movies.head()
Out[29]: 
   movie_id              title
0         1   Toy Story (1995)
1         2   GoldenEye (1995)
2         3  Four Rooms (1995)
3         4  Get Shorty (1995)
4         5     Copycat (1995)

UnicodeDecodeError when reading a dataset from a webpage into a Pandas dataframe

2 Answers