A minimal example here: I have a json file xaa.json whose contents looks like this (two rows from stackoverflow archive):
[
{"Id": 11, "Body": "<p>Given a specific <code>DateTime</code> value", "Title": "Calculate relative time in C#", "Comments": "There is the .net package https://github.com/NickStrupat/TimeAgo which pretty much does what is being asked."},
{"Id": 7888, "Body": "<p>You need to use an <a href=\\"http://en.cppreference.com/w/cpp/io/basic_ifstream\\" rel=\\"noreferrer\\"><code>ifstream</code></a> if you just want to read (use an <code>ofstream</code> to write, or an <code>fstream</code> for both).</p>

<p>To open a file in text mode, do the following:</p>

<pre><code>ifstream in(\\"filename.ext\\", ios_base::in); // the in flag is optional
</code></pre>

<p>To open a file in binary mode, you just need to add the \\"binary\\" flag.</p>

<pre><code>ifstream in2(\\"filename2.ext\\", ios_base::in | ios_base::binary ); 
</code></pre>

<p>Use the <a href=\\"http://en.cppreference.com/w/cpp/io/basic_istream/read\\" rel=\\"noreferrer\\"><code>ifstream.read()</code></a> function to read a block of characters (in binary or text mode). Use the <a href=\\"http://en.cppreference.com/w/cpp/string/basic_string/getline\\" rel=\\"noreferrer\\"><code>getline()</code></a> function (it's global) to read an entire line.</p>
", "Title": null, "Comments": "+1 for noting that the global getline() function is to be used instead of the member function."}
]
I want to load such json files into a dask dataframe. I use:
so_posts_df = dd.read_json('./xaa.json', orient='columns').compute()
I get this error:
ValueError: Unexpected character found when decoding object value
After looking into the contents, I figured that the "\\"' stuff was causing it. So, when I removed them, (the editor - IntelliJ said it was clean and nice looking JSON) and when I ran the same read_json, it was able to read into a df and display them nicely.
So, I have 2 questions: (a) what are the values for the read_json argument "errors" ? (b) How can I properly preprocess the json file before reading into dask dataframe? The presence of double-quotes and the double-escaping seems to be causing an issue.
[This may not be a dask issue at all...]...