I have a gigantic, nested JSON file that I need to parse. I'm currently doing it in Pandas, but would like to find out if I can implement it in Dask for speed/scalability.
My current pandas code is:
import pandas as pd
json_data = pd.read_json(input_file)
feature_df = json_normalize(json_obj["Features"],
record_path=["Cycles","Pools"],
meta=["FeatureID"])
I've gotten as far as the following code in Dask, but I cannot find any more info on how to do this on the interwebs:
import dask.bag as db
import json
json_data = db.read_text(input_file).map(json.loads)
This returns a dask bag that I need to figure out how to properly parse.
My JSON is structured as follows:
{
"FovID":1,
"Features":[
{
"FeatureID":"1_166_155",
"X":166,
"Y":155,
"Cycles":[
{
"CycleID":31,
"Pools":[
{
"PoolID":1,
"BC":"1224",
"Qual":"999",
"Category":"000"
},
{
"PoolID":2,
"BC":"0",
"Qual":"999",
"Category":"000"}
}]}
Solution: Instead of using a dask bag, I followed @mdurant's advice and read the JSON directly into a dataframe as follows:
json_data = dd.read_json(self.input_s6, orient='columns').compute()