2
votes

I have a gigantic, nested JSON file that I need to parse. I'm currently doing it in Pandas, but would like to find out if I can implement it in Dask for speed/scalability.

My current pandas code is:

import pandas as pd
json_data = pd.read_json(input_file)
feature_df = json_normalize(json_obj["Features"], 
    record_path=["Cycles","Pools"],
    meta=["FeatureID"])

I've gotten as far as the following code in Dask, but I cannot find any more info on how to do this on the interwebs:

import dask.bag as db
import json
json_data = db.read_text(input_file).map(json.loads)

This returns a dask bag that I need to figure out how to properly parse.

My JSON is structured as follows:

{
"FovID":1,
"Features":[
    {
        "FeatureID":"1_166_155",
        "X":166,
        "Y":155,
        "Cycles":[
            {
                "CycleID":31,
                "Pools":[
                    {
                        "PoolID":1,
                        "BC":"1224",
                        "Qual":"999",
                        "Category":"000"
                    },
                    {
                        "PoolID":2,
                        "BC":"0",
                        "Qual":"999",
                        "Category":"000"}
                 }]}

Solution: Instead of using a dask bag, I followed @mdurant's advice and read the JSON directly into a dataframe as follows:

json_data = dd.read_json(self.input_s6, orient='columns').compute()
1

1 Answers

3
votes

You could indeed use the bag API to do your work, but it so happens that the dask-dataframe API also has a read_json function which works like the pandas version.

Having used it read_json, you will likely want to use map_partitions to apply your row-wise further reduction of the resultant dictionaries.