7
votes

I'm parsing files that containt json objects. The problem is that some files have multiple objects in one line. e.g.:

{"data1": {"data1_inside": "bla{bl\"a"}}{"data1": {"data1_inside": "blabla["}}{"data1": {"data1_inside": "bla{bla"}}{"data1": {"data1_inside": "bla["}}

I've made a function that tries parsing a substring when there are no open brackets left, but there may be curly brackets in values. I've tried skipping values with checking the start and end of quotes, but there are also values with escaped quotes. Any ideas on how to deal with this?

My attempt:

def get_lines(data):
    lines = []
    open_brackets = 0
    start = 0
    is_comment = False
    for index, c in enumerate(data):
        if c == '"':
            is_comment = not is_comment
        elif not is_comment:
            if c == '{':
                if not open_brackets:
                    start = index
                open_brackets += 1

            if c == '}':
                open_brackets -= 1
                if not open_brackets:
                    lines.append(data[start: index+1])

    return lines
3
What about using json.loads ? docs.python.org/2/library/json.html - JoseKilo
For that to work, there needs to be one object parsed at a time. The same is with ujson. - user1334653
Brute force approach: Read the line character by character, try json.loads with the accumulated string after each character. If it succeeds, start over with accumulating characters. - timgeb
I strongly recommend wrapping all the JSON objects in an array as items: [{...},{...},{...}] and it will be valid JSON and wouldn't need special treatment (which is bound to break sometime) - casraf

3 Answers

2
votes

Simple but less robust version:

>>> import re
>>> s = r'{"data1": {"data1_inside": "bla{bl\"a"}}{"data1": {"data1_inside": "blabla["}}{"data1": {"data1_inside": "bla{bla"}}{"data1": {"data1_inside": "bla["}}'
>>> r = re.split('(\{.*?\})(?= *\{)', s)
['', '{"data1": {"data1_inside": "bla{bl\\"a"}}', '', '{"data1": {"data1_inside": "blabla["}}', '', '{"data1": {"data1_inside": "bla{bla"}}', '{"data1": {"data1_inside": "bla["}}']

This will fail if }{ is contained in a string

As other suggested, you could then try to parse each element. If it's not valid, then we should check this element together with the next one.

Note that r is the result of the code above

accumulator = ''
res = []
for subs in r:
    accumulator += subs
    try:
        res.append(json.loads(accumulator))
        accumulator = ''
    except:
        pass
10
votes

You can use the json raw_decoder! This allows the reading of json strings with extra data after the first json object. An example of usage would be:

>>> dec = json.JSONDecoder()
>>> json_str = '{"data": "Foo"}{"data": "BarBaz"}{"data": "Qux"}'
>>> dec.raw_decode(json_str)
({u'data': u'Foo'}, 15)
>>> dec.raw_decode(json_str[15:])
({u'data': u'BarBaz'}, 18)
>>> dec.raw_decode(json_str[33:])
({u'data': u'Qux'}, 15)

The first part of the tuple is the json object, the second is how much of the string was used when reading it. Therefore a loop like this will allow you to iterate over all the json objects in a string.

dec = json.JSONDecoder()
pos = 0
while not pos == len(str(json_str)):
    j, json_len = dec.raw_decode(str(json_str)[pos:])
    pos += json_len
    # Do something with the json j here
4
votes

The problem is that you can't reasonably split by any character or sequence of characters, because that sequence could always show up in a string as a field value, for example '{"data1": "}{"}{"data2":"foo"}'.

If we assume that every substring in your file/string that is valid JSON must start with '{' and end with '}' (of course, in the general case we'd also have to deal with '[' and ']' characters), here's a brute force approach:

import json

with open('input.txt') as inp:
    s = inp.read().strip()

jsons = []

start, end = s.find('{'), s.find('}')
while True:
    try:
        jsons.append(json.loads(s[start:end + 1]))
    except ValueError:
        end = end + 1 + s[end + 1:].find('}')
    else:
        s = s[end + 1:]
        if not s:
            break
        start, end = s.find('{'), s.find('}')

for x  in jsons:
    print(x)

Demo:

$ cat input.txt 
{"data1": {"data1_inside": "bla{bl\"a"}}{"data1": {"data1_inside": "blabla["}}{"data1": {"data1_inside": "bla{bla"}}{"data1": {"data1_inside": "bla["}}
$ python json_linereader.py 
{u'data1': {u'data1_inside': u'bla{bl"a'}}
{u'data1': {u'data1_inside': u'blabla['}}
{u'data1': {u'data1_inside': u'bla{bla'}}
{u'data1': {u'data1_inside': u'bla['}}

Output for s = '{"data1": "}{"}{"data2":"foo"}'

{'data1': '}{'}
{'data2': 'foo'}

I haven't checked this code for all eventualities with unit tests, but the idea should be clear.