This is a follow-up to Handle an exception thrown in a generator and discusses a more general problem.
I have a function that reads data in different formats. All formats are line- or record-oriented and for each format there's a dedicated parsing function, implemented as a generator. So the main reading function gets an input and a generator, which reads its respective format from the input and delivers records back to the main function:
def read(stream, parsefunc):
for record in parsefunc(stream):
do_stuff(record)
where parsefunc
is something like:
def parsefunc(stream):
while not eof(stream):
rec = read_record(stream)
do some stuff
yield rec
The problem I'm facing is that while parsefunc
can throw an exception (e.g. when reading from a stream), it has no idea how to handle it. The function responsible for handling exceptions is the main read
function. Note that exceptions occur on a per-record basis, so even if one record fails, the generator should continue its work and yield records back until the whole stream is exhausted.
In the previous question I tried to put next(parsefunc)
in a try
block, but as turned out, this is not going to work. So I have to add try-except
to the parsefunc
itself and then somehow deliver exceptions to the consumer:
def parsefunc(stream):
while not eof(stream):
try:
rec = read_record()
yield rec
except Exception as e:
?????
I'm rather reluctant to do this because
- it makes no sense to use
try
in a function that isn't intended to handle any exceptions - it's unclear to me how to pass exceptions to the consuming function
- there going to be many formats and many
parsefunc
's, I don't want to clutter them with too much helper code.
Has anyone suggestions for a better architecture?
A note for googlers: in addition to the top answer, pay attention to senderle's and Jon's posts - very smart and insightful stuff.
parsefunc()
, and passread_record
toread()
. This way, the exceptions could be handled right after the call toread_record()
. - Sven Marnachparsefunc()
is responsible for consuming a record, but then - how is an appropriateparsefunc()
chosen for the rest of the stream? - Jon Clements♦do some stuff
inparsefunc
is a fairly big chunk of code, so I can't simply get rid of it. - georgread
knows in which format the data comes in, and decides which parser to use. - georgeof
defined? - alancalvitti