Invalid syntax when evaluating expressions

Question

I'm getting

SyntaxError: invalid syntax

and

SyntaxError: invalid token

when evaluating a boolean expression in a dictionary:

Test Case

test_expressions = ['(3530A338A58 OR 3533A500H65 OR 3533-555A57 OR 3533A593A36 OR 3533A637A60 OR 3533A636A67 OR 3533A637D30 OR 3533H370A53 OR  3533H370T63 OR 3533H693H95 OR 3533H693T57 OR 3533H873H98) AND NOT (3630H693H66 OR 3630D387A39)', 'NOT (TP(0330-000098) OR TP(0330H750A85))']

test_dict = dict((k, opt_exp_dict_clean[k]) for k in test_expressions if k in opt_exp_dict_clean)

print(test_dict)

{'(3530A338A58 OR 3533A500H65 OR 3533-555A57 OR 3533A593A36 OR 3533A637A60 OR 3533A636A67 OR 3533A637D30 OR 3533H370A53 OR  3533H370T63 OR 3533H693H95 OR 3533H693T57 OR 3533H873H98) AND NOT (3630H693H66 OR 3630D387A39)': '535A8300-503',
 'NOT (TP(0330-000098) OR TP(0330H750A85))': '336A5305-3'}

#Boolean-Normalize
opt_disp_dict_norm = {k: (True if v == "Accepted" else False) for k, v in opt_disp_dict_clean.items()}

#Replace Boolean Expressions
for expression, value in test_dict.items():
    expression = expression.replace("XOR", "is not").replace("OR", "or").replace("AND", "and").replace("NOT", "not")
    #Return Values From True Expressions
    if eval(expression, opt_disp_dict_norm):
        print(value)

Which returns

  File "<string>", line 1
    (3530A338A58 or 3533A500H65 or 3533-555A57 or 3533A593A36 or 3533A637A60 or 3533A636A67 or 3533A637D30 or 3533H370A53 or  3533H370T63 or 3533H693H95 or 3533H693T57 or 3533H873H98) and not (3630H693H66 or 3630D387A39)
               ^

SyntaxError: invalid syntax

And

'NOT (TP(0330-000098) OR TP(0330H750A85))' returns:

  File "<string>", line 1
    not (TP(0330-000098) or TP(0330H750A85))
               ^

SyntaxError: invalid token

I believe it is a problem with the format of my terms since this approach works for a generic case (below). How can I make these terms "evaluation friendly" for python?

The terms are called "options" that are alphanumeric and () and - are the only special characters.

Generic case:

dict1 = {'((A OR D OR E) AND F) AND NOT G': 'Pickles', '(A AND B) OR E': 'Tuna', '(NOT G) AND (A OR B OR C OR D OR E)': 'Salami'}
dict2 = {'A': 'Accepted', 'B': 'Rejected', 'C': 'Rejected', 'D': 'Accepted', 'E': 'Rejected', 'F': 'Accepted', 'G': 'Rejected', 'Z': 'Accepted'}

#Boolean-Normalize

dict2_norm = {k: (True if v == "Accepted" else False) for k, v in dict2.items()}


#Replace Boolean Expressions   
for expression, value in dict1.items():
    expression = expression.replace("XOR", "is not").replace("OR", "or")\    
    .replace("AND", "and").replace("NOT", "not") 
    #Return Values From True Expressions   
    if eval(expression, dict2_norm):    
        print(value)

returns

Pickles
Salami

EDIT 1

The opt_exp_dict_clean is very large and from a csv file. Here is a sample of opt_exp_dict_clean:

opt_exp_dict_clean = { '(TP(2530-000580) OR TP(2530-000583) OR TP(2530-000582) OR 2530A627A05 OR 2530T652A06 OR 2530A250A38 OR 2530H633G23 OR 2530A560E37 OR 2530H636T83 OR 2530H632M38 OR 2530H636F05 OR 2530T333D72 OR 2530T723H37 OR 2530T920F60 OR 2530T983H28 OR 9800H396A37 OR 9800H396A69 OR 9800H502T33 OR 2530H632H82 OR 2530A688D82 OR 2530-003276 OR 2530-002999 OR 2530H363A37 OR 9800H502D03 OR 2530A079H59 OR 2530H335A07 OR 2530H833T37 OR 2530H635G33 OR 2530A093G52 OR 2530H632G50 OR 2530T323E83 OR 2530H508D56 OR 2530H633D08 OR 2530T323E83 OR 2530T537T29 OR 2530T333H39 OR 2530T537D03 OR 2530T323L09 OR 2530T323X30 OR 2530T593T37 OR 2530H673A30 OR 2530H637W80 OR 2530T537N65 OR 2530T537T96 OR 9800H502G93 OR 2530T806G76 OR 2530T333U72 OR 2530T882F90 OR 2530T893F63 OR 2530T920P90 OR 2530T882H68 OR 2530T920J93 OR 2530T806T99 OR 2530T806J33 OR 2530T882N32 OR 2530T537N35 OR 2530T995L50 OR 2530D305H26 OR 2530D305A80 OR 2530D360A85 OR 2530D305F33 OR 2530D362G73 OR 2530D383A32 OR 2530D360M37 OR 2530D360N26 OR 2530D603D39 OR 2530E339H37 OR 2530D6': '333A5720-85',
 'TP(0330-000099) AND NOT (2350-000233 OR 2350-000320 OR 2350A058H33 OR 3930-000322 OR 3630H693H66 OR 3630D387A39 OR 2350H693D00 OR 2350H693A33 OR TP(0330H750A85))': '232A3222-7',
 '(0330-000022 OR 0330-000029) AND NOT ((2323H299A26 AND 3363A333A30) OR 2323H372T80 OR 3930A398A60 OR 9800H396A67 OR 9800H396A86 OR 9800H396A88 OR 9800H396A89 OR 9800H396H25 OR 9800H332A26 OR 9800H332A29 OR 9800H760A20 OR 9800T858T69)': '233A3303-20',
 'TP(0330-000099) AND NOT (2533-000333 OR 2533A793H53 OR 3630H693H66 OR 3630D337A56 OR 3630D387A39 OR TP(0330H750A85))': '232A3203-2',
 '2500T803A90 AND (TP(2523-003280) OR TP(2523-003283) OR TP(2523H873A02) OR TP(2523A396H98) OR TP(2523H639A33) OR 2523T882M53 OR 2523T995J72 OR 2523D350A03 OR 2523D072E33 OR 2523D305F86) AND (2370T869A37 OR 2375T869H29 OR 2370T869A33) AND (TP(3520T372A20) OR 3520T390A32 OR 3520-000253 OR 3520D033U92) AND NOT 3520T383T88': '333A2203-333H',
 '3833-000037 AND NOT TP(0330-000098)': '337A2530-8',
 '2530-002997 OR 2530A033S00 OR 2530A053I63 OR 2530A079H63 OR 2530A093G53 OR 2530A300L37 OR 2530A289T07 OR 2530A560E38 OR 2530A560L23 OR 2530A627A03 OR 2530A688G60 OR 2530A963A32 OR 2530H373A06 OR 2530H363A36 OR 2530H637D68 OR 2530H633G33 OR 2530H633P85 OR 2530H633Z78 OR 2530H636D33 OR 2530T323L37 OR 2530T333L03 OR 2530T537F78 OR 2530T537N36 OR 2530T537T99 OR 2530T537Y00 OR 2530T652A03 OR 2530T920W00 OR 2530T920G83 OR 2530D362G09': '337A5700-28'}

A sample of opt_disp_dict_clean:

opt_disp_dict_clean = {'2530T882N32': 'Accepted', 'TP(0330-000098)': 'Rejected', '2530A627A03': 'Accepted', '2530H633G33': 'Accepted', '2530A300L37': 'Rejected', '2523D072E33': 'Rejected', '3930-000322': 'Accepted', 'TP(0330-000099)': 'Rejected'}

In the example, test_dict is just a subdict of opt_exp_dict_clean

Without test_dict the boolean elevator would read:

#Boolean-Normalize
opt_disp_dict_norm = {k: (True if v == "Accepted" else False) for k, v in opt_disp_dict_clean.items()}

#Replace Boolean Expressions
for expression, value in opt_exp_dict_clean.items():
    expression = expression.replace("XOR", "is not").replace("OR", "or").replace("AND", "and").replace("NOT", "not")
    #Return Values From True Expressions
    if eval(expression, opt_disp_dict_norm):
        print(value)

Answer 1

Obviously, the problem is that neither IDs like 2530T882N32 nor TP(0330-000098) are valid Python variable names and thus produce a syntax error when trying to evaluate expressions with those. You could replace/remove the illegal parts and e.g. start them with a _ to make them legal variable names, but that might come with the risk of name clashes, and it might be difficult to associate them with their original counterparts.

Instead, I'd suggest you wrap all those identifiers into d['...'] to make them strings used for lookup in a dictionary. Then wrap the dictionary with the values into another accordingly named dictionary before passing it into the eval function. Do this after replacing the operators and only match uppercase letters so the operators (then lowercase) are not wrapped, too.

You can use re.sub with a callback function for the replacement. As regex, you can use e.g. TP\([^)]+\)|[-A-Z0-9]+. Note that the TP part has to come first.

expression = expression.replace("XOR", "is not").replace("OR", "or").replace("AND", "and").replace("NOT", "not") 
expression = re.sub(r"TP\([^)]+\)|[-A-Z0-9]+", lambda m: "d['%s']" % m.group(), expression)
if eval(expression, {"d": opt_disp_dict_norm}):
    print(value)

Note, however, that I'm still getting key errors for some of the IDs and the first of the expressions is missing a closing ). To handle missing keys, you could also replace with "d.get('%s', True)" or "d.get('%s', False)" to give those default values.