Your question title asks about "multiple conditions", but very specifically you're looking to match based on any row that itself matches multiple conditions. That goes beyond the common AND operator and into a function that can process a range. You also need to be prepared for a book to be checked out and returned many times, which means there's no single row that manages the status of a given book; VLOOKUP and INDEX/MATCH are off the table too. Instead, you're effectively looking to generate a list of 0 or 1 values that match whether that book was checked out without being returned, and then coloring the cell based on whether there are any rows that match that condition.
To operate on multiple values at a time, you can use ARRAYFORMULA and then combine the output array with OR. However, one of the tricks about ARRAYFORMULA is that, to preserve the invariant about making single-value functions into array-valued functions, you can't use functions that can take arrays. This means that AND and ISBLANK don't work the way you'd like them to, but you can resolve that by using * instead of AND and = "" for ISBLANK.
One such solution (working example):
=OR(ARRAYFORMULA((A1 = $A$5:$A) * ($C$5:$C = "")))
ARRAYFORMULA isn't the only function to operate on a list of values, though; you could also use FILTER directly to only return matching rows. Here, you're checking whether any row has a matching book name and a blank return value, and then confirming that the value is not the #N/A that FILTER returns when nothing matches.
One such solution (working example):
=NOT(ISNA(FILTER($A$8:$C, $A$8:$A = A1, $C$8:$C = "")))
Of course, you can also take advantage of the fact that you're only checking blanks to use tehhowch's solution with COUNTA and FILTER above. However, since that solution won't work for arbitrary expressions, you can use ARRAYFORMULA or FILTER if your needs become more complex.