TLDR: You will be fine. The SSL renegotiations shouldn't happen frequently enough to be noticeable by your end user.
Rant starts here:
Load distribution using DNS is a commonly misunderstood topic that leads into a lot of anecdotal evidence and straw-man arguments. I've been in too many of these meetings.
Here's how I usually settle these arguments:
"Wow yeah that sounds really exoteric [long dramatic pause] but it really can't be that bad since google uses it"
$host encrypted.google.com
encrypted.google.com is an alias for www3.l.google.com.
www3.l.google.com has address 74.125.224.195
www3.l.google.com has address 74.125.224.202
www3.l.google.com has address 74.125.224.193
www3.l.google.com has address 74.125.224.197
www3.l.google.com has address 74.125.224.207
www3.l.google.com has address 74.125.224.206
www3.l.google.com has address 74.125.224.203
www3.l.google.com has address 74.125.224.204
www3.l.google.com has address 74.125.224.196
www3.l.google.com has address 74.125.224.199
www3.l.google.com has address 74.125.224.201
www3.l.google.com has address 74.125.224.194
www3.l.google.com has address 74.125.224.192
www3.l.google.com has address 74.125.224.200
www3.l.google.com has address 74.125.224.205
www3.l.google.com has address 74.125.224.198
Updates:
Is this setup redundant?
It is not inherently redundant in the engineering sense since if one of those ips were to fail it would continue to be served to the customer until a DNS zone change is performed and all downstream caches expire. With that said, most browsers are smart enough to try another ip under these circumstance - reference.
Moreover, a system could easily be devise that instead of requiring a DNS zone change to remove the failed node would, instead, route the ip of the failed instance to a servicing device by simple ip takeover.
Is this setup resilient?
Yes, resilience is achieved by minimizing your failure domain. Back to our example failure of a single ip (and remember these ips may represent load balancers backed by hundred of servers or even an entire data center) the likelihood of a customer hitting that ip is 1/16, or ~6%, (using the google example above). This is inherently more resilient than a system with a single A address, wich would impact 100% of the users, or a system with 2 A records in which the user has an even 50/50 change of hitting a failed resource.