First I would check if any assertions can fail for any reasons. So disabling any assertions and running most primitive version of the script without any possibility of failure (other than based on sampler response) would be my first step. If it doesn't help...
HTTP Sampler will fail for return code < 200, or return code >= 400, or any Java exception, which will happen when request could not be sent, or response from the server was not received (in full). Most commonly those are timeouts related to various underlying problems:
Lack of available ports on JMeter side. Since you indicated that problem happens only with more JMeter users, it is a possibility. On paper you have 65535 ports, but both Windows and Linux are limiting number of available ports to few thousands by default. Make sure you have enough ports to run the load, and monitor port usage as you ramp threads up.
Even if all ports are available, many ports may be sitting for a few minutes in TIME_WAIT or CLOSE_WAIT state, depending on how your client and server interact and other network issues. So theoretically you may have 65k ports, but practically you don't have enough. In this case it's worth checking why it happens (can point to a bug) and possibly reduce the time ports will be waiting in this *_WAIT state.
You also need to make sure you have enough RAM, JVM heap and CPU: monitor to see if there's a bottleneck that can cause such slowness that breaks the ability of a Sampler to finish its work successfully.
If all of the above is working as expected, then either server or someone between client and server (load balancer, proxy, firewall) is causing this issue.
I would start from the server, and verify that all requests you are sending are received by a server. E.g. if you send 2000 requests, and server receives 1000, which finish successfully, then from server perspective everything is fine, but Jmeter will be at 50% error rate.
Also check your JMeter log definitions and make sure failures and exceptions are written to them properly (they are multi-line messages, and may cause log definitions to break when getting to them, and thus you wont' see them). For example run HTTP Sampler that points to non-available server. See if you can diagnose what returned exception was.
Beyond that - figuring out which part is "lying" to you would be an approach, but hard to offer any more specific advice based on current information