What Nicolas Heitz is saying seems to be correct.
I ran a simple test locally to reproduce this, using TIBCO Designer to send and receive large messages with approx 4Mb size and a queue configured with maxbytes set to 100Kb,
This is what the queue looked like before the test started:

First scenario, no receivers on the queue.
It was possible to send one 4MB message to the queue, but any message after this fails.

Second scenario, one receiver not acknowledging messages
Starting point, one 4MB pending message on the queue
Sending a second message works

It appears to be possible to keep sending messages, as long as there is an active receiver on the queue processing messages without having acknowledged them yet.

Conclusion
In case of a slow consumer not acknowledging the messages it receives fast enough, maxbytes will not limit the pending msgs size on a queue.
If there is no consumer on the queue, then maxbytes will be checked before a message is sent, and if the limit is reached, the message producer will receive an error from the ems server an no new messages can be sent.
The maxbytes property is not blocking a message larger than maxbytes from being sent to the queue.