Log4j do not have such an appender out of box. But you may do following:
Add appenders for third party libraries which uses gigabytes of data ("I have a gigabytes of log written from third party libraries into ERROR category"). And configure your appenders such that they do not use such a large amount of storage.
RollingFileAppender
Configure this third party library appender as rolling file. After some time they will use oldest file and you can keep only latest logs.
JDBCAppender
This appender accepts usage of stored procedures. Write a stored procedure so that it will do what you want.
For your first requirement add a calculated column or calculate this value in stored procedure and add to to your table columns. Then control this value to decide if you need to insert this log line to database.
For your second requirement You can take hash value (MD5,SHA etc) of exception message. If this hash value exists in database table you may ignore to insert it. Or you may count how many of it exists and decide how to insert to database accordingly.
You do not need to use enterprise database for these purposes, you may use apache derby for example, then all of your logs stay in application server. I think this one,JDBCAppender, meets your requirements more closely.