@RodrigoRibeiro's answer explains the Level bits, but once you get rid of universe levels, what does the type Set → Set → Set have to do with binary relations?
Suppose you have a binary relation R ⊆ A × B. The propositional way of modeling it is to create some indexed type R : A → B → Set such that for any a : A, b : B, R a b has inhabitants iff (a, b) ∈ R. So if you want to talk about all relations over A and B, you have to talk about all A- and B- indexed types, i.e. you have to talk about RelationOverAandB = A → B → Set.
If you then want to abstract over the relation's left- and right-hand base type, that means the choice of A and B are not fixed anymore. So you have to talk about REL, such that REL A B = A → B → Set.
What, then, is the type of REL? As we have seen from the REL A B example, it takes the choice of A and B as two arguments; and its result is the type A → B → Set.
So to summarize: given
A : Set
B : Set
we have
REL A B = A → B → Set
which itself has type Set (remember, we're ignoring universe levels here).
And thus,
REL : Set → Set → Set ∎