I'm not going to say I recommend what follows, but for completeness it actually is possible to define such a Functor.
The Functor typeclass demands that you can fmap any function into your Functor. This means, normally, that it's difficult to ensure invariants which demand typeclass instances. However, we can contort this situation quite a bit and actually end up getting away with a Functor instance. In practice, what it means is that we can use the type system to ensure that we we defer our rebalancing to times that are more convenient.
First, we'll introduce a demand on the above type. In particular, we'll give it a Monoid instance which maintains balance. This works out fine since Monoid doesn't require our container be polymorphic.
instance Ord a => Monoid (BalancedTree a) where
mempty = Leaf
mappend Leaf Leaf = Leaf
mappend Leaf b = b
mappend b Leaf = b
mappend (Node a l1 r1) (Node b l2 r2) = ... -- merge and rebalance here
Now, using this instance we can write functions which correspond almost to a Monad instance for BinaryTree. In particular, we need it in order to combine our new trees as build using bindBin, the almost version of (>>=) on binary search trees.
returnBin :: a -> BinaryTree a
returnBin a = Node a Leaf Leaf
bindBin :: Ord b => BinaryTree a -> (a -> BinaryTree b) -> BinaryTree b
bindBin Leaf _ = Leaf
bindBin (Node a l r) f = bindBin l f <> f a <> bindBin r f
Then we introduce this very strange type (which requires the RankNTypes extension)
newtype FBinaryTree a =
FBinaryTree (forall r . Ord r => (a -> BinaryTree r) -> BinaryTree r)
There are many ways of thinking about this, but we'll just note that there's an isomorphism between FBinaryTree a and BinaryTree a witnessed, basically, by returnBin and bindBin.
toF :: BinaryTree a -> FBinaryTree a
toF bt = FBinaryTree (bindBin bt)
fromF :: Ord a => FBinaryTree a -> BinaryTree a
fromF (FBinaryTree k) = k returnBin
and finally, as FBinaryTree inherits some of the properties of the Cont monad or the Yoneda lemma type, we can define a Functor instance for FBinaryTree!
instance Functor FBinaryTree where
fmap f (FBinaryTree c) = FBinaryTree (\k -> c (k . f))
So that now, all we must do is convert our BinaryTrees into FBinaryTrees, perform our Functor operations there, and then drop back down to BinaryTree as needed. Smooth sailing, right?
Well, almost. It turns out that there's a huge efficiency price we pay for this. In particular, it's easy to have exponential blowups occur when using types like FBinaryTree. We can avoid these by sending FBinaryTree through BinaryTree from time to time by using
optimize :: Ord a => FBinaryTree a -> FBinaryTree a
optimize = toF . fromF
which, as the type shows, requires us to have an Ord instance right there. In fact, the code will use the Ord instance there to perform the needed rebalancing.