Preliminary notes: Given the evidence presented here, I will assume you are using:
type GenericQ r = forall a . Data a => a -> r from syb, and
gmapQ :: Data a => (forall d. Data d => d -> u) -> a -> [u] from Data.Data.
Please let me know me if I'm mistaken about that. Also, any foralls in what follows will be written explicitly.
There is more than meets the eye here. As Li-yao Xia suggests, it is a matter of generalisation involving the type of op. There are three relevant facts about your first definition of shallowest:
Before generalisation, the inferred type of op is Data d => d -> f a. Given the Data d constraint, Rule 1 of the monomorphism restriction (see subsection 4.5.5 of the Report) means d in this type cannot be generalised.
In the body of shallowest, op shows up in two places. The first one is op z, with z :: a1 being bound and constrained at top level by the signature of shallowest. The upshot is that this occurrence of op does not require generalisation of the argument type: as far as it is concerned, the type of op could be forall f a. a1 -> f a, monomorphic in the type variable a1 (I took this terminology from subsection 4.5.4 of the Report).
The other occurrence, though, is gmapQ op z. gmapQ has a rank-2 type, requiring a polymorphic argument. That being so, this occurrence requires generalisation of the argument type of op, as noted at the end of Li-yao Xia's answer.
#1 and #3 are contradictory requirements, and so you get a type error, which can be avoided either by disabling the monomorphism restriction or by demanding op to be polymorphic on the argument type with a signature. Thanks to the other occurrence of op described in #2, the situation is reported as a mismatch involving the two occurrences.
Here follows a more minimal extended example, which might help to see what is going on. (If you are going to plop the following snippets into GHCi, besides -XRankNTypes you should also set -XMonomorphismRestriction and -XNoExtendedDefaultRules in order to see the same results.)
This is a function with a rank-2 type, which will play the role of gmapQ:
glub :: (forall x. Show x => x -> String) -> String
glub f = f 7
Now let's try a scenario similar to that involving shallowest...
foo1 :: forall a. Show a => a -> String
foo1 x = bar x ++ glub bar
where
bar = show
... and there is your error:
<interactive>:506:23: error:
• Couldn't match type ‘x’ with ‘a’
‘x’ is a rigid type variable bound by
a type expected by the context:
forall x. Show x => x -> String
at <interactive>:506:18-25
‘a’ is a rigid type variable bound by
the type signature for:
foo1 :: forall a. Show a => a -> String
at <interactive>:505:1-38
Expected type: x -> String
Actual type: a -> String
• In the first argument of ‘glub’, namely ‘bar’
In the second argument of ‘(++)’, namely ‘glub bar’
In the expression: bar x ++ glub bar
• Relevant bindings include
bar :: a -> String (bound at <interactive>:508:3)
x :: a (bound at <interactive>:506:5)
foo1 :: a -> String (bound at <interactive>:506:1)
Adding a wildcard where the signature of bar should go gives an additional error which is slightly more suggestive:
foo2 :: forall a. Show a => a -> String
foo2 x = bar x ++ glub bar
where
bar :: _
bar = show
• Found type wildcard ‘_’ standing for ‘a -> String’
Where: ‘a’ is a rigid type variable bound by
the type signature for:
foo2 :: forall a. Show a => a -> String
at <interactive>:511:1-38
To use the inferred type, enable PartialTypeSignatures
• In the type signature: bar :: _
In an equation for ‘foo2’:
foo2 x
= bar x ++ glub bar
where
bar :: _
bar = show
• Relevant bindings include
x :: a (bound at <interactive>:512:5)
foo2 :: a -> String (bound at <interactive>:512:1)
Note how the wildcard "standing for a -> String" is stated as a separate fact from a being bound by the type signature of foo2. I believe that corresponds to the distinction bewteen monomorphic in a type variable and polymorphic that I alluded to in point #2 above.
Giving bar a polymorphic type signature makes it work:
foo3 :: forall a. Show a => a -> String
foo3 x = bar x ++ glub bar
where
bar :: forall b. Show b => b -> String
bar = show
And so does making the definition of bar pointful, which evades the monomorphism restriction by making it a "function binding" rather than a "simple pattern binding":
foo4 :: forall a. Show a => a -> String
foo4 x = bar x ++ glub bar
where
bar x = show x
For the sake of completeness, it is worth noting that no constraint on the type means no monomorphism restriction:
foo5 :: forall a. Show a => a -> String
foo5 x = bar x ++ glub bar
where
bar = const "bar"
A related situation involves using bar twice, but without a rank-2 function:
foo6 x y = bar x ++ bar y
where
bar = show
Which type will GHC infer for foo6?
GHCi> :t foo6
foo6 :: Show a => a -> a -> [Char]
The arguments get the same type, as doing otherwise would require generalisation of bar, which requires a type signature (or pointfullness, etc.):
foo7 x y = bar x ++ bar y
where
bar :: forall a. Show a => a -> String
bar = show
GHCi> :t foo7
foo7 :: (Show a1, Show a2) => a1 -> a2 -> [Char]
Since I didn't mention it yet, here is an analogue to your second shallowest:
foo8 :: forall a. Show a => a -> String
foo8 x = bar x
where
bar = show
It is worth emphasising that bar is not actually being generalised here: it is monomorphic in the type variable a. We can still break this example, by messing with foo7 rather than with bar:
foo9 = bar
where
bar :: _
bar = show
In this case, bar is not generalised, and neither is foo (now pointfree and without a signature). That means the monomorphic type variable is never resolved. In terms of Rule 2 of the monomorphism restriction, it becomes an ambiguous type variable:
<interactive>:718:14: error:
• Found type wildcard ‘_’ standing for ‘a0 -> String’
Where: ‘a0’ is an ambiguous type variable
To use the inferred type, enable PartialTypeSignatures
• In the type signature: bar :: _
In an equation for ‘foo9’:
foo9
= bar
where
bar :: _
bar = show
• Relevant bindings include
foo9 :: a0 -> String (bound at <interactive>:716:5)
<interactive>:719:13: error:
• Ambiguous type variable ‘a0’ arising from a use of ‘show’
prevents the constraint ‘(Show a0)’ from being solved.
Relevant bindings include
bar :: a0 -> String (bound at <interactive>:719:7)
foo9 :: a0 -> String (bound at <interactive>:716:5)
Probable fix: use a type annotation to specify what ‘a0’ should be.
These potential instances exist:
instance Show a => Show (ZipList a)
-- Defined in ‘Control.Applicative’
instance Show Constr -- Defined in ‘Data.Data’
instance Show ConstrRep -- Defined in ‘Data.Data’
...plus 64 others
...plus 250 instances involving out-of-scope types
(use -fprint-potential-instances to see them all)
• In the expression: show
In an equation for ‘bar’: bar = show
In an equation for ‘foo9’:
foo9
= bar
where
bar :: _
bar = show
Adding a type signature to bar in the definition of foo9 won't help -- it just changes the point from which the error is reported. Changing bar to something without a constraint does eliminate the error, as it makes it possible to generalise both bar and foo.