An input is an atom or a pair; if the input is a pair then it may contain either atoms or more pairs. When the input is a pair, you should add 1 to the maximum nesting level of the sub-pairs. Otherwise the input is not a pair, and the maximum nesting level is 0. Note that in Racket, '() is not a pair?.
These observations lead to a very simple definition:
#lang racket
(define (max-depth xss)
(if (pair? xss)
(+ 1 (apply max (map max-depth xss)))
0))
Here, (map max-depth xss) evaluates to a list of the maximum nesting levels of all sublists within xss. To find the maximum, apply is needed to apply max to the list of arguments. Since the sublists themselves are already nested one level deep, 1 is added to the maximum.
max-depth.rkt> (max-depth 'x)
0
max-depth.rkt> (max-depth '())
0
max-depth.rkt> (max-depth '(1 2 3))
1
max-depth.rkt> (max-depth '(a (b (c (d)))))
4
max-depth.rkt> (max-depth '((((((0)))))))
6
max-depth.rkt> (max-depth '(a b (c d (e f (g)) h) (i j) (k (l (m (n (o) p))) q) r))
6