The formula involves an attempt to estimate the box-counting dimension of the image of this tree:

The box-counting dimension can be estimated by
dim(E) ≈ -log(N)/log(r)
where N is the number of squares that cover an image and r is the side length of the squares in the cover. In fact, the dimension is defined to be the limit of this expression as r goes down to zero. Thus, we typically have a sequence of r values that go down to zero which yields a sequence of N values that grow up to ∞. We can then extrapolate the limit from those values.
Now, in the formula to which you refer:
-diff(log(n))./diff(log(r))
n and r are both finite sequences generated by a box-counting command. Thus, in the numerator, diff(log(n)) forms the sequence of consecutive differences of the log applied to those n values and, in the denominator diff(log(r)) does a similar thing for the r values. The dotted quotient then forms the pairwise quotient of these two differences. The mean of the results is evidently one way to extrapolate the limit. I wouldn't think it would be one of the best ways, though.