This is cross-track distance described here
dxt = asin( sin(δ13) ⋅ sin(θ13−θ12) ) ⋅ R
where
δ13 is (angular) distance from start point to third point
θ13 is (initial) bearing from start point to third point
θ12 is (initial) bearing from start point to end point
R is the earth’s radius
You can calculate needed distance and bearings using formulas from given page
distance
a = sin²(Δφ/2) + cos φ1 ⋅ cos φ2 ⋅ sin²(Δλ/2)
c = 2 ⋅ atan2( √a, √(1−a) )
d = R ⋅ c
where
φ is latitude, λ is longitude, R is earth’s radius (mean radius = 6,371km);
bearing
θ = atan2( sin Δλ ⋅ cos φ2 , cos φ1 ⋅ sin φ2 − sin φ1 ⋅ cos φ2 ⋅ cos Δλ )
where
φ1,λ1 is the start point, φ2,λ2 the end point (Δλ is the difference in longitude)
note that angles need to be in radians to pass to trig functions