Today while struggling to make my C++ code (using Ooura FFT library) to give the same results as Matlab does, I finally found a problem and a solution.
In Matlab calculating reverse coefficients for a grid of amplitude-frequency response is done in a following way (code fragment from Matlab's internal fir2() function):
%H contains 8192 points of AFR data
Hconj = [H conj(H(npt-1:-1:2))]; % Fourier transform of real series
ht = real(ifft(Hconj)); % Symmetric real series
as a result we get back 16384 bins, and the second half of them can be thrown away, but the first half can later be used as FIR coefficients.
But if I do the same in Ooura using Real DFT rdft() function, I get coefficients which create a mirror effect in the resulting AFR, all the frequencies on the AFR plot are divided by 2.
So an idea came to me: in my C++ code I made my H twice as big (16384 points) and filled them all with frequency data without mirroring. And voila! it worked, now I got 16384 points, throw away everything after 8192 point and now the resulting AFR matches the Matlab's.
I was convinced that all standard FFT implementations need this mirroring. Is it just a quirk of Ooura that it DOES NOT need mirroring data in input, or maybe there is something else going on?