I'm trying to understand floating point math in C#. I have read a lot of documents and papers over the entire day , about floating points and sad to say most of the time I have only an incling about the core issues. I'm quite good at English not being a native speaker but sometimes I still miss context. What I still know from my studies is that floating point is just a representation of 4 bytes and that doing math on it will always be wrong because of it. You can compare with an epsilon for error margin but even that is just a close approximation. That being said to point out what I understand this is the case I'm struggling with.
[Test]
public void LibHfzCompressedReadTest()
{
LibHfZ lib = new LibHfZ();
HfzFile file1 = lib.loadFile("c:\\demo2_HF.hf2.gz", HfzFormat.LIBHFZ_FORMAT_HF2_GZ);
//HfzFile file2 = lib.loadFile("c:\\fractal-16bit.hf2.gz", HfzFormat.LIBHFZ_FORMAT_HF2_GZ);
List<float> data = file1.getTileData((short)0);
double HFmin = 0;
double HFmax = 0;
for(int i=0;i < data.Count;i++)
{
double f = data.ElementAt(i);
System.Console.WriteLine("float: " + f);
if(i == 0)
{
HFmin = HFmax = f;
}
else
{
if (f < HFmin && f != 0)
{
System.Console.WriteLine("new min: " + f);
HFmin = f;
}
if (f > HFmax && f != 0)
{
System.Console.WriteLine("new max: " + f);
HFmax = f;
}
}
}
System.Console.WriteLine("HFmin: " + HFmin);
System.Console.WriteLine("HFmax: " + HFmax);
float Precis = file1.header.Precis;
System.Console.WriteLine("Precis: " + Precis);
float BlockLevels = (float)((HFmax - HFmin) / Precis) + 1;
System.Console.WriteLine("BlockLevels: " + BlockLevels);
// calc scale
float VertScale = (float)(HFmax - HFmin) / BlockLevels;
if(VertScale<=0)
{
VertScale = 1.0f;
}
System.Console.WriteLine("VertScale: " + VertScale);
//System.Console.WriteLine(file2);
}
========output==========
HFmin: 0
HFmax: 1019,586
Precis: 0,01
BlockLevels: 101959,6
VertScale: 0,009999902
The end result SHOULD be VertScale:0,009999821 but it's VertScale: 0,009999902
I'm sure the input data is is right, cause I just read it from a tested and verified file. My personal hunch is that it has something to do with the comparison on HFmax and HFmin, maybe they are giving false positives or negatives causing the end result of HFmax and HFmin to be wrong.
I've been through a lot of stackoverflows about how you can compare for equality on a floating point with the epsiolon but I haven't came across one for higher or lower ... Any insights and please not another paper :( I've read dozens today already I just want to get this understood and fixed :(
UPDATE I added a check <= 0 on the VertScale to see if it would evaluate and set it to 1.0f however the output is the same. I tried this with doubles as well as recommended in the first comments however this was still the same result.
dataset: http://users.telenet.be/sunspot/hfzcsharpsource/testfloatarray.7z
double
instead offloat
for all variables of typefloat
(leaveList<float>
alone). Is that better (whatever "better" may mean) ? – Kris Vandermotten