Variable precision

9 Oct 2006 7:58 PM

John Dann

Just a curiosity really, but I am curious as to the explanation:

I'm trying to read an external DLL written in C++ I think using
DLLImport. The DLL version number is available from a DLL function
which returns a float variable. The value should be 2.3 but a .Net
(double) variable is returning 2.9999xxxxxx. where xxxx is a sequence
of other digits.

Obviously in retrieving the value into the .Net environment something
is ever so slightly perturbing the original value. Anyone know what
this might be?

JGD

10 Oct 2006 4:47 AM

Mattias Sjögren

John,

> The value should be 2.3 but a .Net
>(double) variable is returning 2.9999xxxxxx. where xxxx is a sequence
>of other digits.

This is something you have to get used to when working with floating
point numbers. Not all numbers can be represented exactly. See
http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems or
http://www.extremeoptimization.com/resources/Articles/FPDotNetConceptsAndFormats.aspx
for more details.

Mattias

--
Mattias Sjögren [C# MVP] mattias @ mvps.org
http://www.msjogren.net/dotnet/ | http://www.dotnetinterop.com
Please reply only to the newsgroup.