Home All Groups Group Topic Archive Search About

Single v. Double Help

Author
7 Apr 2010 10:55 PM
Todd Heiks
Why is 'ans1' true if double and false for single?

Module Module1
    Sub Main()

        Dim x As Double / Single
        Dim y As Double / Single

        Dim ans1 As Boolean
        Dim ans2 As Boolean

        x = 0.3000000000000001
        y = Math.Round(x, 2)

        ans1 = y = 0.3
        ans2 = Math.Round(y, 2) = 0.3

    End Sub
End Module

Thanks

Author
7 Apr 2010 11:03 PM
Herfried K. Wagner [MVP]
Todd --

Am 08.04.2010 00:55, schrieb Todd Heiks:
> Why is 'ans1' true if double and false for single?
>[...]
>          Dim x As Double / Single
>          Dim y As Double / Single
>
>          Dim ans1 As Boolean
>          Dim ans2 As Boolean
>
>          x = 0.3000000000000001
>          y = Math.Round(x, 2)
>
>          ans1 = y = 0.3
>          ans2 = Math.Round(y, 2) = 0.3

The reason is that the result cannot be represented exactly.  It's in
general not recommended to compare floating point values of single and
double precision using the '=' operator.

More information:

(Complete) Tutorial to Understand IEEE Floating-Point Errors
<URL:http://support.microsoft.com/?scid=kb;EN-US;42980>

--
  M S   Herfried K. Wagner
M V P  <URL:http://dotnet.mvps.org/>
  V B   <URL:http://dotnet.mvps.org/dotnet/faqs/>