|
web
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Single v. Double HelpModule Module1 Sub Main() Dim x As Double / Single Dim y As Double / Single Dim ans1 As Boolean Dim ans2 As Boolean x = 0.3000000000000001 y = Math.Round(x, 2) ans1 = y = 0.3 ans2 = Math.Round(y, 2) = 0.3 End Sub End Module Thanks Todd --
Am 08.04.2010 00:55, schrieb Todd Heiks: > Why is 'ans1' true if double and false for single? The reason is that the result cannot be represented exactly. It's in >[...] > Dim x As Double / Single > Dim y As Double / Single > > Dim ans1 As Boolean > Dim ans2 As Boolean > > x = 0.3000000000000001 > y = Math.Round(x, 2) > > ans1 = y = 0.3 > ans2 = Math.Round(y, 2) = 0.3 general not recommended to compare floating point values of single and double precision using the '=' operator. More information: (Complete) Tutorial to Understand IEEE Floating-Point Errors <URL:http://support.microsoft.com/?scid=kb;EN-US;42980> -- M S Herfried K. Wagner M V P <URL:http://dotnet.mvps.org/> V B <URL:http://dotnet.mvps.org/dotnet/faqs/>
Use a static form of the EventLog class?
How to tell if folder item is file or folder? null reference exception could result at runtime How to get the list of all SQL Server foreign keys in a database by using Microsoft.SqlServer.Manage Problem passing exception from AppDomain want to convert string into byte array hw i can do it datagridview Combobox column Adding/copying Worksheet to Excel Programmatically add Sub Menus to a ContextMenuStrip HTML report in vb.net |
|||||||||||||||||||||||