|
web
newsgroups
|
|||||||||||||||||||||||
|
|||||||||||||||||||||||
Strange float behaviourcompiled in "Release" mode and ONLY when the .NET debugger is not attached to the process. I have some drawing code that checks to see if a data point is inside the bounding rectangle of a graph. if it is not then the point is not shown. I find that in the deployed code the final data point does not show. I therefore went through the code adding message boxes to see if i could spot any problems. I eventually did something like this Dim graphRectangle as RectangleF graphRectangle = graph.GridRect MessageBox.Show(graph.GridRect.Right = graphRectangle.Right).ToString) MessageBox.Show(graph.GridRect.Right - graphRectangle.Right).ToString) MessageBox.Show((graph.GridRect.Right = CType(graphRectangle.Right, Single)).ToString()) MessageBox.Show(BitConverter.ToInt32(BitConverter.GetBytes(graph.GridRect.Right), 0).ToString) MessageBox.Show(BitConverter.ToInt32(BitConverter.GetBytes(graphRectangle.Right), 0).ToString) now you would expect the graphRectangle variable to be the same as the graph.GridRect property but it is not. The first message box returns false The second message box shows 2.098..E-05 The third and fourth message boxes show 11443800 indicating that the binary values are the same. So what is happening!! I have found using ctype to convert the value to a single on each side of the equals operator yields the correct value. I also found it completly impossible to reporduce this problem outside my project. Any ideas anyone?? VB.NET on .NET 1.1 sp1 <tedd_n_a***@yahoo.com> wrote in message
Show quoteHide quote news:1156519694.213601.50360@74g2000cwt.googlegroups.com... Just a visitor around here.... If floats are causing you greif, here's some > > now you would expect the graphRectangle variable to be the same as the > graph.GridRect property but it is not. > > The first message box returns false > The second message box shows 2.098..E-05 > The third and fourth message boxes show 11443800 indicating that the > binary values are the same. > > So what is happening!! I have found using ctype to convert the value > to a single on each side of the equals operator yields the correct > value. I also found it completly impossible to reporduce this problem > outside my project. > > Any ideas anyone?? > > VB.NET on .NET 1.1 sp1 good info. What Every Computer Scientist Should Know About Floating-Point Arithmetic http://docs.sun.com/source/806-3568/ncg_goldberg.html (Complete) Tutorial to Understand IEEE Floating-Point Errors http://support.microsoft.com/kb/42980/en-us FP Representation http://www.cprogramming.com/tutorial/floating_point/understanding_floating_point_representation.html fwiw, do a web search on comparing numbers to EPSILON instead of each other... for example, If X = Y Then Do Something End If can be replaced by If Abs(X - Y) < EPSILON Then Do Something End If fwiw, you can find the value for EPSILON in the C header files that ship with VS... or, just figure out the minimum resolution you need and create your own EPSILON_LIKE constant. -- Ken Halter - MS-MVP-VB (visiting from VB6 world) - http://www.vbsight.com Please keep all discussions in the groups.. In Loving Memory - http://www.vbsight.com/Remembrance.htm
Saving changes made in DataGridView
hex to unicode: problem Atomic Operation? One Last Class Question - Example that works DirectX rendered inside a control? Problems with SaveAs Database Current namespace, stack and lineno How to make double click event to call and execute the mouseup event code I/O Access through window handle? |
|||||||||||||||||||||||