How do I make my computer unstupid?

19 Jun 2006 9:48 PM

JAM

I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
(0.01999999999999996) what is wrong with it?

19 Jun 2006 10:42 PM

Herfried K. Wagner [MVP]

"JAM" <J**@discussions.microsoft.com> schrieb:
>I tell my computer a math problem (19.98-20.00) and it gives me a wako
>answer
> (0.01999999999999996) what is wrong with it?

Nothing is wrong. This behavior is by design.

Binary floating point and .NET
<URL:http://www.yoda.arachsys.com/csharp/floatingpoint.html>

--
M S Herfried K. Wagner
M V P <URL:http://dotnet.mvps.org/>
V B <URL:http://classicvb.org/petition/>

20 Jun 2006 6:22 AM

bob@datasync.com

Jam,

I don' know much about arithmetic, but I can think of a lot of "stupid"
answers to your question, such as: To make your computer unstupid,
don't unfix it!

Good luck.

JAM wrote:
Show quoteHide quote

> I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
> (0.01999999999999996) what is wrong with it?

20 Jun 2006 8:53 AM

M. Posseth

a computer is not a human :-)

from a computers point of view the answer is completely valid

computers work with the binary system humans work with the decimal system

so to get a human expected answer you should convert the values to the
decimal system and then perform your math

(CType(19.98, Decimal) - CType(20.0, Decimal)

regards

Michel Posseth

Show quoteHide quote

"JAM" wrote:

> I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
> (0.01999999999999996) what is wrong with it?

21 Jun 2006 11:43 PM

Jay B. Harlow [MVP - Outlook]

M.Posseth,
I would recommend:

Dim d As Decimal = 19.98D - 20.0D

Which causes the 19.98 & 20.0 to be Decimal literals & not Double literals.

Being Decimal literals saves you from converting the values to Decimal at
run time...

At the very least it makes the expression easier to read ;-)

--
Hope this helps
Jay B. Harlow [MVP - Outlook]
..NET Application Architect, Enthusiast, & Evangelist
T.S. Bradley - http://www.tsbradley.net

Show quoteHide quote

"M. Posseth" <MPoss***@discussions.microsoft.com> wrote in message
news:1D706ADA-27FB-4665-B3A7-6066B9B666A4@microsoft.com...
|
| a computer is not a human :-)
|
| from a computers point of view the answer is completely valid
|
| computers work with the binary system humans work with the decimal system
|
| so to get a human expected answer you should convert the values to the
| decimal system and then perform your math
|
| (CType(19.98, Decimal) - CType(20.0, Decimal)
|
|
| regards
|
| Michel Posseth
|
|
|
|
| "JAM" wrote:
|
| > I tell my computer a math problem (19.98-20.00) and it gives me a wako
answer
| > (0.01999999999999996) what is wrong with it?

20 Jun 2006 8:59 AM

M. Posseth

this is a nice article that explains the subject clearly

http://support.microsoft.com/default.aspx?scid=kb;en-us;214118

note that in VB.net we have the decimal datatype now that solves this issue
however it comes with a prize and that is resources ( speed, memory )

regards

Michel Posseth [MCP]

Show quoteHide quote

"JAM" wrote:

> I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
> (0.01999999999999996) what is wrong with it?