Integer to number, too many decimals

17 Jun 2010 2:19 PM

Eric

Hi,

In a data table I have records numbered from 801 to 809.
I also use these numbers as row number then 8.01 to 8.09.

To do so I just devide the record number by 100 and put it in a variable of
type decimal (also tried double and single).

At one point I compare this value with a number to do an evaluation:

itemnr = CDec(rw("itemnr")) / 100

"If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"

Well it should look like that, but it actually showing:
" If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
8.0600000000000005 Or itemnr = 8.0700000000000003 Then"

How can I get rid of the extra decimals?
I don't need them and it's confusing.

rg,
Eric

17 Jun 2010 2:52 PM

Armin Zingler

Am 17.06.2010 16:19, schrieb Eric:
Show quoteHide quote

> Hi,
>
> In a data table I have records numbered from 801 to 809.
> I also use these numbers as row number then 8.01 to 8.09.
>
> To do so I just devide the record number by 100 and put it in a variable of
> type decimal (also tried double and single).
>
> At one point I compare this value with a number to do an evaluation:
>
> itemnr = CDec(rw("itemnr")) / 100
>
> "If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"
>
> Well it should look like that, but it actually showing:
> " If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
> 8.0600000000000005 Or itemnr = 8.0700000000000003 Then"
>
> How can I get rid of the extra decimals?
> I don't need them and it's confusing.

If 'itemnr' is Decimal or Double, I can't repro the problem:

      Dim value As Decimal = 803
      Dim itemnr As Double    'As Decimal: same behavior

      itemnr = value / 100

      If itemnr = 8.03 Then
         MsgBox("equal")    'always shown
      End If

--
Armin

17 Jun 2010 3:41 PM

J.B. Moreno

Eric <E***@discussions.microsoft.com> wrote:

Show quoteHide quote

> Hi,
>
> In a data table I have records numbered from 801 to 809.
> I also use these numbers as row number then 8.01 to 8.09.
>
> To do so I just devide the record number by 100 and put it in a variable of
> type decimal (also tried double and single).
>
> At one point I compare this value with a number to do an evaluation:
>
> itemnr = CDec(rw("itemnr")) / 100
>
> "If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"
>
> Well it should look like that, but it actually showing:
> " If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
> 8.0600000000000005 Or itemnr = 8.0700000000000003 Then"

Fractional numbers are always approximate --- if you're going to
proceed with this, you're going to have to deal with the rounding
issue.

I don't understand what you're doing well enough to say what you should
do, but I'd reconsider it if I was you....

--
J.B. Moreno

17 Jun 2010 5:19 PM

OmegaSquared

Hello, Eric,

The numbers 8.xx are being interpreted as floating point values. Appending
a "D" to them should correct the problem that you are seeing. That is, try:

"If itemnr = 8.03D Or itemnr = 8.04D Or ...

It is very important to have a clear understanding of the different number
types that can be used and what their limitations are. Floating point values
particularly seem to cause some programmers much confusion. See:

http://docs.sun.com/source/806-3568/ncg_goldberg.html

for a good detailed explanation.

Cheers,
Randy