Page 1 of 1

nice bug (fPart(64) > 0)

Posted: Sun 20 Jan, 2008 10:03 am
by King Harold
fPart(log(2^64)/log(2))
I would think that is 0 (zero) because log(2^64)/log(2) = 64
but no, apparently it's a different kind of 64, which looks exactly like 64 on the screen but is actually a bit bigger.

the result is 1E-12
this does not happen with fPart(64

not really 'breaking news' I know, but this bug caught me when I wanted to know what numbers n and m (from N) for which n^m (where the ^ means power, not xor) comes closest to 2^64-1 for n != 2^x.
I'm not exactly sure why I wanted to know this, but I was doing some stuff with the Diffie-Hellman exchange so it might have something to do with that...

Posted: Sun 20 Jan, 2008 10:10 am
by Dwedit
Yay, rounding errors...

Re: nice bug (fPart(64) > 0)

Posted: Thu 29 Apr, 2010 2:37 pm
by TheUnnamed
Also on the 83 and 84 (all models that I've encountered thus far) i^x for any x>=7 yields a wrong answer.
and On the Nspire all logs that should resolve to an integer greater then 2 is are not actually the integer its slightly off so like log(8,2)-3=-1E-13 that became a real problem when I didn't know this and tried to index an array on log(expr(something in the form of a*x^n)/a,2) it took me a week to figure out that i had to add round the result to 0 decimal places to make it work. :lol: