nice bug (fPart(64) > 0)

A General Discussion forum for TI calculators

Moderator: MaxCoderz Staff

Post Reply
King Harold
Calc King
Posts: 1513
Joined: Sat 05 Aug, 2006 7:22 am

nice bug (fPart(64) > 0)

Post by King Harold »

fPart(log(2^64)/log(2))
I would think that is 0 (zero) because log(2^64)/log(2) = 64
but no, apparently it's a different kind of 64, which looks exactly like 64 on the screen but is actually a bit bigger.

the result is 1E-12
this does not happen with fPart(64

not really 'breaking news' I know, but this bug caught me when I wanted to know what numbers n and m (from N) for which n^m (where the ^ means power, not xor) comes closest to 2^64-1 for n != 2^x.
I'm not exactly sure why I wanted to know this, but I was doing some stuff with the Diffie-Hellman exchange so it might have something to do with that...
User avatar
Dwedit
Maxcoderz Staff
Posts: 579
Joined: Wed 15 Dec, 2004 6:06 am
Location: Chicago!
Contact:

Post by Dwedit »

Yay, rounding errors...
You know your hexadecimal output routine is broken when it displays the character 'G'.
TheUnnamed
New Member
Posts: 1
Joined: Wed 28 Apr, 2010 2:34 pm

Re: nice bug (fPart(64) > 0)

Post by TheUnnamed »

Also on the 83 and 84 (all models that I've encountered thus far) i^x for any x>=7 yields a wrong answer.
and On the Nspire all logs that should resolve to an integer greater then 2 is are not actually the integer its slightly off so like log(8,2)-3=-1E-13 that became a real problem when I didn't know this and tried to index an array on log(expr(something in the form of a*x^n)/a,2) it took me a week to figure out that i had to add round the result to 0 decimal places to make it work. :lol:
Post Reply