for your 1 bit pleasure!

All times are UTC

Post new topic Reply to topic  [ 3 posts ] 
Author Message
 Post subject: nice bug (fPart(64) > 0)
PostPosted: Sun 20 Jan, 2008 10:03 am 
Calc King

Joined: Sat 05 Aug, 2006 7:22 am
Posts: 1513
I would think that is 0 (zero) because log(2^64)/log(2) = 64
but no, apparently it's a different kind of 64, which looks exactly like 64 on the screen but is actually a bit bigger.

the result is 1E-12
this does not happen with fPart(64

not really 'breaking news' I know, but this bug caught me when I wanted to know what numbers n and m (from N) for which n^m (where the ^ means power, not xor) comes closest to 2^64-1 for n != 2^x.
I'm not exactly sure why I wanted to know this, but I was doing some stuff with the Diffie-Hellman exchange so it might have something to do with that...

Reply with quote  
 Post subject:
PostPosted: Sun 20 Jan, 2008 10:10 am 
Maxcoderz Staff
User avatar

Joined: Wed 15 Dec, 2004 6:06 am
Posts: 579
Location: Chicago!
Yay, rounding errors...

You know your hexadecimal output routine is broken when it displays the character 'G'.

Reply with quote  
PostPosted: Thu 29 Apr, 2010 2:37 pm 
New Member

Joined: Wed 28 Apr, 2010 2:34 pm
Posts: 1
Also on the 83 and 84 (all models that I've encountered thus far) i^x for any x>=7 yields a wrong answer.
and On the Nspire all logs that should resolve to an integer greater then 2 is are not actually the integer its slightly off so like log(8,2)-3=-1E-13 that became a real problem when I didn't know this and tried to index an array on log(expr(something in the form of a*x^n)/a,2) it took me a week to figure out that i had to add round the result to 0 decimal places to make it work. :lol:

Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 3 posts ] 

All times are UTC

Who is online

Users browsing this forum: No registered users and 4 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB ® Forum Software © phpBB Group | DVGFX2 by: Matt