MaxCoderz

for your 1 bit pleasure!

All times are UTC




Post new topic Reply to topic  [ 3 posts ] 
Author Message
 Post subject: nice bug (fPart(64) > 0)
PostPosted: Sun 20 Jan, 2008 10:03 am 
Offline
Calc King

Joined: Sat 05 Aug, 2006 7:22 am
Posts: 1513
fPart(log(2^64)/log(2))
I would think that is 0 (zero) because log(2^64)/log(2) = 64
but no, apparently it's a different kind of 64, which looks exactly like 64 on the screen but is actually a bit bigger.

the result is 1E-12
this does not happen with fPart(64

not really 'breaking news' I know, but this bug caught me when I wanted to know what numbers n and m (from N) for which n^m (where the ^ means power, not xor) comes closest to 2^64-1 for n != 2^x.
I'm not exactly sure why I wanted to know this, but I was doing some stuff with the Diffie-Hellman exchange so it might have something to do with that...


Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: Sun 20 Jan, 2008 10:10 am 
Offline
Maxcoderz Staff
User avatar

Joined: Wed 15 Dec, 2004 6:06 am
Posts: 579
Location: Chicago!
Yay, rounding errors...

_________________
You know your hexadecimal output routine is broken when it displays the character 'G'.


Top
 Profile  
Reply with quote  
PostPosted: Thu 29 Apr, 2010 2:37 pm 
Offline
New Member

Joined: Wed 28 Apr, 2010 2:34 pm
Posts: 1
Also on the 83 and 84 (all models that I've encountered thus far) i^x for any x>=7 yields a wrong answer.
and On the Nspire all logs that should resolve to an integer greater then 2 is are not actually the integer its slightly off so like log(8,2)-3=-1E-13 that became a real problem when I didn't know this and tried to index an array on log(expr(something in the form of a*x^n)/a,2) it took me a week to figure out that i had to add round the result to 0 decimal places to make it work. :lol:


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 3 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 3 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB ® Forum Software © phpBB Group | DVGFX2 by: Matt