Okay, let's get down and dirty with networking
I will not keep any secrets anymore on how I do things or how I intend to do things. The protocol is mine anyway, even though it's not finished yet, and whoever wants to make an implementation of it is more than welcome. Even if Kerm wishes to make his own implementation for DoorsCS that's compatible with my routines, he's free to do so
I've been thinking about speed and data integrity mainly, which are two big issues in networking about which many, many articles and many books are written. My point of view here is that I want things to
work,
be simple and preferably
be fast. Or at least fast enough to be useful.
I think I'm going to have to make two versions of the library (and the protocol); one that is optimized for simple data transfer between two(!) calculators, mainly aimed at game programmers who need speed and don't care about the theoretical possibilities of networking, and one that is built for networking, showing the possibilities of calc networks and slower applications like turn based strategy games, chatting, file transfers, the lot.
Texas Instruments
does not use an RS232 protocol, like I read somewhere and have since believed to be true, according to the information that Wikipedia has on RS232. So I don't know what it uses, but I know how it works, and for some while now I know
why they use it.
The TI protocol pulls one line low to send a one or the other to send a zero. The connected calculator pulls the other line low as soon as it has stored the bit to acknowledge it's transfer. I always laughed at that primitive protocol because I expected a clocked protocol to be much faster. Unfortunately my software never really managed to beat the TI protocol...
Because the calculator doesn't have a reliable clock built in, you can't be sure how fast the receiving calculator can process the incoming data. Therefor, to prevent congestion errors, I have to limit the transmission speed to a "safe speed" that empirically has to turn out to be useful. This certainly works, but it is far from optimal. And this is exactely the reason why TI uses their own weird protocol; acknowleding every single bit means that you know exactely when you can send the next bit. So even though it's a silly and slow protocol, there's less time wasted on waiting, and the throughput is dynamic and usually higher than with my own protocol. So basically, that sucks.
Other than that however, the TI protocol has no advantages; it is entirely impossible to use it for networking, for example. That's exactely where my segregation comes into play. I want to make a simple, small and fast library using some implementation of TI's protocol that can exchange bytes, data blocks or strings fast from one calculator to another to accomodate game programmers, and a more complex networking library using my own protocol.
The former is not a very interesting thing to talk about, because it uses known technology in a welltried manner and only the wrapping is more shiny than TI's version (it should be more easy to use, better documented, and perhaps a tad faster, but that's about it). I hope to be able to release that rather soon. If I'm really lazy, I'll even use Ben's routines (with his permission I hope), or the built-in TI routines underneath my shiny wrapping. You'll understand that the keyword here is simplicity and supporting game programmers in coding multiplayer games.
The latter however, will be in perpetual beta as it has been the last couple of months, just to further investigate the possibilities and boundaries of calculator networking as my personal playground. I've been brainstorming a lot about a form of "agreed upon transfer speed" to close the gap with TI's protocol a bit in terms of speed.
The network needs to have a certain speed setting that nodes can, kind of, query the primary calculator about, and a variable containing the address of the "slowest" calculator. You can't in any way store the speed, because different calculators have a different grasp of "time", whether you define time in clockcycles or in milliseconds. This is why the speed has to be measured by each calculator individually from the primary calculator. The slowest calculator has to be known, because when a new calc joins the network it's speed has to be compared to the slowest speed to see if we need to set a new maximum speed.
The maximum possible transfer speed with any calculator can be determined by introducing a bit of error checking on the receiving end. We cannot acknowledge every bit in a network like the TI protocol does because it would interfere, so we have to take acknowledging a level higher. When sending data over, two things can go wrong. Bits can flip because of interference (which increases with higher speeds) or because the receiver can't keep up, or the receiver can miss a few bits after a transfer. These are two things we can check on. Missing bits is an easy one; I can use a timeout after which the transfer gets closed brute force, and I know how many bytes I should receive (preceding length byte), so when my timer fires and I'm missing bits there's something wrong and I can inform the sender that the data was corrupted.
Flipped bits is a whole other topic. Again; books have been written about error detection and correction, but I'll stick to the simplest option, taking the risk of letting a few mistakes pass unnoticed. I'll probably introduce a parity bit with every byte I send, indicating how many (an even or an odd number) bits in the sent byte should have been one. If the parity bit doesn't agree with the data on the receiving end, then there's an odd number of bits flipped. If an even number of bits have flipped though, a parity bit will not be enough. But since I check parity for every byte, and only one byte in a stream has to be corrupt for the transfer to go wrong, I guess that'll solve itself.
So we can detect transfer problems in two ways, and inform either the user (the program that's using the CLAP) or the library about this. I guess I'd best let that depend on what state we're in. If we're testing the connection speed, I want the library to handle the error by sending back a NACK so the primary calculator to throttle the network if neccessary. If we're in normal receive mode I'll inform the user with some switch that we've received data, but that it is corrupted.
Another option would of course be requesting a retransmit from the sender of the stream, to assure correct transfers, but that's starting to look like a reliable data transfer protocol a bit too much to be funny
I'd need to introduce all kinds of buffers and whatnot, and I'm not in the mood for that
So, bottom line is this:
Calculator connects >
Measure maximum speed between slowest nodes by increasing the speed untill it goes wrong >
Adapt network speed to maximum speed >
Transfers should be pretty reliable, if they are corrupt anyway: user gets informed
This shouldn't throttle the network any more than absolutely neccessary, but still make it possible to connect all kinds of processors with different speeds. (This has been a concern of mine from the beginning; I wanted to make a networking library for all calculator models, though I didn't want to statically limit the speed to the slowest calculator type...)
That's about it for now, your ideas and feedback are welcome as always