[VB.NET]File Handling Optimization

[thegamefreak0134]

First of all, this is my first post from my own internet connection, and not the schools internet. Hooray for dial up! Its not great, but it gets me the unfiltered access I need to better pursue my programming. (Half of the good programming tutorials are blocked there under "games". Yay for freedom!)

Anywho, this message was typed whilst I was offline, and thus may or may not be quite accurate anymore. I still need this help though. Here is the message:

I am in the long process of writing a few compressors/decompressors for the PC. I am focusing on simple things (RLE,SRLE,Gap encoding, etc.) to basically build up my skills in the area. I am having some issues with my file access being awful slow though, and was wondering if you could help me out with that.

I am going to link to my entire project, since my code is a bit too lengthy to post. (non-Visual Studio users should be able to open the source in notepad, although the form-designer generated stuff will need to be skipped.)

My main issue I believe is my file access. I have to do a lot of byte-by-byte access for this kind of thing, and I believe that my methods of doing so may be a bit slow. Can someone (or multiple someones) please take a quick look at the code and give me some pointers in the speed department? I have a very new computer with a very nice processor, and it still takes it around 30 seconds to compress a single file around 1.25 MB. Winzip can go faster than this. Heck, windows XP can go faster than this, and that's a pretty sad statement there, considering that WinXP doesn't tie up tie up the system resources like this program does...

Anywho, keep in mind that I am relatively new at this. Oh, and unless it will cause a speed increase, I would prefer to keep my loops in "while" format, as I find them much more readable personally compared to the "for" format. (Plus it gives me more flexible control in planning as to what exactly my counter variables do.) This is a personal thing, similar to how I like to use "x = x + 1" in C as opposed to "X++", when they compile down to the same thing.

Also, my code for all the compression routines is similar, so an example you give me in the RLE compressor section for example I can then use and apply to the other 7 or 8 similar sections. (I cheated and copied and pasted. OMG! Not that )

Anywho, I know y'all can probably tell me a million things I can improve speed wise in this source, and that is exactly what I need. Criticize away folks!

-thegamefreak0134

*EDIT* Oops, I suppose actually linking to the file helps... http://www.darknovagames.com/newfolder/ ... ressor.zip

Sorry! OK, now you can have fun.

[cjgone]

First of all, this is my first post from my own internet connection, and not the schools internet. Hooray for dial up! Its not great, but it gets me the unfiltered access I need to better pursue my programming. (Half of the good programming tutorials are blocked there under "games". Yay for freedom!)

Proxy servers work quite nicely (Unless they're blocked too).


Ooh, and have fun with that.

[benryves]

Never, ever use the VB6 file functions - unless you are using VB6, of course.

Look at the System.IO namespace for all the file reading and writing classes you should be using.

If you need to quickly dump all of the data from a file into an array of bytes (and back from the array of bytes to a file on disk), the best methods to use are File.ReadAllBytes and File.WriteAllBytes.

However, these are .NET 2.0 methods, and the code looks like .NET 1.x code. You can download a free version of the VB.NET 2005 IDE (.ISO) to write .NET 2.0 apps, and unless you have a very good reason not to I recommend you use them.

Knowing how to do it manually is still useful, though. To read/write files you need a couple of things - a FileStream that can be used to read/write the file and a BinaryReader or BinaryWriter that can be used to make chucking binary data around easier.

(I would put in sample code, but having no IDE to hand it'd have to be in C# - if you need some, watch this space and I'll post when I have access to a VB.NET IDE).

Your code reads/writes to the input and output file during the compression. Whilst this is the least memory-taxing method (as you don't use much in the way of temporary storage) it is much, much slower than compressing an array in memory (RAM access is many times faster than disk access). The OS can also perform optimisations when it comes to dumping a large array to disk rather than have to put up with multiple writes of a byte at a time.

[kv83]

heh. i'm trying something similiar... writing and reading mii files works great!

[thegamefreak0134]

I knew you'd come through ben! Thanks a bunch. I'll change my code over as soon as I get home. (I should be able to figure out the stuff now that I can pull out the info in the library.)

I'll post my optimized code as soon as I have it written (should be a fairly straightforward task) and let y'all know the results.

-thegamefreak

[benryves]

Bear in mind that if you just want to compress files for your application in some way, there's the System.IO.Compression namespace that contains classes for handling compressed streams using the GZip and Deflate algorithms.

[thegamefreak0134]

No, I'm writing this program to teach myself about compression, so I definately want to write the routines myself. They work well apparently (well, all except RLE, which was expected) but unsurprisingly I can't get any files smaller than winzip. That is my ultimate goal. Don't know if I will ever get there, but it's a great learning experience.

-gamefreak

[Dwedit]

I made a (slow) LZ compressor in C++ using the multimap class.