Pages: [1]   Go Down

Author Topic: Now we have 13x faster MD5 checksums  (Read 260 times)

Jubadub

  • Gold Member
  • *****
  • Offline Offline
  • Posts: 425
  • New Member
Now we have 13x faster MD5 checksums
« on: July 03, 2024, 11:45:47 AM »

Someone incredibly awesome just made and released this for Mac OS, allowing us to calculate MD5 checksums, which is extremely useful for checking file integrity, which is very important after a download:
http://macintoshgarden.org/apps/md5classic

The only other program we could use before up until now AFAIK was Checksum 1.3: http://macintoshgarden.org/apps/checksum-13

The main problem with Checksum 1.3 is that it is a pure 68k app. But md5classic 1.0 up there is both PowerPC native and 68k native. Runs on every commercialized version of Mac OS, too (!), from System 1 to Mac OS 9.2.2.

This stuff is useful all the time, sometimes we have broken downloads, and we ought to check if they went fine. For big files, it is a HUGE pain using Checksum 1.3, since it's almost 5x slower, but now it is no more as big a deal. In fact, even for small files, thanks to an annoying confirmation box, it takes time/patience to close Checksum 1.3 every single time, but with md5classic, you can simply Cmd+Q to quit without any extra step.

Source code is available there, too, for those interested. It seems to be pure C, and includes a CodeWarrior Pro 6 project file. That is the last version of CW that can compile 68k apps out-of-the-box.
« Last Edit: July 13, 2024, 08:32:59 AM by IIO »
Logged

ssp3

  • Platinum Member
  • *****
  • Offline Offline
  • Posts: 851
Re: Now we have 5x faster MD5 checksums
« Reply #1 on: July 03, 2024, 08:53:54 PM »

<like>  :)
Logged
If you're not part of the solution, you're part of the problem.

Jubadub

  • Gold Member
  • *****
  • Offline Offline
  • Posts: 425
  • New Member
Re: Now we have 5x faster MD5 checksums
« Reply #2 on: July 05, 2024, 12:53:47 PM »

The MD5 algorithm got improved and now the title of this thread is wrong:

It's not 5x times. Now it is 6.5x faster! :)
Logged

Greystash

  • Gold Member
  • *****
  • Offline Offline
  • Posts: 240
  • Tinkerer
    • Mac-Classic.com
Re: Now we have 6.5x faster MD5 checksums
« Reply #3 on: July 05, 2024, 03:11:16 PM »

Very handy thanks for sharing!
Logged

Jubadub

  • Gold Member
  • *****
  • Offline Offline
  • Posts: 425
  • New Member
Re: Now we have 5x faster MD5 checksums
« Reply #4 on: July 12, 2024, 10:01:39 AM »

It's not 5x times. Now it is 6.5x faster! :)

Well, the tool further evolved in such a hardcore manner, that it got rewritten in pure, carefully-curated, highly-optimized 68k assembly, then the same for PPC assembly, resulting in one of the wildest FAT applications EVER as far as sheer expertise is concerned.

When the program started, it was almost 5x faster than the only solution we had on PPC.
Then it evolved, and became 6.5x faster.
THEN it evolved AGAIN, and it became TEN TIMES faster.

One would think that's the limit. But nope. It got further and further squeezed... And now it is almost 13x faster.

13 TIMES

That's how much faster we can verify the integrity of transferred data, such as Macintosh Garden downloads, in Mac OS 9.2.2 and earlier on PPC Macs now. What used to take 13 minutes to check before (say, perhaps something like a DVD image download) now takes ONLY ONE MINUTE. Just stupid fast. Sheer madness. I love it.

The hand-crafted PPC assembly is yet to appear in a future version, by the way, but it seems it will only be ever-so-slightly-faster than the current version, because as it turns out, CodeWarrior Pro 6.3 actually compiles VERY efficient PowerPC code. It seems to be a far better compiler than its 68k C compiler (and, no, THINK C was even worse at 68k compilation than CWPro 6, so 68k users can really rejoice the beautiful hand-written 68k assembly this brilliant software contains).

Theoretically-speaking, there is even more room for improvement: the existing MD5 algorithm can be rewritten to take advantage of AltiVec in G4s (and G5s). Maybe there's potential to leverage GPUs, as well, in particular for pre-G4 PPC Macs. And, finally, MD5 seems to only have little-endian implementations, so a big endian implementation without any byte swapping or similar could be invented/discovered (all OSes seem to just byte-swap for big endian, we checked). Some hashing algorithms are done in big endian, though, like the SHA family, but not MD5 (yet).

All-in-all, the Garden thread is very educational in all sorts of ways, so that's a recommended read, as well.

New versions are planned, but they are mostly quality-of-life features, rather than further speed optimizations.

Note that, as of now, the source code repository wasn't updated in a while, but I expect that to change with the release of the next version, hopefully.
Logged

IIO

  • Global Moderator
  • Platinum Member
  • *****
  • Offline Offline
  • Posts: 4541
  • just a number
Re: Now we have 5x faster MD5 checksums
« Reply #5 on: July 13, 2024, 08:35:49 AM »

that it got rewritten in pure, carefully-curated, highly-optimized 68k assembly, then the same for PPC assembly, resulting in one of the wildest FAT applications EVER as far as sheer expertise is concerned.

sounds a bit like cliff´s ("OS923") attitude, maybe he is related to him?

Logged
insert arbitrary signature here

Jubadub

  • Gold Member
  • *****
  • Offline Offline
  • Posts: 425
  • New Member
Re: Now we have 5x faster MD5 checksums
« Reply #6 on: July 13, 2024, 11:31:26 AM »

that it got rewritten in pure, carefully-curated, highly-optimized 68k assembly, then the same for PPC assembly, resulting in one of the wildest FAT applications EVER as far as sheer expertise is concerned.

sounds a bit like cliff´s ("OS923") attitude, maybe he is related to him?

I am convinced those are 2 completely unrelated people. I don't find them similar, but it's true both are experts in their respective "things".

Also, the speed got slightly bumped up again, but that will probably be the last time: some minor PPC assembly hand-made tweaks pushed up the speed enough that now it is 13.5x faster. (Checksum 1.3: 147.0 seconds, md5classic 1.0b5: 10.9 seconds)

Since the speed bump this time was smaller, version 1.0b5 is only available on GitHub in the form of source code right now. But it's easy to compile and keep it, which I did, by simply unpacking the SIT-ed .mcp and .rsrc files, opening the .mcp file in CodeWarrior Pro 6, selecting "PPC" or "FAT" from the target dropdown box, clicking on "Make" button and voila. Just click-click-click and done, anyone can do it without needing to know anything about programming or CodeWarrior.

To push the speed even beyond will require AltiVec and/or GPUs. Or something entirely different like that. Figuring out a big endian algorithm of MD5 could potentially also speed something up. Other than that, I think this is it, the ultimate MD5 hashing algorithm and program on PPC and 68k.

Now "sidd" (the developer) will give HFS+ support a go... This would enable us to break the 2 GB barrier for file checksumming directly from Mac OS for the first time since 1984, 40 years later. I hope he can make it happen.
Logged
Pages: [1]   Go Up
 

Recent Topics