Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)

[dosdude1]

Cool! 
Silly question - does compression impact data transfer rate anyhow?

Not at all, all it does is decompress the LZSS-compressed OS9 NDRV binary upon system power on, at that point it is loaded into memory and is then executed by the OS during boot.

[ssp3]

Not at all, all it does is decompress the LZSS-compressed OS9 NDRV binary upon system power on, at that point it is loaded into memory and is then executed by the OS during boot.

Thanks for the info! You did a great job and big favour to Mac users!


Off-topic. Are you familiar with ARM thumb mode firmware reversing? I'm thinking of liberating some WD Firewire external enclosure(s). I've been partially successful, but lacking further skills. If interested, let me know, I'll PM you.

[dosdude1]

Not at all, all it does is decompress the LZSS-compressed OS9 NDRV binary upon system power on, at that point it is loaded into memory and is then executed by the OS during boot.

Thanks for the info! You did a great job and big favour to Mac users!


Off-topic. Are you familiar with ARM thumb mode firmware reversing? I'm thinking of liberating some WD Firewire external enclosure(s). I've been partially successful, but lacking further skills. If interested, let me know, I'll PM you.

I've never worked with anything ARM, so I'm not immediately familiar. Though I could always take a look and figure it out.

[joevt]

I have another absolutely MASSIVE update for you all... I was able to implement an LZSS decompression algorithm in the ROM, and use it to decompress a stored LZSS-compressed version of the OS9 NDRV. As such, the ROM has been reduced in size to UNDER 128K! This means that you can now flash it onto cards WITHOUT having to replace the original 128k EEPROM! Of course this image also includes my patched OS9 and OS X NDRVs that remove the EEPROM ID check. The updated ROM image is attached, along with my custom Forth LZSS decompression implementation. Hope you all enjoy!

EDIT: Updated ROM padding to 128K for Flashrom support.

Which lzss compression algorithm did you use?

Last time I looked at your ndrv (earlier in this thread), it was 54648 bytes decompressed and 30527 bytes compressed, but the compressed version in this update is 31377 bytes. So you're either using a different compression or a different ndrv. I tried decompressing this using the C algorithm I was using before but that turns the 31377 bytes into 96180 which is way too large. Compressing that produces 31229 bytes - not matching the 31377 input bytes - which means my lzss is different than yours (the lzss can't be successful unless a round trip of decompression and compression produces output that matches the input).

Your decompression algorithm in fcode is 572 bytes. You can shrink that to 429 bytes (143 less bytes) if you use headerless instead of external. This doesn't matter in this case since you've achieved the goal of shrinking the rom to < 128K but it could be useful for other storage device roms or for creating a 64K rom.

[dosdude1]

I have another absolutely MASSIVE update for you all... I was able to implement an LZSS decompression algorithm in the ROM, and use it to decompress a stored LZSS-compressed version of the OS9 NDRV. As such, the ROM has been reduced in size to UNDER 128K! This means that you can now flash it onto cards WITHOUT having to replace the original 128k EEPROM! Of course this image also includes my patched OS9 and OS X NDRVs that remove the EEPROM ID check. The updated ROM image is attached, along with my custom Forth LZSS decompression implementation. Hope you all enjoy!

EDIT: Updated ROM padding to 128K for Flashrom support.

Which lzss compression algorithm did you use?

Last time I looked at your ndrv (earlier in this thread), it was 54648 bytes decompressed and 30527 bytes compressed, but the compressed version in this update is 31377 bytes. So you're either using a different compression or a different ndrv. I tried decompressing this using the C algorithm I was using before but that turns the 31377 bytes into 96180 which is way too large. Compressing that produces 31229 bytes - not matching the 31377 input bytes - which means my lzss is different than yours (the lzss can't be successful unless a round trip of decompression and compression produces output that matches the input).

Your decompression algorithm in fcode is 572 bytes. You can shrink that to 429 bytes (143 less bytes) if you use headerless instead of external. This doesn't matter in this case since you've achieved the goal of shrinking the rom to < 128K but it could be useful for other storage device roms or for creating a 64K rom.

Originally I was planning on using the C lzss implementation you use as it was, however when I was doing initial tests with it, I found that decompressing the NDRV after compressing it with the same implementation resulted in garbage. I fixed this by simply changing the following:

Code: [Select]

#define EI 11
#define EJ  5
#define F ((1 << EJ) + 1)

After making these changes, that implementation began working properly to compress and decompress the NDRV, but resulted in the slightly larger compressed file size. I then wrote the Forth implementation based on the decoding logic of that C implementation.

Also, I intentionally left the named FCode words in there as I thought it would help prevent a potential issue where tokenized FCode numbers may conflict with those already in the ROM (as I wasn't able to detokenize the ROM into a re-tokenizable format). I got around this issue by making a slight modification to the tokenizer source code, starting the FCode number at 0xA00 instead of 0x800, to ensure the generated FCode numbers would not conflict with those already there. So of course I could re-tokenize it headerless, but no need to bother since it already fits into the 128K ROM as it is.

[joevt]

Originally I was planning on using the C lzss implementation you use as it was, however when I was doing initial tests with it, I found that decompressing the NDRV after compressing it with the same implementation resulted in garbage.

Not sure why the decode didn't work for you. Did you try the C decode algorithm or only the fcode decode algorithm? I found that using the default C decode/encode, the decode of the encoded ndrv correctly produced bytes that match the original ndrv:

Code: [Select]

/Volumes/Work/Programming/lzss/lzss/lzss e driver,AAPL,MacOS,PowerPC.bin driver,AAPL,MacOS,PowerPC.lzss
text:  54648 bytes
code:  30527 bytes (55%)

/Volumes/Work/Programming/lzss/lzss/lzss d driver,AAPL,MacOS,PowerPC.lzss driver,AAPL,MacOS,PowerPC_reverse.bin

bbdiff driver,AAPL,MacOS,PowerPC.bin driver,AAPL,MacOS,PowerPC_reverse.bin
/Volumes/Work/Open Firmware and Name Registry/ROM Firmtek 1S2/dosdude/driver,AAPL,MacOS,PowerPC_reverse.bin and /Volumes/Work/Open Firmware and Name Registry/ROM Firmtek 1S2/dosdude/driver,AAPL,MacOS,PowerPC.bin are identical.

md5 driver,AAPL,MacOS,PowerPC.bin driver,AAPL,MacOS,PowerPC_reverse.bin
MD5 (driver,AAPL,MacOS,PowerPC.bin) = dc939ef1b4a4145f552c012be20211d7
MD5 (driver,AAPL,MacOS,PowerPC_reverse.bin) = dc939ef1b4a4145f552c012be20211d7


Also, I intentionally left the named FCode words in there as I thought it would help prevent a potential issue where tokenized FCode numbers may conflict with those already in the ROM

Fcode words get fcode numbers regardless if they are external or headers or headerless.

(as I wasn't able to detokenize the ROM into a re-tokenizable format).

You manually inserted the fcode of the compression algorithm and compressed ndrv into the fcode of the rom and then updated the checksum?

My DumpPCIRom.sh script can create tokenizable Forth text from fcode. (I need to add a small fix for non-PCI ROM fcode files, then I should make a GitHub repository). It's part of the process I used for modifying Nvidia GPU firmwares so they can work in Old World Macs for example.
 https://forums.macrumors.com/threads/question-how-powerful-of-a-graphics-card-will-work-in-a-beige-power-macintosh-g3.2303689/
It was also used nearly 20 years ago in the work for flashing PC Radeon graphics cards (7000, 8500, 9000, 9100) for Mac. Back then it was an MPW script.

I got around this issue by making a slight modification to the tokenizer source code, starting the FCode number at 0xA00 instead of 0x800, to ensure the generated FCode numbers would not conflict with those already there.

My tokenizer has a tokenizer command that you can put into the Forth text that changes the fcode number that will be used for the next word.

Code: [Select]

tokenizer[ a00 next-fcode ]tokenizer
I should fix my DumpPCIRom.sh script to add that line when it encounters a word definition that is not using the expected next fcode number as in the case with your compressed rom. next-fcode isn't usually necessary. It's there so that the tokenization can more closely match the original fcode since the names produced by DumpPCIRom.sh for headerless words includes the fcode number as part of the name like in this example:

Code: [Select]

: colon_definition_function_801         \ (801)     [0b5 0b7]
    my-space                            \           [103]
    9                                   \           [010]
    +                                   \           [01e]
    dup                                 \           [047]
    " config-b@"                        \           [012]
    $call-parent                        \           [209]
    5                                   \           [010]
    or                                  \           [024]
    swap                                \           [049]
    " config-b!"                        \           [012]
    $call-parent                        \           [209]
    ;                                   \           [0c2]


[V.Yakob]

EDIT: Updated ROM padding to 128K for Flashrom support.

Which computer do you use Flashrom on?
I don't have a PC, but I installed it via brew Flashrom on the PowerMac G4 MDD, and it seems that the PPC version does not support "--programmer satasii", despite the fact that it is stated in man.
I also installed Debian on this machine, but the result is the same.

In the near future I will try to upload this firmware through the programmer.

[joevt]

Which computer do you use Flashrom on?
I don't have a PC, but I installed it via brew Flashrom on the PowerMac G4 MDD, and it seems that the PPC version does not support "--programmer satasii", despite the fact that it is stated in man.
I also installed Debian on this machine, but the result is the same.

In the near future I will try to upload this firmware through the programmer.

For macOS, doesn't Flashrom require DirectHW to do PCI stuff or memory mapping stuff? I made a DirectHW for all macOS versions and architectures but I only tested the pci stuff with pciutils (lspci, setpci, pcitree.sh).
It might not have all the stuff required for satasii even for Intel macOS?
According to the flashrom makefile, satasii requires: DEPENDS_ON_RAW_MEM_ACCESS and DEPENDS_ON_LIBPCI
I think DirectHW.kext can do both of those but some other stuff might be required. I have not tried building or using flashrom.

[ssp3]

Can't you just rename the firmware file to ROMFILE.1S2 and replace with it the original that's inside the Mac "SeriTek1S2Flasher.app"?

[dosdude1]

EDIT: Updated ROM padding to 128K for Flashrom support.

Which computer do you use Flashrom on?
I don't have a PC, but I installed it via brew Flashrom on the PowerMac G4 MDD, and it seems that the PPC version does not support "--programmer satasii", despite the fact that it is stated in man.
I also installed Debian on this machine, but the result is the same.

In the near future I will try to upload this firmware through the programmer.

In order to use Flashrom, you must do so using a PC running Linux (though there may be a DOS version you can use as well, not sure). Flashrom under Mac OS does require DirectHW.kext, so I wouldn't bother with that as I don't know if DirectHW.kext can be compiled for PowerPC. I do also have this copy of Flashrom for PowerPC Linux; you could try that and see if that works: http://dosdude1.com/files/flashromppc.zip

[dosdude1]

Can't you just rename the firmware file to ROMFILE.1S2 and replace with it the original that's inside the Mac "SeriTek1S2Flasher.app"?

That application does not allow you to flash a "blank" card, or one that doesn't have one of the three "supported" 512K EEPROMs installed. I did already patch the OS9 version of this tool to be able to flash an unflashed card with any EEPROM, which I've attached here (already bundled with my compressed 128K ROM image), however in my testing I've found that sometimes the tool just locks up depending on the EEPROM you have installed. Definitely at least worth a try, though.

[V.Yakob]

I made a DirectHW for all macOS versions and architectures

Can you tell me how to compile a project to get kext? I only get a unix file.
I have very little experience in xcode, for all the time I collected 2-3 applications that I found on GitHub.

Can't you just rename the firmware file to ROMFILE.1S2 and replace with it the original that's inside the Mac "SeriTek1S2Flasher.app"?

The firmware update utility performs a check, and if the card is not the same, it simply does not display it.
I have a modified board, the utility displays it, but does not display the non-modified card.
I also tried to use the firmware utility from Wiebe -- sees the board, starts the process, but can't interact, the process fails.
Developers have always been cunning and defended themselves from people like us.

[V.Yakob]

I do also have this copy of Flashrom for PowerPC Linux; you could try that and see if that works: http://dosdude1.com/files/flashromppc.zip

I tried it, on 10.4 and Debian -- it doesn't work.
While I'm trying to build DirectHW.kext from the project joevt, but I can't do it yet.

[dosdude1]

Originally I was planning on using the C lzss implementation you use as it was, however when I was doing initial tests with it, I found that decompressing the NDRV after compressing it with the same implementation resulted in garbage.

Not sure why the decode didn't work for you. Did you try the C decode algorithm or only the fcode decode algorithm? I found that using the default C decode/encode, the decode of the encoded ndrv correctly produced bytes that match the original ndrv:

Code: [Select]

/Volumes/Work/Programming/lzss/lzss/lzss e driver,AAPL,MacOS,PowerPC.bin driver,AAPL,MacOS,PowerPC.lzss
text:  54648 bytes
code:  30527 bytes (55%)

/Volumes/Work/Programming/lzss/lzss/lzss d driver,AAPL,MacOS,PowerPC.lzss driver,AAPL,MacOS,PowerPC_reverse.bin

bbdiff driver,AAPL,MacOS,PowerPC.bin driver,AAPL,MacOS,PowerPC_reverse.bin
/Volumes/Work/Open Firmware and Name Registry/ROM Firmtek 1S2/dosdude/driver,AAPL,MacOS,PowerPC_reverse.bin and /Volumes/Work/Open Firmware and Name Registry/ROM Firmtek 1S2/dosdude/driver,AAPL,MacOS,PowerPC.bin are identical.

md5 driver,AAPL,MacOS,PowerPC.bin driver,AAPL,MacOS,PowerPC_reverse.bin
MD5 (driver,AAPL,MacOS,PowerPC.bin) = dc939ef1b4a4145f552c012be20211d7
MD5 (driver,AAPL,MacOS,PowerPC_reverse.bin) = dc939ef1b4a4145f552c012be20211d7


I just tested it again, and it definitely did work, so I must have just done something dumb I didn't realize. Guess I could try again with those parameters set back to their original values, but I don't see a need since it's already under 128K in size.

Also, I intentionally left the named FCode words in there as I thought it would help prevent a potential issue where tokenized FCode numbers may conflict with those already in the ROM

Fcode words get fcode numbers regardless if they are external or headers or headerless.

Yes, I quickly realized that LOL.

(as I wasn't able to detokenize the ROM into a re-tokenizable format).

You manually inserted the fcode of the compression algorithm and compressed ndrv into the fcode of the rom and then updated the checksum?

Correct, along with changing the size in the FCode header and the number of blocks in the PCI header. That was the only thing I could do not being able to re-tokenize the whole thing. I did try your DumpPCIRom.sh script, but it segfaulted when attempting to decode the original ROM when using that perl script, and produced too many untokenizable words when trying to decode it with the NDRVs stripped out (which apparently were what was causing the segfault).

I got around this issue by making a slight modification to the tokenizer source code, starting the FCode number at 0xA00 instead of 0x800, to ensure the generated FCode numbers would not conflict with those already there.

My tokenizer has a tokenizer command that you can put into the Forth text that changes the fcode number that will be used for the next word.

Code: [Select]

tokenizer[ a00 next-fcode ]tokenizer
I should fix my DumpPCIRom.sh script to add that line when it encounters a word definition that is not using the expected next fcode number as in the case with your compressed rom. next-fcode isn't usually necessary. It's there so that the tokenization can more closely match the original fcode since the names produced by DumpPCIRom.sh for headerless words includes the fcode number as part of the name like in this example:

Code: [Select]

: colon_definition_function_801         \ (801)     [0b5 0b7]
    my-space                            \           [103]
    9                                   \           [010]
    +                                   \           [01e]
    dup                                 \           [047]
    " config-b@"                        \           [012]
    $call-parent                        \           [209]
    5                                   \           [010]
    or                                  \           [024]
    swap                                \           [049]
    " config-b!"                        \           [012]
    $call-parent                        \           [209]
    ;                                   \           [0c2]


Ah, I didn't realize that.

[dosdude1]

I do also have this copy of Flashrom for PowerPC Linux; you could try that and see if that works: http://dosdude1.com/files/flashromppc.zip

I tried it, on 10.4 and Debian -- it doesn't work.
While I'm trying to build DirectHW.kext from the project joevt, but I can't do it yet.

Try my patched OS9 copy of the SeriTek updater utility posted above, it just may work. Of course ensure you're booted into Mac OS 9, not using Classic mode. I am still working on patching the OS X version, but it is significantly more difficult to do so.

[ssp3]

While I'm trying to build DirectHW.kext from the project joevt, but I can't do it yet.

Here, try this v1.6 (Intel/PPC). Just built it on Intel 10.6.8, xcode 3.2.6 with 10.4/10.5/10.6 sdk's. Still got two errors on build, but I don't want to dive deeper into it. Current master version produced even more errors, so I skipped it. I'm flying blind here, so it's not tested.

* Why don't you guys provide pre-built binaries on you githubs? Why do we all have to go thru all the pains? 

** BUILD SUCCEEDED **

cc DirectHW.c -dynamiclib -framework IOKit -o libDirectHW.dylib
In file included from DirectHW.c:19:
MacOSMacros.h:54:6: warning: #warning x86_64
MacOSMacros.h:70:6: warning: #warning SDK 10.6+
cc -static -c DirectHW.c -o libDirectHW.a
In file included from DirectHW.c:19:
MacOSMacros.h:54:6: warning: #warning x86_64
MacOSMacros.h:70:6: warning: #warning SDK 10.6+
mv libDirectHW.dylib build/Release/libDirectHW.dylib
mv: rename libDirectHW.dylib to build/Release/libDirectHW.dylib: No such file or directory
make[1]: *** [libs] Error 1
make: *** [directhw] Error 2

[V.Yakob]

Here, try this v1.6. Just built it on Intel 10.6.8, xcode 3.2.6 with 10.4/10.5/10.6 sdk's.

Cool, thanx, I don't have OS X 10.6, I tried building for Intel 13.2 and PPC 10.4.

Try my patched OS9 copy of the SeriTek updater utility posted above, it just may work. Of course ensure you're booted into Mac OS 9, not using Classic mode. I am still working on patching the OS X version, but it is significantly more difficult to do so.

I tried the utility, it works!!
But this board apparently requires additional modifications, because after flashing the PowerMac G4 MDD (2003) won't boot. The boot hangs on the gray screen, the USB does not work (there was a trembling on my back).
Power on -> bong -> a boot disk search icon appears -> grey screen.
If you pull out the board, the computer is working properly.

EPROM Chip -- 39SF010A
Voltage regulator (U1) -- AMS 2911 1.8 0525
Voltage regulator (U2) -- AMS 2911CD 3.3 0323

There are no other boards for testing yet.

[ssp3]

For what it's worth, there's this interesting part by (S)ATAman in Firmtek pirate thread:
http://macos9lives.com/smforum/index.php/topic,5268.msg38303.html#msg38303
Note that he mentions Sil3112DeviceNub.kext

And then there's a "InstallPPC.app" on SeriTek1S2_5.3.1.dmg, which has uncompressed
FT_ATA_Sil3112.kext and  Sil3112DeviceNub.kext
inside and which probably does just what (S)ATAman describes.
I guess, if one replaces the FT_ATA_Sil3112.kext with patched one (uncompressed) the app could be used to assist in flashing the cards.

First, we need to tell the system that there is an ATA device nub.
The simplest:

1) sudo cp -r  /Volumes/SeriTek2SE4Flash/Sil3112DeviceNub.kext /tmp
2) sudo cp -r /tmp/Sil3112DeviceNub.kext /System/Library/Extensions

Loading the kernel extension temporarily:
3) sudo cp -r /Volumes/SeriTek2SE4Flash/FT_ATA_Sil3132E.kext /tmp
4) sudo kextload /tmp/FT_ATA_Sil3132E.kext

After few seconds the Silicon Image 3124 controller is operational.
Do not attach any drives to it, proceed with flashing.

[ssp3]

I tried the utility, it works!!
But this board apparently requires additional modifications, because after flashing the PowerMac G4 MDD (2003) won't boot.

dosdude1, I think it would be cool if you could release the OS9 only stripped down version that does not use any kind of compression. One less variable in testing.

[dosdude1]

Here, try this v1.6. Just built it on Intel 10.6.8, xcode 3.2.6 with 10.4/10.5/10.6 sdk's.

Cool, thanx, I don't have OS X 10.6, I tried building for Intel 13.2 and PPC 10.4.

Try my patched OS9 copy of the SeriTek updater utility posted above, it just may work. Of course ensure you're booted into Mac OS 9, not using Classic mode. I am still working on patching the OS X version, but it is significantly more difficult to do so.

I tried the utility, it works!!
But this board apparently requires additional modifications, because after flashing the PowerMac G4 MDD (2003) won't boot. The boot hangs on the gray screen, the USB does not work (there was a trembling on my back).
Power on -> bong -> grey screen.
If you pull out the board, the computer is working properly.

EPROM Chip -- 39SF010A
Voltage regulator (U1) -- AMS 2911 1.8 0525
Voltage regulator (U2) -- AMS 2911CD 3.3 0323

There are no other boards for testing yet.

The voltage regulators could be causing this issue, apparently that can happen on some QuickSilver and/or MDD systems with some Sil3112 cards. Did you try this card in another machine? It does look like you were able to get it to boot in something, considering System Profiler sees it. Oh, and one other thing. Make sure you do NOT have two cards installed. The drivers in my ROM have been modified to remove the EEPROM ID check, and will conflict with another card if it still has the stock, unmodified ROM.