Author Topic: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)  (Read 150934 times)

Offline joevt

  • Enthusiast Member
  • ***
  • Posts: 65
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #160 on: January 25, 2023, 10:35:19 AM »
A couple things: the fcode appears to be missing the fcode-end at the end of the rom.
Code: [Select]
" 5.1.3.1S2"
encode-string
" rom-revision"
property

fcode-end

pci-end

And this function is weird because the if ... then crosses a repeat. It's like the if is a conditional break or leave instruction.
Code: [Select]
: colon_definition_function_8a2 \ (8a2) [0b5 0b7]
get-msecs \ [125]
begin \ [0b1]
readstatusreg \ [859]
dup \ [047]
0<> \ [035]
over \ [048]
0ff \ [010]
<> \ [03d]
and \ [023]
swap \ [049]
0c0 \ [010]
and \ [023]
40 \ [010]
<> \ [03d]
and \ [023]
if \ (0x14) [014]
get-msecs \ [125]
over \ [048]
- \ [01f]
1388 \ [010]
< \ [03a]
while \ (0x6) [014]
repeat \ (0xffd5) [013 0b2]
then \ [0b2]
drop \ [046]
; \ [0c2]
I don't think it's a problem. It just looks weird. My tokenizer accepts this interpretation and produces the following fcode that results in the same interpretation
Code: [Select]
001418: b(:) \ [0x0b7]
001419:     get-msecs \ [0x125]
00141B:     b(<mark) \ [0x0b1]
00141C:         readstatusreg \ [0x859]
00141E:         dup \ [0x047]
00141F:         0<> \ [0x035]
001420:         over \ [0x048]
001421:         b(lit) \ [0x010] 0xff
001426:         <> \ [0x03d]
001427:         and \ [0x023]
001428:         swap \ [0x049]
001429:         b(lit) \ [0x010] 0xc0
00142E:         and \ [0x023]
00142F:         b(lit) \ [0x010] 0x40
001434:         <> \ [0x03d]
001435:         and \ [0x023]
001436:         b?branch \ [0x014] 0x14
001439:             get-msecs \ [0x125]
00143B:             over \ [0x048]
00143C:             - \ [0x01f]
00143D:             b(lit) \ [0x010] 0x1388
001442:             < \ [0x03a]
001443:             b?branch \ [0x014] 0x6
001446:                 bbranch \ [0x013] 0xffffffd5
001449:             b(>resolve) \ [0x0b2]
00144A:         b(>resolve) \ [0x0b2]
00144B:     drop \ [0x046]
00144C:     b(;) \ [0x0c2]

Offline dosdude1

  • Enthusiast Member
  • ***
  • Posts: 42
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #161 on: January 25, 2023, 11:09:43 AM »
It comes out to around 138K. Most of the space is being taken up by the OS X NDRV, but that is already compressed (it's an mkext shoved in as binary data).

Indeed, starting from around offset 0x21d70 it contains all zeroes.
I know nothing about ROM dissasembling, their structures etc. (I wish, I knew, since I'm trying to finalize my WD MyBook Studio Firewire jailbreak), but, can't the firmware, by someone skilled in the trade
a) be optimized at the machine code level manually, or/and
b) decompiled and then re-compiled with different optimization level (min size) or with different compiler
to get the final file size under 128k?

I've loaded the firmware in Hopper and, when looking at some of the procedures in pseudo-code mode and checking/unchecking 'remove potentially dead code' checkbox, there were some differences.
It's not exactly that simple... There are two NDRVs in the ROM that need to be patched. One for OS X, and one for OS 9. There is no EEPROM ID check in the actual FCode itself. For the OS X one, the first thing you need to do is extract the binary blob into a raw binary file, which will then create an mkext file you can work with. Next, you'll need to unpack the mkext. You can do so using "mkextunpack" under OS X. This should be done under an older OS X version (I used 10.4 Tiger) on a PowerPC machine for best results, though for this step it doesn't really matter. Once you have the kexts extracted, NOW you can open them in IDA, or other disassembler of choice, and begin reverse-engineering. "FT_ATA_Sil3112.kext" is the one where the EEPROM ID check is implemented. You WILL need at least some knowledge of PowerPC assembly in order to make much sense of it, and make any changes. Once you determine and make desired binary patches, you will then need to reverse the extraction process... Use OS X 10.4 on PPC to create new mkext with modified kexts (you can use "kextcache"), and get the modified mkext back into the EEPROM -- That's the tricky part, which I had to make a few scripts for, use FCode tokenizer, etc.

The OS 9 binary is quite a bit easier... All you need to do for that one is extract the binary blob into a binary file, and start reverse-engineering... It is not compressed or packed in any way.

Lastly, the zeroes at the end of the ROM are not needed, they are just there from when the ROM image was dumped from a 512K EEPROM. They can simply be removed.

Offline dosdude1

  • Enthusiast Member
  • ***
  • Posts: 42
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #162 on: January 25, 2023, 11:32:17 AM »
I looked at the dosdude 1S2_512-patched.rom.

My tokenizer saves 199 bytes by converting numbers like 0,1,2,3 from 5 bytes to 1 byte. The fcode is 138343 bytes. 135.1 KiB.
Without the fcode that encodes the ndrv and mkext, the fcode is 10595 bytes. 10.3 KiB.
lzss cannot compress the mkext - it grows from 70069 to 75172 bytes.
lzss does a pretty good job on the ndrv - it shrinks from 54648 to 30527 bytes.
That's probably sufficient to get the size of the rom to less than 128 KiB (after you add forth code that does lzss decompression). You wouldn't have to compress all the fcode, or even the fcode that produces the ndrv. Instead compress the ndrv binary, include that as a series of encode-bytes+ in the forth code, then add forth code that uses the lzss decompress function on the result of that to create the blob for the ndrv property.

The lzss decompression function is less than 50 lines of C code so it should be pretty easy to convert it to forth.
This is probably the best way to go... I'll start working on this and see if I can get that implemented.

Offline ssp3

  • Platinum Member
  • *****
  • Posts: 710
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #163 on: January 25, 2023, 01:53:00 PM »
It's not exactly that simple...
Thanks for the insight. I learn something new every day  8)
P.S. Love your Mac OS X USB Drive Creator. Excellent tool!
If you're not part of the solution, you're part of the problem.

Offline joevt

  • Enthusiast Member
  • ***
  • Posts: 65
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #164 on: January 25, 2023, 05:54:04 PM »
It's not exactly that simple... There are two NDRVs in the ROM that need to be patched. One for OS X, and one for OS 9. There is no EEPROM ID check in the actual FCode itself. For the OS X one, the first thing you need to do is extract the binary blob into a raw binary file, which will then create an mkext file you can work with. Next, you'll need to unpack the mkext. You can do so using "mkextunpack" under OS X. This should be done under an older OS X version (I used 10.4 Tiger) on a PowerPC machine for best results, though for this step it doesn't really matter. Once you have the kexts extracted, NOW you can open them in IDA, or other disassembler of choice, and begin reverse-engineering. "FT_ATA_Sil3112.kext" is the one where the EEPROM ID check is implemented. You WILL need at least some knowledge of PowerPC assembly in order to make much sense of it, and make any changes. Once you determine and make desired binary patches, you will then need to reverse the extraction process... Use OS X 10.4 on PPC to create new mkext with modified kexts (you can use "kextcache"), and get the modified mkext back into the EEPROM -- That's the tricky part, which I had to make a few scripts for, use FCode tokenizer, etc.

The OS 9 binary is quite a bit easier... All you need to do for that one is extract the binary blob into a binary file, and start reverse-engineering... It is not compressed or packed in any way.

Lastly, the zeroes at the end of the ROM are not needed, they are just there from when the ROM image was dumped from a 512K EEPROM. They can simply be removed.
What's the point of an mkext in a PCI rom? If the Mac OS X kernel can use the mkext, it can also use the extracted kexts saved to disk. I suppose the mkext makes using read only media easier (can you connect sata CD or DVD's to the SATA card?). Also, without the mkext, you need to be able to copy the kexts to the drive. I suppose if you were using XPostFacto, the kexts could be copied to a helper disk (built in SCSI or ATA).

Anyway, if you think there are PCI cards with 64K ROMs, then removing the mkext would make the firmware fit in those cases.

Offline joevt

  • Enthusiast Member
  • ***
  • Posts: 65
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #165 on: January 25, 2023, 06:15:53 PM »
This is probably the best way to go... I'll start working on this and see if I can get that implemented.
Maybe consider using dingusppc to test your decompression algorithm.

I use a script named run to start dingusppc:
Code: [Select]
#!/bin/bash
cd /Volumes/Work/Programming/dingusppc/joevt-dingusppc/build/bin
thewindow="$(osascript -e '
tell application "Terminal"
    set currenttab to do script ""
    tell currenttab
        tell current settings
            set custom title to "dingusppc Modem Port"
            set title displays device name to false
            set title displays shell path to false
            set title displays window size to false
            set title displays settings name to false
            set number of columns to 140
            set number of rows to 80
        end tell
        do script "clear; sleep 2; cd /Volumes/Work/Programming/dingusppc/joevt-dingusppc/build/bin; socat UNIX-CLIENT:dingussocket -,cs8,parenb=0,echo=0,icanon=0,isig=0,icrnl=0" in currenttab
    end tell
    return currenttab
end tell
')"

# 2.4
./dingusppc -d -b "/Volumes/Work/Open Firmware and Name Registry/ROM PowerPC Mac/ROM PM G3/ROMs/@FFC00000 len-400000" --serial_backend=socket --rambank1_size=256 --rambank2_size=256 --rambank3_size=256 --fdd_img=FATFLOPPY.img --pci_B1=Nv47

sleep 1
osascript -e 'tell app "Terminal" to close '"${thewindow#*of }"

Then I execute it using ./run
Include -d for debug. When you start dingusppc, it will show the debug prompt, then you can printenv and setenv and nvedit to modify Open Firmware settings. I set input and output device to ttya and auto-boot? to false so it will enter Open Firmware after you leave debug mode by typing go

Include --serial_backend=socket so that serial port will go to a socket (not available for Windows).

The run script uses AppleScript to automatically open that socket into a new Terminal.app window. Or you could manually enter the socat command in a new Terminal.app window that you create yourself.

Use my fork at https://github.com/joevt/dingusppc
Use the xcode-project fork.
It requires installing sdl2 which you can do with homebrew.

Create code for the sata pci card similar to promise20269.c   This implementation doesn't handle any of the registers but it is enough to load the rom. Since it doesn't handle the registers, the fcode takes a while to complete but it does complete. Another option is to just replace the rom (since I didn't include it in the fork) because you just want to test decompression of the ndrv.

To add a PCI card to slot C1, add to the command line:
 --pci_C1=PCIDeviceName

Offline ssp3

  • Platinum Member
  • *****
  • Posts: 710
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #166 on: January 26, 2023, 06:20:19 AM »
Question - are there any tools available that can extract above mentioned mkext and ndrv or do I have to find them manually in hex editor? Preferably without compiling that requires capstone, macports, brew, High Sierra and all that kind of stuff  ;) Asking out of pure curiosity.
If you're not part of the solution, you're part of the problem.

Offline joevt

  • Enthusiast Member
  • ***
  • Posts: 65
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #167 on: January 26, 2023, 08:27:53 AM »
Question - are there any tools available that can extract above mentioned mkext and ndrv or do I have to find them manually in hex editor? Preferably without compiling that requires capstone, macports, brew, High Sierra and all that kind of stuff  ;) Asking out of pure curiosity.
It's a process. Use detok to convert a rom to text (or use my DumpPCIRom.sh script which uses detok and produces forth code that can be used with toke). Extract the lines that encode the ndrv or mkext properties, use search and replace to remove the non-hex stuff, use xxd to convert hex to binary, research mkext, google utilities that can build mkext or extract kexts.
 https://forums.macrumors.com/threads/question-how-powerful-of-a-graphics-card-will-work-in-a-beige-power-macintosh-g3.2303689/

Offline IIO

  • Platinum Member
  • *****
  • Posts: 4439
  • just a number
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #168 on: January 26, 2023, 11:15:54 AM »

What's the point of an mkext in a PCI rom?

i was thinking something similar.

in case it is impossible to fit everything onto the card, maybe it could make sense to provide the OSX functionality with a premade kext installer and have only the OS9 driver on the card?

the requirement of installing a driver for a card is not a big deal for the enduser.
insert arbitrary signature here

Offline dosdude1

  • Enthusiast Member
  • ***
  • Posts: 42
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #169 on: January 26, 2023, 11:31:40 AM »

What's the point of an mkext in a PCI rom?

i was thinking something similar.

in case it is impossible to fit everything onto the card, maybe it could make sense to provide the OSX functionality with a premade kext installer and have only the OS9 driver on the card?

the requirement of installing a driver for a card is not a big deal for the enduser.
That's definitely a possibility, though my goal was to retain the original ROM's functionality. If compressing the OS 9 NDRV doesn't end up working out, I'll just modify the ROM and rip out the OS X mkext. The kexts can indeed simply be installed into OS X as normal.

Offline ssp3

  • Platinum Member
  • *****
  • Posts: 710
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #170 on: January 26, 2023, 12:11:52 PM »
It's a process. Use detok to convert a rom to text (or use my DumpPCIRom.sh script which uses detok and produces forth code that can be used with toke). Extract the lines that encode the ndrv or mkext properties, use search and replace to remove the non-hex stuff, use xxd to convert hex to binary, research mkext, google utilities that can build mkext or extract kexts.
 https://forums.macrumors.com/threads/question-how-powerful-of-a-graphics-card-will-work-in-a-beige-power-macintosh-g3.2303689/

Eek! That's way too much body movement for someone who just curious. (Disclaimer - I don't have the card and don't plan to use any).
EDIT. Found "FT_ATA_Sil3112.kext v.5.3.1" inside the InstallPPC.app that's on 5.3.1 firmware/software dmg, although its binary is 255k.
« Last Edit: January 26, 2023, 06:32:43 PM by ssp3 »
If you're not part of the solution, you're part of the problem.

Offline refinery

  • Gold Member
  • *****
  • Posts: 383
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #171 on: January 26, 2023, 06:19:05 PM »
I would think the purpose of having the mkext would be for cases where somebody is trying to boot off the disk on the card... if you have no disk on the internal bus, how's it going to load a driver for the card from a disk it doesnt know how to access yet at that stage of booting?
got my mind on my scsi and my scsi on my mind

Offline IIO

  • Platinum Member
  • *****
  • Posts: 4439
  • just a number
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #172 on: January 26, 2023, 06:44:38 PM »
I would think the purpose of having the mkext would be for cases where somebody is trying to boot off the disk on the card... if you have no disk on the internal bus, how's it going to load a driver for the card from a disk it doesnt know how to access yet at that stage of booting?

of course you are right, i forgot about that scenario. you wouldnt be able to boot into X from it.
insert arbitrary signature here

Offline dosdude1

  • Enthusiast Member
  • ***
  • Posts: 42
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #173 on: January 26, 2023, 08:50:07 PM »
I would think the purpose of having the mkext would be for cases where somebody is trying to boot off the disk on the card... if you have no disk on the internal bus, how's it going to load a driver for the card from a disk it doesnt know how to access yet at that stage of booting?

of course you are right, i forgot about that scenario. you wouldnt be able to boot into X from it.
Once you got the kext into the target install of OS X on the target drive, it will be able to boot from it just fine... Just a bit of a pain, having it embedded in the ROM is much more convenient.

Offline joevt

  • Enthusiast Member
  • ***
  • Posts: 65
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #174 on: January 26, 2023, 11:18:33 PM »
I would think the purpose of having the mkext would be for cases where somebody is trying to boot off the disk on the card... if you have no disk on the internal bus, how's it going to load a driver for the card from a disk it doesnt know how to access yet at that stage of booting?
You are saying that Mac OS X can't boot from a disk unless there's a kext in firmware.

So how does Mac OS X boot from a built-in SCSI or ATA disk? Those do not have kexts in firmware.

I believe Open Firmware is used to read kexts or kext cache but you can check BootX source code to verify that.
 https://www.oreilly.com/library/view/mac-os-x/0596003560/ch02s01s01.html
 https://flylib.com/books/en/3.126.1.47/1/
 https://github.com/apple-oss-distributions/BootX/blob/814114e6a6cf10dfa512c520f5a1c1fb9c58a432/bootx.tproj/include.subproj/fs.h
BootX contains HFS+ file system driver that calls Open Firmware.

Offline ssp3

  • Platinum Member
  • *****
  • Posts: 710
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #175 on: January 27, 2023, 03:56:15 AM »
Apparently, kextcache by default produces compressed mkexts. Probably that's the reason why lzss couldn't compress it further.
What happens if one makes uncompressed mkext and then compresses it with lzss? Any gain in size reduction?
(I can't test it myself, I have neither original 513 mkext, nor any 10.4/10.5 PPC machines).

From kextcache man:
Quote
-compressed
              Compress the mkext or prelinked kernel (enabled by default).
-uncompressed
              Do not compress the mkext or prelinked kernel.  If specified as
              the only other argument with -c, uncompresses an existing pre-
              linked kernel file in place.

Also, what about upx or mpress? Do they exist for PPC and, if so, can they be used on kexts?
« Last Edit: January 27, 2023, 04:31:54 AM by ssp3 »
If you're not part of the solution, you're part of the problem.

Offline ssp3

  • Platinum Member
  • *****
  • Posts: 710
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #176 on: January 27, 2023, 05:04:59 AM »
Once you got the kext into the target install of OS X on the target drive, it will be able to boot from it just fine... Just a bit of a pain, having it embedded in the ROM is much more convenient.

Re. embedded in ROM. There's this bit from SeriTek/1S2, version 5.3.1 install instructions:
Quote
WHEN TO UPGRADE FIRMWARE
If your current firmware is working for you, FirmTek does not
recommend updating the firmware.
<snip>
One exception to this rule: Leopard users will want to upgrade. The
new 5.3.1 firmware corrects a Leopard wake on sleep issue. However,
Tiger users that are not experiencing problems should stay with
their existing firmware.

So, to cover the Leopard wake on sleep issue, one has to be prepared to deal with 5.3.1 firmware, which is 175k in size compared to 138k of 5.1.3.
If you're not part of the solution, you're part of the problem.

Offline peeperpc

  • Enthusiast Member
  • ***
  • Posts: 48
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #177 on: January 27, 2023, 10:04:46 AM »
Once you got the kext into the target install of OS X on the target drive, it will be able to boot from it just fine... Just a bit of a pain, having it embedded in the ROM is much more convenient.

Re. embedded in ROM. There's this bit from SeriTek/1S2, version 5.3.1 install instructions:
Quote
WHEN TO UPGRADE FIRMWARE
If your current firmware is working for you, FirmTek does not
recommend updating the firmware.
<snip>
One exception to this rule: Leopard users will want to upgrade. The
new 5.3.1 firmware corrects a Leopard wake on sleep issue. However,
Tiger users that are not experiencing problems should stay with
their existing firmware.

So, to cover the Leopard wake on sleep issue, one has to be prepared to deal with 5.3.1 firmware, which is 175k in size compared to 138k of 5.1.3.

I tried 5.3.1 firmware for a day, with 10.5.8 as the main OS (MDD Mac).

- With 5.1.3, using a 2TB Seagate drive with an ATA/SATA adapter as the boot drive, it takes noticeably longer before the system finds the drive and starts booting from it. But with 5.3.1, the startup time is back to normal. Yes, the boot drive is on the internal ATA100 bus but somehow the firmware can still interfere.

- With 5.1.3, I can use smartmontools with the drive connected to the card. But with 5.3.1, I can't.

- I've had no sleep issue with 5.1.3 nor 5.3.1.

In the end, I reverted to 5.1.3. Since smartmontools is more important for me than shorter boot time.

Offline dosdude1

  • Enthusiast Member
  • ***
  • Posts: 42
  • New Member
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #178 on: February 03, 2023, 12:27:18 PM »
I have another absolutely MASSIVE update for you all... I was able to implement an LZSS decompression algorithm in the ROM, and use it to decompress a stored LZSS-compressed version of the OS9 NDRV. As such, the ROM has been reduced in size to UNDER 128K! This means that you can now flash it onto cards WITHOUT having to replace the original 128k EEPROM! Of course this image also includes my patched OS9 and OS X NDRVs that remove the EEPROM ID check. The updated ROM image is attached, along with my custom Forth LZSS decompression implementation. Hope you all enjoy!

EDIT: Updated ROM padding to 128K for Flashrom support.
« Last Edit: February 03, 2023, 02:08:30 PM by dosdude1 »

Offline ssp3

  • Platinum Member
  • *****
  • Posts: 710
Re: Disk Speed Upgrades (aka The Bootable PCI SATA & SSD thread)
« Reply #179 on: February 03, 2023, 12:33:39 PM »
Cool!  8)
Silly question - does compression impact data transfer rate anyhow?
If you're not part of the solution, you're part of the problem.