Classic Mac OS Software (Discussions on Applications) > Hacking the System, Mac OS 9.3, and Beyond !

CDG5 release

(1/4) > >>

Nanopico and Elliot have been working on
--- Quote --- a brand new MPW-based build system for the Mac OS ROM, with lots of information about its internals, and opportunities for hacking.
--- End quote ---

Elliot want you all know that :

--- Quote ---# CDG5

CDG5 is a project aiming to reverse, patch and rebuild the Macintosh NewWorld ROM. The contents of this repository can be built using the Macintosh Programmer's Workshop (MPW) into a complete "Mac OS ROM" file, which can then be used to boot Mac OS 9.2.2 on any compatible NewWorld Mac.

Reversing the ROM is an ongoing, open-ended process (and heaps of fun!). It was put on hold for a couple of months while we moved our build system away from a motley collection of Python scripts. This repo currently uses only unlinked binaries in lieu of reversed code, but this will improve very quickly. Binaries are in `:Prebuilt:LateNewWorld:`, and we prefer to keep them in Rez format so that we can apply compile-time conditionals. (Once we understand MPW's PPCAsm better, our annotated NanoKernel disassembly will be merged in.)

## Build

Because we deal with generously sized binaries, you should increase MPW's memory allocation to well over 8 MB before you build. Clone this repo, set MPW's current directory to your local clone, and run the script `:Tools:FixRepo` from your Worksheet:


(Hit Command-Return or the numpad Enter key to issue the command.)

The `:Make:Build` script wraps the `:Make:NewWorld.make` makefile. Set MPW's current directory to the cdg5 repo, then:

:Make:Build NewWorld

Your file will end up in `:BuildResults:NewWorld:Mac OS ROM`.

As we develop useful patches to the ROM code, we will conditionalize them using feature flags. Flags are declared in `:Make:FeatureList` and enabled in `:Make:NewWorld.make`.

## NewWorld ROM

The first Macintosh shipped with a 64 KB ROM containing much of the high-level Macintosh Toolbox as well as low-level hardware-specific code. While the Mac still requires a disk with a System file to boot, having much of the OS in ROM saves loading it from a slow disk drive and conserves precious RAM. Buggy code in the ROM can be overridden with code on disk. The Toolbox ROM also makes it hard to produce an unauthorized Mac clone, and allows a graphical interface to be shown very early in the boot process.

Subsequent Macs have progressively larger ROMs, the size doubling with every step. The "Universal" 68k ROMs contain structures describing the hardware on several supported machines, and select one of these at runtime. Power Macs started shipping with a 4 MB "RISC" ROM, which has the bottom three megabytes laid out like a familiar Toolbox ROM and the remaining one megabyte dedicated to PowerPC-specific code and structures.

Machines with fast disks and plentiful, cheap RAM don't benefit as much from a large built-in ROM. Starting with the iMac, Apple's improved Open Firmware implementation loads and runs a "Mac OS ROM" file in the System Folder. Macs that do this are termed "NewWorld" Macs, and are equal to the set of Macs that have a built-in USB port.

## Structure

This section attempts to correlate the structure of the NewWorld ROM and boot process with this source tree.

### Open Firmware

An Open Firmware implementation is present in all PCI-equipped Macs (this excludes the first, NuBus-equipped 61xx/71xx/81xx Power Macs).

To boot from a partition, the Open Firmware on a NewWorld Mac (and possibly any CHRP machine?) searches the "blessed" folder for a file of type `tbxi` and parses the `\<CHRP-BOOT>` structure within. On Mac OS 8-9 this file is `System Folder:Mac OS ROM`. On Mac OS X it is `System:Library:CoreServices:BootX`. Open Firmware seems to ignore everything in this file after the ASCII "EOF" character, leaving us free to append a lot of useful data to it. Those parts of the file that are useful in early boot are linked by the MPW shell script `:NewWorld:WrapMacOSROM` from several other sources. Some code fragments used much later in the boot process are also appended to the data fork for convenience. Lastly, an Adler-32 checksum is appended to the file by the script `:NewWorld:ChecksumMacOSROM`, which wraps a small C utility.

For the file to be bootable, the `\<COMPARTIBLE>` field must match the "compatible" key in the root of the machine's Open Firmware device tree. If they do not match, the partition will not be treated as bootable and will not show up in the Open Firmware graphical boot picker.

Open Firmware then executes the Forth code in the `\<BOOT-SCRIPT>` code of the file, which is copied by `WrapMacOSROM` from `:NewWorld:BootScript.fs`. This code checks the Adler-32 checksum mentioned above, allocates room in RAM for the the Trampoline ELF file and its "Parcels" data structure, copies these from fixed offsets in the file, and hands control to the Trampoline. Instead of a Parcels structure, Mac OS ROM v1.x files (those that shipped with NewWorld G3 Macs before the first Power Mac G4s were released) have a simple LZSS-compressed 4 MB Toolbox ROM. Newer v2.x files have been called "NewNewWorld" ROMs.

### Trampoline and Parcels

The Trampoline prepares an environment for the PowerPC Toolbox ROM to run in, and then hands it control. It contains some fascinating strings, and its debug output can be captured over telnet. But being contained in an ELF file, it is not easy to build in MPW. I have a couple of ideas about this, and would like suggestions.

The Trampoline understands both the native PowerPC and emulated 68k parts of the Mac OS, and initialises structures to be used by both: (this is not a complete list)
- a mapping from the PowerPC to the 68k interrupt architecture (The PowerPC has one interrupt line, managed by an OpenPIC chip, while the 68xxx chips have eight, with specific roles assigned by the Mac OS.)
- the ConfigInfo struct, describing what the PowerPC NanoKernel needs to know about the hardware
- the "Universal" ProductInfo struct, inherited from the 68k ROMs, describing what the 68k ROM code needs to know about its emulated environment
- a flattened version of the device tree, modified as directed by the Toolbox parcels
- a template page table for the NanoKernel to use

The 4 MB Toolbox ROM and several native PowerPC drivers are stored as parcels. The `:Tools:ToolSource:ParcelLayout.c` tool links the Parcels blob accoding to the description in `:NewWorld:Parcels.r`. Blobs get compressed using the LZSS algorithm, which is necessary to keep the data fork of the Mac OS ROM file within Open Firmware's size limit of 4 MB. Our LZSS implementation in `:Tools:ToolSource:LZSS.c` is naive with poor but good-enough compression, but it is fast enough not to bother caching compressed blobs. This is the third rewrite. There will be no more!

### NanoKernel and Emulator

The top megabyte of the PowerPC ROM contains code and structures needed to bootstrap an emulated 68k environment. The `:RISC:` folder contains PowerPC assembly files (most notably `NanoKernel.s` and `Emulator.s`) and `:RISC:RISC.make` contains build commands.

The NanoKernel (camel case preserved for authenticity) is at the very bottom of the Mac OS stack. It controls the PowerPC exception vectors and memory map, and is the only code (except for the firmware's "Runtime Abstraction Services" (RTAS) code) to run with virtual memory disabled.

Because hardware initialization code was moved from the OldWorld Toolbox ROM to the NewWorld Trampoline, the Trampoline hands control to the NanoKernel directly. A C prototype for the handover call would look like this:
void NKInit(
   /*r3*/ struct ConfigInfo *ConfigInfop,
   /*r4*/ struct ProcessorInfo *ProcessorInfop,
   /*r5*/ struct SysInfo *SysInfop,
   /*r6*/ /*not sure*/,
   /*r7*/ OSType RTASEnable, /* 'rtas' or 0 */
   /*r8*/ void RTASDispatch(char *),
   /*r9*/ struct HWInfo *HWInfo

Gary Davidian wrote the first version of the NanoKernel, along with the Emulator, for the first "Piltdown Man" (PDM) Power Macs. Rene Vega later rewrote the NanoKernel to add symmetric multiprocessing support and many other goodies. Version 1 shipped in the ROM of every OldWorld Power Mac, and in Mac OS ROM file versions up to 1.4. Version 2 never shipped in a physical ROM, but can be loaded from a 'krnl' resource in the System file during boot to replace a running v1 kernel. It also came as part of Mac OS ROM file versions 1.6 and later. Both forms, in the System file and the Toolbox image, debuted with Mac OS 8.6.

While NK v1 only really arbitrates between the emulated 68k and native PowerPC execution modes, v2 allows multiple preemptible tasks to be scheduled, with their own protected address spaces and sophisticated interprocess communications. One of these tasks is the "blue" Mac OS task, which contains the entire cooperative Mac OS environment. The NanoKernel delegates most PowerPC interrupts to the blue task as 68k interrupts. The Emulator is non-reentrant and runs only inside the blue task.

The ability to spawn tasks outside the blue task is exposed by version 2 of the Multiprocessing Services API (available on Carbon too!). The later point releases of Mac OS 9 provide glue code for preemptive tasks ("MTasks") to call blue-only routines remotely.

While the Emulator only ever runs within the blue task, its structures are global, and the NanoKernel is aware of it. For now we know fairly little about the Emulator, but it is likely we will need to look more closely, because it seems to be the immediate barrier to running Mac OS 9 on the Mac mini. When the blue task is first scheduled, the emulated 68k acts like a freshly booted chip, and begins executing code in the Toolbox ROM.

### 68k ROM

The 68k ROM has three main components: the MainCode image, the ROM resources, and the Slot 0 DeclROM. `RomLayout` builds a 3 MB 68k ROM image according to the layout specified in `:Make:NewWorldLayout.r`.

#### MainCode image

This monolithic block of 68k code lives at the bottom of ROM. It begins with a somewhat declarative header and contains bootstrap code for a (real or emulated) 68k, code to identify the host machine model, code to chainload the boot blocks and System file from a startup partition, code implementing the A-trap mechanism, and all the 68k Toolbox Managers. A lot of this code should be very easy to unlink. When the 68k Emulator "wakes up", it follows the 68k RESET vector into the MainCode image.

#### ROM resources

There are several (obviously nonrelocatable) Memory Manger blocks in the ROM, which the Resource Manger sees as loaded resources. The resource list for `RomLayout` comes from `:Resources:RomResources.r`. Both `RomLayout` and `ParcelLayout` get their binaries from the resource files built in `:BuildResults:NewWorld:Rsrc:`.

Among their many uses, ROM resources are the preferred way to put high-level PowerPC code in the Toolbox ROM. There is a half-baked mechanism for ROM resource selection at runtime, using 64-bit fields, but it is only used to select some resources related to AppleTalk and floating-point calculation. We could possibly resurrect it to select resources on newer Macs.

#### Empty space

Unused space in the Toolbox ROM is filled with the pattern 'kckckckc'. You can find Kurt Clark's name elsewhere too. On a NewWorld Mac, a physical RAM page full of 'kc's will be unmapped from the Toolbox ROM and used elsewhere, saving about 800 KB of memory.

#### Slot 0 DeclROM

The deprecated Slot Manager uses "Declaration ROMs" to describe NuBus cards. The logic board sits in a fictitious "Slot 0" and has its own DeclROM, which can contain drivers, at the top of the 68k Toolbox ROM. On later Power Macs this structure is tiny and vestigial. We have only bothered to extract it as a raw binary, `DeclData`.

## Classic Environment

The Classic Environment ("BlueBox") also used a Parcels structure, albeit with fewer hardware drivers. Comments in `:NewWorld:Parcels.r` describe what is in each version of the structure.

## Versions

Most binaries have been taken from Mac OS ROM v9.6.1 in Apple's NetBoot package. This has the best hardware support of any extant version, although it seems not to support the Mirrored Drive Door Power Mac G4s very well. For this reason, some newer code from the v10.2.1 "Custer" ROM is also in the repo. For now, it can be enabled with the feature flags CusterBacklightParcel and CusterUSBShimKeyboard.

iMic's patched "Generic" Mac OS ROM v10.2.1 has been used as the source for the file's resource fork. The resource fork, as well as those two code fragments in the data fork, allow a Mac OS ROM to act as a System Enabler. Understanding these better so that they can be built more intelligently is a priority. Many resources are localised: this helps to narrow down the list of resources used by low-level parts of the system.

--- End quote ---

We are proud of our members.


--- Quote from: Protools5LEGuy link=topic=3661.msg24245#msg24245 ---We are proud of our members.

--- End quote ---
Yes, indeed, for a long time I think we are seeing the beginning of a new era ... ;D

Awesome !

I will say thank you all for the support and encourage meant that has been given all around.

Quick history lesson.
This whole things started with iMic trying to get OS 9 on his G4 iBook (correct me if I'm wrong) and the attempts at getting the FW800 MDD's booting 9. 
This went a long way. 
I have the last model iBook which didn't work with all the fixes that were out there.
I got involved and pushed this further. The only reason I even started this was because I wanted to see the OS 9 desktop on my iBook G4 without some emulator or anything.
Didn't have to work, just wanted to see it. I did that a long time ago now. And me being stubborn just kept it going as now I want to see it actually work correctly. Later ELN joined in.  Got to give that kid credit.
Now we are on this roller coaster ride to do way more than I ever even imagined when I even started thinking about this. 

So again thank you (I'm sure ELN is very grateful as well)

And here is the first thread I started on this insanely crazy journey. (Who'd have thought we'd even get this far?),2727.msg16615.html

Awesome, way to go Nanopico and Elliott! :D

MacOS Plus:
  I'm currently heavily reorganizing my computer work area and taking better stock of all the loose parts I have.  The software development component of the Mac OS 9 project may be a bit over my head but hardware is my strong side.  When more updates to the OS are available to test I will be quite well ready for additional testing here on MDD FW800 and Xserve G4 platforms.  These machines are already running OS 9 now to the extent hardware support is working so far, but there's still plenty that can be tested.  If anyone would like particular expansion cards tested in these systems I would be happy to try if I have them.

  Great work, guys!


[0] Message Index

[#] Next page

Go to full version