Mac OS nanokernel

[Protools5LEGuy]

https://en.wikipedia.org/wiki/Mac_OS_nanokernel

The Mac OS nanokernel is an extremely simple operating system kernel serving as the basis of most PowerPC based system software versions 7 through 9, predating OS X. The initial revision of this software is an extremely simple, single tasking system which delegates most tasks to an emulator running the Motorola 68K version of the operating system. The second major revision supports multitasking, multiprocessing, and message passing, and would be more properly called a microkernel. Unlike the 68K-derived Mac OS kernel running within it, the PowerPC kernel exists in a protected memory space and executes device drivers in user mode. The nanokernel is completely different from the Copland OS microkernel, although they were created in succession with similar goals.

System 7.1.2—Mac OS 8.5.1

The original nanokernel, and the tightly integrated Mac 68k emulator, were written by emulation consultant Gary Davidian.[4] Its main purpose is to allow the existing Motorola 68K version of the operating system to run on new hardware. As such, the normal state of the system is to be running 68K code. The operating system does little until activated by an interrupt, which is quickly mapped to its 68K equivalent within the virtual machine.

Other tasks may include switching back to PowerPC mode, if necessary, upon completion of the interrupt handler, and mapping the Macintosh virtual memory system to the PowerPC hardware. However, as the software is little documented, these might instead be handled by the emulator running in user mode.

This nanokernel is stored on the Mac OS ROM chip integrated into Old World ROM computers, or inside the Mac OS ROM file on disk on the New World ROM computers, rather than being installed in the familiar sense.

Interim development

Progress after 1994 demanded additional functionality. A forward-looking architecture was introduced for PCI card drivers in anticipation of Copland OS, which supported memory protection. The Open Transport networking architecture introduced standardized PowerPC synchronization primitives. The DayStar Digital Genesis MP Macintosh clone requires kernel extensions to support multiprocessing. This evolution would later affect the overhaul to the nanokernel in Mac OS 8.6.

Mac OS 8.6 and later

Mac OS 8.6's nanokernel was rewritten by René A. Vega to add Multiprocessing Services 2.0 support. PowerMacInfo (distributed in the Multiprocessing SDK) is an application that displays statistics about the nanokernel's operation.

René A. Vega's explanation of basic Mac OS architecture

https://groups.google.com/forum/#!msg/comp.sys.mac.programmer.help/tO0iuTNETGc/oTwfHPfuqXoJ

In all Mac OS 9.x and earlier systems that ran with the first PowerPC machines, the system was built with a nanokernel. That
kernel and the original emulator was the brainchild of Gary Davidian, a superb engineer who developed a tiny piece of code to
create two protected contexts one for the 68K emulator and one for native (PowerPC) code, ensuring a seamless transition from 68K
based hardware to PowerPC. That 18 KB kernel handles address relocation, hooks for virtual memory, exception management, low level
I/O support, alignment handling and high performance instruction emulation.

Atop that sits the upper layers of the OS, I/O drivers, graphics, process manager, etc. There is no kernel layer above the
nanokernel, just managers and servers.

I rewrote the nanokernel in OS 8.6, adding multitasking, multiple address space support, fully protected memory, multiprocessing
support, a high performance scheduler, enhanced power management. That increased its size to 50K (80K with full debugging
support). The upper layers didn't exploit much of that support in 8.6, though the notable change was the blue task blocking when
idle thence reducing battery consumption by 40% on laptops. Through the subsequent releases up to 9.x, more of the system used
those modern facilities as did in-house and 3rd party applications.

> When designing a kernel, its authors can choose one of several design
> philosophies. If the kernel contains a lot of stuff in it, then it's
> called a monolithic kernel.

A good example is OS X. Mach, IOkit, BSD are pretty much integrated into a monolithic kernel.

> >Now here is my question:
> >
> >I understand the Mac OS X/Darwin has a nanokernel.

No. Mac OS X has a kernel by way of the integrated facilities.

> At first I was
> >under the impression that a nanokernel was a Unix specific thing,

Not really. Most unix systems have kernel functions integrated with the upper layers. A few unix system have a microkernel
underneath them. mklinux is one such system.

>
> >then after some very quick research on the web, I concluded that no,
> >prior Mac OS's make use of a nanokernel also.

The nanokenrel was used starting with system 7 on 6100s.

> > In particular, OS 9.1 makes use of a nanokernel.

Yes.

> Mac OS X uses the Mach kernel, developed at Carnegie-Mellon. It's a
> microkernel -- it implements some stuff, but leaves a lot of stuff to be
> implemented outside of the kernel.

Yes, but at this point OS X integrates many more functions into the kernel.

> Mac OS 9 uses a nanokernel. The nanokernel implements a very small
> number of things -- basically the lowest level of interaction with the
> processor and other components of the system is in the nanokernel,

The nanokernel implements a  fair amount: multitasking, multiprocessing support, multiple virtual address spaces, memory
protection, server based VM, server based I/O, address space management kernel calls. There are over a hundred kernel calls.

> >So, what I am interested in, is this: When you run Classic mode inside
> >of OS X, you are 'taking' in some way from a bootable 9.1 OS (I guess
> >usually) that sits on another partition on your hard drive. That
> >bootable 9.1 OS must also make use of a nanokernel. Does it make use
> >of it while you are using Classic mode? It would have to in some way,
> >correct?

What is done (and this is clever), is the kernel SPIs are used as an abstraction layer. The native nanokernel is replaced by a
shim which either directly implements kernel calls or uses OS X services to implement other kernel calls. It's a little bit more
complicated than this, but essentially this is what takes place.

>  I believe that the Mac OS 9 nanokernel is not involved in
> the Mac OS X Classic action.

Insofar as the native code is not used, but the SPI remains.

> By replacing interactions with the
> nanokernel with interactions with the Mac OS X kernel, the Classic
> environment becomes just another process running under Mac OS X.

Mostly so. There are other high level issues such as coordinating window drawing, unifying high level event management, drag and
drop clip board management. One important thing is under OS 9, Mac OS (that which most people interact with) is just another
process running as a task by the nanokernel.

> >How does Classic mode work? Is it, from the programming standpoint,
> >just an entire 9.1 OS running inside of OS X?

Mostly with fancy plumbing to bridge services between the two OSs, and a pseudo kernel supporting the nanokernel's SPI.

> (Meeroh says)

> Basically, this is my understanding:
>
> Mac OS 9 structure is
> Applications & Drivers -> System -> Nanokernel
>
> Mac OS X structure is
> Applications -> System -> Kernel & Drivers
>
> Mac OS X with Classic structure is
> Mac OS 9 Applications & Drivers -> Mac OS 9 System in Classic -> [abstraction layer] Mac OS
> X Kernel and Drivers
>

I added the abstraction layer. Also, to makes things run faster there are various virtual machine-like assists in the mach kernel
for classic.

Classic runs surprisingly well, up to 90% of native performance on X if you throw a lot of RAM into your machine to accomodate the
combined working sets of the two systems.

https://groups.google.com/forum/?hl=en#!topic/comp.sys.mac.programmer.help/3NK-jAdmOqg

[nanopico]

Glad to see this all put together in one spot.

[ELN]

Hi all,

I thought I should write up a brief discussion of the Mac OS Nanokernel. This is a very early draft. Once it has matured here a bit, I will set up a corresponding page on my cdg5 wiki (github.com/elliotnunn/cdg5/wiki).

First up: nanokernels aren't really a thing. A microkernel is an operating system kernel that delegates a great number of OS functions to userspace processes, which have no (or less) direct access to hardware and which follow the normal rules for process preemption, etc. The Nanokernel (I'll capitalise the name of the Mac OS one) does something like this. Hardware interrupt handling and a bunch of other stuff is farmed out to a single "blue" task, which contains the "System 7-style" Mac OS environment. Other tasks were spun off from within the blue task, and could be scheduled alongside the blue task or on a second processor.

The Nanokernel came in two quite different versions, and shipped in a few different ways. Version 1.x shipped with all beige Power Macs and in some early releases of the NewWorld ROM (I am not sure which ones). Version 2.x shipped inside later NewWorld ROM releases, and inside a "krnl" System file resource in 8.6 or later. On OldWorld machines, some bizarre kernel-replacement process takes place surprisingly late in the boot process. Worse still, from 8.6 to at least 9.0.4, kernel replacement seems to be optional.

[ELN]

I think I'll just keep posting to this thread, and collate all this info at a later date. Nanopico and I have been continuing our email exchange. While he works on the Trampoline and System file stuff, I'm hacking away at the Nanokernel.

After testing out IDA (what a mess) and trying to wrestle with the GNU Binutils, I have settled on rolling my own disassembler and static kinda-analyser. It is called Statin, and it uses Capstone as its disassembly backend and Networkx to maintain a directed graph of control flow. Check it out: https://github.com/elliotnunn/cdg5/blob/master/statin.py

At the moment Statin is hardly a general solution to anything, but I think that it has some potential. I am using it to hack on version 02.28 of the Mac OS Nanokernel, which as far as I know was the final release. I can only find it in Mac OS ROM version 10.2.1.

A quick update on what shipped when. Nanokernel v2 (the rewritten version that I am interested in) was released in mid-1999 as part of Mac OS 8.6. Its chief new feature was the ability to encapsulate the cooperative Mac OS environment inside a single preemptively scheduled "blue task". Other preemptive tasks could then be spun off from the blue task using version 2 of the Multiprocessing Services API, and could share the CPU with the blue task. Not bad for a point release.

On OldWorld systems, which already have Nanokernel v1 in ROM, the blue task loads Nanokernel v2 from a System file "kern" resource early in the boot process. The new Nanokernel gets executed and performs some shenanigans to squeeze out the old Nanokernel, then resumes the normal boot process. If this patch process fails, then boot can still continue with the old nanokernel. This is why Sheepshaver (which has only ever worked with Nanokernel v1) can boot 8.6 and 9.0 (and, rumour has it, 9.1). By 9.2, this was definitely no longer possible, and Nanokernel v2 was a hard requirement (but do remember that the only OldWorld machines on which 9.2 officially ran were beige G3s). This is why people talk about Sheepshaver needing to emulate an MMU to run the final releases of 9 -- because Nanokernel v2 absolutely needs hardware VM. In fact, the Sheepshaver devs had to do a considerable amount of hacking just to get v1 to run without an MMU.

On NewWorld systems, things are thankfully a bit simpler. Nanokernel v2 occupies the same place in the ROM-in-RAM image that Nanokernel v1 used to -- up to about 100K at offset 0x310000. Mac OS ROM files versioned 1.4 or earlier shipped with Mac OS 8.5.1 or earlier and contain the old Nanokernel. Files versioned 1.6 or later shipped with Mac OS 8.6 or later and in "Mac OS ROM Update 1.0", and contain the new Nanokernel. But Nanokernel v2 really became useful with the release of the first Power Mac G4s, which were dual-core and AltiVec-equipped, a few months after OS 8.6. Side note: the Mac OS ROM file in these machines, versioned 2.5.1, is a bit of an oddball. It introduced the "parcels-based" format used in subsequent releases and by the Classic environment, and it has quite a large number of extraneous PEFs appended outside its compressed region, making it quite large.

The binary code in a System kern resource is the same as what you would find in a NewWorld ROM image. You can swap them over (I tried it). Anyone want to give Nanokernel 02.28 a go in an OldWorld machine? I'm pretty sure you'd be the first.

[DieHard]

I read and re-read the last post 4 times !

If this shit doesn't absolutely fascinate you, then you are brain dead.  This is awesome stuff.  I doubt the Apple camp ever thought anyone would dissect the code or unveil/crack open the secrets they locked away in the Black Box.

You guys are amazing !   The level of tenaciousness is unsurpassed.  Please keep going !  The Mac OS 9 2017 modified nanokernel and System ROM 11.0 is set to be released before Feb 2017

[Philgood]

A milestone is born

[ELN]

Check out Apple's old MT-SMP mailing list. Right up to the end of the Classic era, Apple luminaries (including even the brilliant René Vega himself) were answering developers' questions about the Nanokernel and Multiprocessing Services API with each new OS release.

And some very good work was getting done on 9! Even 9.2, which was rushed to release just as the team were being moved on, brought some great incremental changes to the multiprocessing environment. Any Toolbox call could be made transparently from an MPTask, and farmed out to the blue task for execution only as necessary (say for a Memory Manager call). If any of us get around to coding up an H.264 QuickTime codec, this will help.

[MacOS Plus]

  The switch to nanokernel 2 at OS 8.6 may finally explain to me why some device support didn't survive past 8.5.1.  I'd have to go digging around in my hardware collection to remember the one thing in particular that pissed me off - it was either the Radius VideoVision or the Targa 2000 video capture systems.  Maybe the nanokernel had nothing to do with it but I'm still hopeful that eventually some of the accessory hardware that never made it to OS 9 can have support migrated, even some of the Nubus cards.

[ELN]

Well, if you can put your hardware in an OldWorld machine (and I would be very surprised if you couldn't) then you can test this hypothesis. Trash the System kern resource in OS 8.6 and see if it works then.

And just out of curiosity -- is this hardware compatible with "Virtual Memory" (such as it is)?

[MacOS Plus]

  I'll get back to you on this.  I have to dig around for the manuals and support disks.  After further thought I believe the Radius VideoVision topped out at 8.1 and it was the Targa 2000 that went only to 8.5.1.  These two devices were video capture cards with Quicktime integration.  They also provided a dedicated second monitor at television resolution through the Monitors control panel.

  Why I have these at all, and why I was interested in boosting their OS support, was that the SonicStudio audio editing software/hardware I have has the ability and direct support built-in to integrate with video capture and playback using these devices.  This allows sample-accurate sync (supposedly, I'm betting only frame-accurate) between captured video and the edited audio.  If it worked well it sure would beat having to run with external video tape and timecode, as I did in my professional work all the time.  This also ties into my previously mentioned Medianet hardware because the video capture can be run through the Medianet board's accelerated SCSI bus.  I only ever tried it once, and on a Quadra 840AV with the built-in capture hardware at that.  It barely worked because the machine was terribly under-powered for the job, but it did actually work.

  I'm going to go do some reading.  I think I know where my Targa manual is.

[MacTron]

This is a great and revealing info.

I read and re-read the last post 4 times !

I have lost my account. 

[ELN]

Hi all,

I just wanted to let you know that I have been making progress on the Nanokernel, and staying in close touch with nanopico. We now have a quite neat way to build several parts of a Mac OS ROM image from scratch. "Scratch" includes the output of my Capstone-based disassembler (ppcdisasm.py) and annotation script (nk2disasm). The many tools required for the build are here:

https://github.com/elliotnunn/toolboxtoolbox

And the parts of the ROM, including my latest disassembly, are here:

https://github.com/elliotnunn/cdg5

The included makefile also makes it quite easy to launch a qemu instance with your newly built ROM. It would be *super cool* and save me a lot of time if anyone could replace my very slow and naive LZSS compressor with a more clever one that works identically. Dependencies for the disassembly include Capstone, NetworkX and a copy of NK 02.28 (you can rip it with an older version of my tools, or just PM me). Deps for the rebuild are just a GNU toolchain and the stuff in the toolboxtoolbox repo.

Of course, let me know if you get it building!

[ELN]

Could this thread please be moved to the "Hacking the System" subforum?

[Knezzen]

Could this thread please be moved to the "Hacking the System" subforum?

Done!

[ELN]

I've posted on the Emaculation forum about the Nanokernel work. I've had a breakthrough in making the build system portable using Docker, so now more people can pitch in.

http://www.emaculation.com/forum/viewtopic.php?f=34&t=9098

[ELN]

Hi folks,

Anyone keen to help us out by testing the kernel build system? I could really use some data points...

Cheers!

[rwbaskette]

I may be able to help test if you have some procedures to verify the build process.