Mac OS 9.2.2 Memory Limit of 1.5 GB... Some Answers

[IIO]

That's true. The OS9 Helper basically restore some drivers and resources from 9.1, that were deliberately removed from Mac Os 9.2 to disable the 9.2 boot into OldWorld macs.

shouldnt that be in our next update for the 922 universal install?

[OS923]

In 32-bit Windows there's a similar problem. The "lost" memory is called Invisible Memory, Hidden Memory or Unmanaged Memory. There are several programs for Windows which can use this lost memory as a disk cache or RAM disk. This is probably also possible in OS 9 (as a device driver and there's an open source example from Apple). Normal programs would not be able to access this memory. www.romexsoftware.com has a programming interface for this lost memory. All programs which use this programming interface may be running at the same time. That's how we would best organize this if we get it to work.

[nanopico]

In 32-bit Windows there's a similar problem. The "lost" memory is called Invisible Memory, Hidden Memory or Unmanaged Memory. There are several programs for Windows which can use this lost memory as a disk cache or RAM disk. This is probably also possible in OS 9 (as a device driver and there's an open source example from Apple). Normal programs would not be able to access this memory. www.romexsoftware.com has a programming interface for this lost memory. All programs which use this programming interface may be running at the same time. That's how we would best organize this if we get it to work.

From all the work we have done so far, these solutions won't work.
Kernel and Trampoline first have to be patched.  The trampoline has to tell the kernel that it can use 2 GB of RAM, but it has to lock the space used by the emulator.  The kernel will have to be modified to map the physical RAM to virtual RAM addresses.  There are system calls to ask to get into a mode where you can use the physical address, which is used by hardware drivers.  Virtual memory address still has to exist though so that the call can ask for the physical address of a specific virtual address.  There is no virtual address for the missing RAM.
It is possible to bypass the memory manager and use any memory you want as long as it has a virtual address.

One thing to keep in mind.  Apple shipped computers that officially support 2 GB of RAM that were released before OS X was available.
Now because X was just around the corner you could argue that they didn't put any effort into fixing it.  8.6 shipped with the G4 AGP.  That supported 2 GB.   OS X didn't become available until about 2 years later. So OS 9 came next.  Problem still not resolved.  It was probably a significant undertaking so they waited for OS X (stupid fucking idea if you ask me). Did people ever complain about being limited by this barrier?  I would think that Apple would have tried to resolve this so that the customer wouldn't get incredibly pissed off. I would think that they would at least make some level of effort to make the full RAM available. But it's Apple, they make a lot of completely bat shit crazy ideas (well I think they do).

Not sure where that was going.  Point being, it's possible to get it back.  How is the first question. Stability is the next.  We still don't know how much we would have to patch yet to make everything stable, even if using it as a RAM disk.  Kernel, Memory Manager, Process Manager and Trampoline are all going to need some level of patching to make this work before we can even tackle the stability issue.
An extension/driver will not solve this.  If you think it will, you're more than welcome to write one.

[OS923]

It wouldn't require patching all these difficult things like Kernel, Memory Manager, Process Manager and Trampoline. Therefore, it has most chance of success. If they solved it this way on Windows, then I presume that patches were too difficult to get it right under all circumstances. A RAM-disk or cache is then better than nothing.

[Daniel]

We would need to be able to patch the Process Manager and be able to mess with the data structures of the Memory Manager. The method I consider most likely to work is to use NanoKernel calls to map in the unused memory in one contiguous area somewhere. If we just want a ram disk, we are pretty much done here. If we want it to be usable memory, we have a much harder time. We could try to extend to Process Manager's heap out to the new memory, but there is a bunch of other stuff in between them that we can't move. We would have to patch the process manager so that it doesn't use the big hole in the middle of its heap. If we want a single application to use all the memory, as MacTron wants, we additionally have to stick a non-relocatable block in the application heap that goes around the memory hole. That way, the memory manager doesn't try to use the part of the application heap that isn't actually part of the application heap.

The good news is that the process manager is one of the last things to start up in the boot process and it is loaded from a resource. It is definitely possible to make an extension that does some NK calls and overrides a resource. That doesn't mean it will be easy, but it can be done.

[nanopico]

We would need to be able to patch the Process Manager and be able to mess with the data structures of the Memory Manager. The method I consider most likely to work is to use NanoKernel calls to map in the unused memory in one contiguous area somewhere. If we just want a ram disk, we are pretty much done here. If we want it to be usable memory, we have a much harder time. We could try to extend to Process Manager's heap out to the new memory, but there is a bunch of other stuff in between them that we can't move. We would have to patch the process manager so that it doesn't use the big hole in the middle of its heap. If we want a single application to use all the memory, as MacTron wants, we additionally have to stick a non-relocatable block in the application heap that goes around the memory hole. That way, the memory manager doesn't try to use the part of the application heap that isn't actually part of the application heap.

The good news is that the process manager is one of the last things to start up in the boot process and it is loaded from a resource. It is definitely possible to make an extension that does some NK calls and overrides a resource. That doesn't mean it will be easy, but it can be done.

Moving the Process Manager heap and informing of a non-relocatable block is one idea I had a while back.  Not sure if it was discussed here or not. The memory manager works so tightly with the Process Manager that there is likely a patch to that as well.  An extension will be the best bet as you are correct that the Process Manager is one of the last things started prior to the finder.  From everything I have read, the Memory Manager is what starts the Process Manager.  This is the reason I say the Memory Manager would need patching.  It would have to know to load the Process Manager heap higher and set the block it can't use.

[nightstalkerpoet]

Just jumping in here, trying to look through posts to see what resources everyone is using.

Most likely, this is being used, but if not: https://web.archive.org/web/20030801125655/http://developer.apple.com:80/documentation/macos8/mac8.html

The Memory Manager section seems to give a lot of information on altering low-level OS memory behavior.

This also looks potentially more easily searchable:
http://mirror.informatimago.com/next/developer.apple.com/

[nanopico]

Just jumping in here, trying to look through posts to see what resources everyone is using.

Most likely, this is being used, but if not: https://web.archive.org/web/20030801125655/http://developer.apple.com:80/documentation/macos8/mac8.html

The Memory Manager section seems to give a lot of information on altering low-level OS memory behavior.

This also looks potentially more easily searchable:
http://mirror.informatimago.com/next/developer.apple.com/

I can't speak for anyone else, but yes I have read most of that. All the documented memory management stuff isn't loaded or available until too late.  The issue for the limit is the 68K emulator.  It needs to be loaded way early and so it needs to be loaded into a different portion of memory before it can be called an fully initialized/started so that it can run from a different memory area.

[nightstalkerpoet]

I'm still working on trying to understand a lot of this - teaching myself programming specifically to help here.

Is there a way to replace the Apple 68k emulator with the one found in Speed Doubler, and would that help?

[Daniel]

Inserting the Speed Doubler emulator directly into the Mac OS ROM file would be really cool, but it probably won't solve this problem.
The programming knowledge to look for should be some combination of:
•68k Assembly
•PowerPC Assembly
•Familiarity with Macintosh Programmer's Workshop
•Various languages that compile to 68k or PPC assembly (mainly c, pascal, and c++)
•Forth and Open Firmware(not useful for solving the 1.5GB limit but good for other projects)

The major components that are relevent to cracking the 1.5GB limit are:
•the NanoKernel
•the 68k Emulator
•the modern (figment) Memory Manager (the only thing in this list we have the complete official source to)
•the Virtual Memory Manager
•the Process Manager

If you want a challenge, try looking through an early 7.x system file to figure out how Apple utilizes unused NuBus slot space to increase addressable memory on 24-bit systems (this is mentioned in Inside Macintosh:Memory). If that code is findable and usable, that could be 95% of the solution.

[nightstalkerpoet]

Thanks, I'll take a look at those!

From reading this page, it appears that the potential conflict could be in a REALLY stupid place - fonts.
https://68kmla.org/forums/index.php?/topic/38338-crazy-ram-in-9600/

I got there simply due to a curiosity of the PowerMac 9600 and Ram.

If 256mb EDO Ram was installed (https://harddiskdirect.com/228471-001-compaq-256mb-edo-ecc-60ns-168-pin-dimm-memory-module.html) in that system, you'd have access to 3gb of ram. I've read MDDs don't like any more than 2gb (though there's potential there for up to 4gb), but maybe the 3gb limit could be pushed in these models for RAM intensive operations that don't require as much speed.

[nanopico]

Thanks, I'll take a look at those!

From reading this page, it appears that the potential conflict could be in a REALLY stupid place - fonts.
https://68kmla.org/forums/index.php?/topic/38338-crazy-ram-in-9600/

I got there simply due to a curiosity of the PowerMac 9600 and Ram.

If 256mb EDO Ram was installed (https://harddiskdirect.com/228471-001-compaq-256mb-edo-ecc-60ns-168-pin-dimm-memory-module.html) in that system, you'd have access to 3gb of ram. I've read MDDs don't like any more than 2gb (though there's potential there for up to 4gb), but maybe the 3gb limit could be pushed in these models for RAM intensive operations that don't require as much speed.

Well the OS knows about all 2 GB, but it is only assigning 1.5 to the process manager and the rest is assumed to be system memory by the OS, mapped or unmapped.  Part of the view right now is that the process manager may not be able to handle two seqments of RAM. One option is to somehow give the process manager all the RAM, but mark the 68k emulator as not usable and leave it alone.   

256 EDO in the 9600 wouldn't work afaik.  It limits each dimm to 128.  Limit of the hardware memory controller (not the MMU in the CPU).
Same goes for the MDD.  The memory controller can only handle 2GB.  It's a hardware limitation on those machines.

[ELN]

The NK knows about all 2 GB but is instructed by the Trampoline via the ConfigInfo structure to allocate a maximum of Ox5FFFE000 to the primary address range (PAR). The upper layers are informed about the leftover physical memory via the SystemInfo structure, but can only access it indirectly by asking the kernel to map it into the logical address space.

The PAR is limited to that size because of some fixed-address data structures that the upper layers expect to find in the 5 and 6 segments. The kernel gets instructed to create these, too. The infoRecord page at 5FFFE000 can be moved with a few patches to the system, but the structures and code a bit higher up are too deeply rooted to move.

[darthnVader]

You know, many years ago I remember reading and article that technically the G4 could address 4 GB of Ram, but there was some catch to it, and I forget the exact terminology used, so bear with me.

It had something to do with using 4 GB only 2 GB could be made available due to some flaw in how the G4 was made, or how it worked. Something like the other 2 GB could only be addressed as "Shadow" memory, or "Backside" memory?

I don't think latter chips like the 7447/4478 were even made at the time the article was published.

So, maybe, hope beyond hope, later G4's could address unto 4 GB of Ram?

Something similar to PAE?

I mean, with OS 9, virtual memory is disabled with over 1 GB of ram, but OS X can use VM with over that. If we could get it to work, at least partition it as a RAM disk, and run our VM on that RAM disk?

[adespoton]

Like I mentioned, running a studio under OS9 in the box will definitely run out of RAM at one time or another. 

Yes, G4s are cheap, but having multiple noisy G4s with extra cables and monitors are NOT the answer for many users; it may look cool, but many musicians have very little space due to nagging wives and other family obligations and most of the time are crammed in a small corner of the den as the "DAW cave".

So, I really believe, any efforts of maximizing OS 9 to "squeeze" more out of it, are not in vain.

Besides, it's fun to do what the Pros would not do... you act like that had a choice, they were hired employees that signed non-disclosures and were specifically told what to code, and what not to code.  Even if it was changing only a relative small amount of code to raise the RAM ceiling, I doubt any programmer that was making good money would make that argument with management.  They did as they were told, not as they wanted to... so to imply that "the pros couldn't do it, so how can we", may not be an accurate assessment of the facts.

Indeed.  Many of the fixes for OS 9 are actually fairly well understood in certain circles.  The issue here was that Jobs was burying OS 9 to drive sales of OS X and simplify the engineering chain.  So most of the people with the access and expertise to fix the problems... were moved elsewhere.  Must be frustrating for some of them, to know what the bugs were, what the proposed fixes were in the pipeline, and not only not be able to fix them, but not even be able to talk about what the issues/fixes were.