It's pretty cool to see you folks get a whole 128 MB above the original limit. Hopefully progress has been made since 2017 - I wanna see if the limit was successfully increased to 2048 MB. Any news?
Nothing really changed. The available memory can't go past 0x68060000 because that's where the 68k Emulator's code lives. And there's several other well-known regions at 0x68080000 (the 68k Emulator's dispatch table), 0x68FFE000 (the Kernel Data Page), and 0x68FFF000 (the Emulator Data Page). Quite a few random parts of the Toolbox ROM and System File reference these parts of the address space, so they can't really be moved.
So increasing the limit further would need modifications to the Process Manager, and maybe the Memory Manager too. I'm pretty sure there were attempts to investigate them, but they petered out quite a while ago.
Can't we move the 68k emulator to a much higher address point (e.g. close to the 2 GB ceiling), and use the current address as a pointer to it?
This way, we could just make sure Mac OS does not try to allocate on ONLY that pointer address (and end of RAM), and modify even the System and/or the Mac OS ROM file if needed not to try to allocate there, but to allow any RAM allocation past that point.
Or... something. I'm literally just mumbling thoughts here.
That's the sort of change which would require changing all the random code that directly touches the 68k Emulator. It isn't built to look for a pointer there, so it doesn't. And finding that code is a difficult, maybe impossible task. Even some extensions poke the 68k Emulator. Speed Doubler for certain, though there may be others.
To go past them, there needs to be support for "holes" in the address space, that can be part of an Application Heap, but can't be used as normal memory. The Memory Manager part of this is pretty straightforward. It doesn't hand out memory that has already been allocated, and it is predictable enough that the right sequence of NewPtr and DisposePtr calls will allocate memory blocks at exactly the right spots. It isn't even necessary to change the Memory Manager's code.
The tricky part about this is the Process Manager. The code to make Application Heaps needs to be modified quite a bit. In addition to blocking off the holes from the Application itself (done by calling the Memory Manager as described above), it also needs to be careful about where to put Application Heaps. While Heaps can safely contain holes, they cannot start or end directly on a hole. Trying to do that will instantly crash (the 68k Emulator's code and dispatch table are read-only. And the Emulator Data Page has tons of code pointers that are regularly used, among all the other important stuff it contains). And also, if the Process Manager grows or shrinks a Heap, it has to not mess any of this up.
The current Process Manager doesn't do any of those things. And nobody knows how to make it do them (aside from "Just reverse-engineer it, and modify the right spots").
Safely handling a pointer-sized hole in the address space is exactly as hard as handling a larger hole, and only slightly easier than handling multiple holes.