Mac OS 9.2.2 Memory Limit of 1.5 GB... Some Answers

[nanopico]

OMG

Basic memory map for any one that cares. 
Assume 2 GB of ram installed.
Assume your system heap with your extensions and all that fun jazz takes 32 MB.
System zones and heaps set up at address 0 and use up to 32 MB
Process manager zone created at 1.5 GB and grows down to meet the system. (1.5 GB - 32 GB)
About this mac should now show show 32 MB for Mac OS and 1.5 GB - 32 MB for free space.
But it shows 1.5 GB - 32 MB as free space and then 32 MB + 512 MB for the Mac OS.
Alright back to the memory map (it's generalized here, but if you want more details I can post the full map of my system)

• 0 - 32 MB system
• 32 MB - 1.5 GB Process Manager
• 1.5 GB - 3.75 GB not accessible. Can not read or write to anything in that range
• 3.75 GB - 4 GB readable and writeable and if you do either your system will crash as this appears to be mappings for the hardware addresses and registers

Point is to fix this bug ( do consider this a bug ) The early boot that sets up the MMU in the cpu needs to map out the additional 512 MB so it's accessible and the process manager needs to be told to start at 2 GB and grow down. Since everything is relocatable with in the process manager (if it wasn't we'd all be stuck with like 32 mb of RAM) and that 512 MB of missing memory isn't currently mapped, this should cause almost no problems going forward.

So on to the two things that need to be found to fix this.

[nanopico]

Mild update.

Eln kindly pointed out to me in our most recent conversation there are two things mapped just over the 1.5GB mark. I had forgotten about this so it changes things, but only slightly.  So we are looking into how to move those and then see what happens with the creation of the process manager and possible having to update one value there.
So good news is....    This is becoming more of a reality

[ELN]

but a few guys scratchin at notepad on a forums site can do it when the real pros couldnt? i dunno guys...

We are disassembling and patching the Nanokernel.

https://github.com/elliotnunn/cdg5/blob/master/kernel-disasm-script.py

[Protools5LEGuy]

We are very pleased to have the Dream Team among us!

The troll has been kicked out.

99% Mac OS 9 Lives! users love every little update on these topics.

[nanopico]

We are very pleased to have the Dream Team among us!

The troll has been kicked out.

99% Mac OS 9 Lives! users love every little update on these topics.

Updates with progress towards the goal may slow for a bit.  We are rewriting some tools so that we can do the entire process in OS 9.
Yes we are crazy enough to do the whole thing using MPW.  So while we retool we might not see much in the way of a functional result.

And if any one would like to help out, I am looking to verify the memory map as we believe it to be.
Even though I am testing and working on this on 23 different Macs I am looking for anyone who want's to help as I have several that are same model and there are still others I don't have. I'd like to get details from a variety of machines with and with out 2GB installed.

So here are the instructions.

• Install Macsbug
        
  • Download macsbug
  • Put it in your system folder
  • Reboot
     
•       Press command + the power button (both at the same time).  This gets you into macsbug
     
•       type
  
        log memoryMap
     
•       Type the following
  
        dm 5FFFe000 FF0
  
        It will pause for a moment at that point type
  
        command + space
     
• once the last command finishes type
        
           dm 68000000 ffff0
        
        again when it pauses type
  
                         command + space
     
• type
  
        log
     
• type the letter g and your good to go

You can now uninstall macsbug by deleting it from the system folder and rebooting

You do need to hit enter after every command (I'm hoping that is obvious to everyone, but who knows)

You will now have a file called memoryMap on your desktop. If you would be so kind as to send that file to me, that would be awesome.

I may ask again for some of you to help in verifing the hardware pci range mapping, but that is a bit harder to put into easy commands as each machine has different hardware so the ranges are different and it's a lot of trial and error to get the range.
[/list]

[Protools5LEGuy]

Yes we are crazy enough to do the whole thing using MPW.  So while we retool we might not see much in the way of a functional result.

So here are the instructions.

   Install Macsbug
      
   

Is this version?

http://macos9lives.com/smforum/index.php/topic,3565.0.html

http://macintoshgarden.org/apps/macsbug-663

[nanopico]

Yes we are crazy enough to do the whole thing using MPW.  So while we retool we might not see much in the way of a functional result.

So here are the instructions.

   Install Macsbug
      
   

Is this version?

http://macos9lives.com/smforum/index.php/topic,3565.0.html

http://macintoshgarden.org/apps/macsbug-663

Yes that one.
Sorry I should have gave a link to that.
If any instructions are not clear to anyone, please let me know and I'll update them.
I don't want anyone to feel they can't help.

[LillyOmegaWolf]

Still need someone to trial it too? Or am I a bit late to the party?
I would love to get the full 2gb of ram working on my G4, if only because I enjoy tweaking and pushing limits   so Im happy to copy a new install onto a spare drive and test if you still need testers.

[InspectorG]

Still need someone to trial it too? Or am I a bit late to the party?
I would love to get the full 2gb of ram working on my G4, if only because I enjoy tweaking and pushing limits   so Im happy to copy a new install onto a spare drive and test if you still need testers.

Same here!

[nanopico]

Still need someone to trial it too? Or am I a bit late to the party?
I would love to get the full 2gb of ram working on my G4, if only because I enjoy tweaking and pushing limits   so Im happy to copy a new install onto a spare drive and test if you still need testers.

Same here!

There isn't anything to test at the moment.
The primary factor limiting the memory is thought to have been identified, but not confirmed as work has to be done to move some things in the OS.
As of yesterday I am not working on this for an unknown length of time.  ELN has picked up on this and is running with it along with another fine guy, I'm not sure if it's cool to identify him so right now I'm just going to call him awesome dude.
All is still moving forward, but don't expect to hear anything from me for god knows when.

[LillyOmegaWolf]

Well you have put a lot of work into this. So I can understand a break, especially if theres two awesome dudes who have taken this up.

[ELN]

All right, here goes. In the spirit of working in the open:

I think I have broken the 1.5 GB limit...

...by 64 MB.

So now it's the 1.5625 GB limit. But it's definitely a start. First, some explanation.

Before the Trampoline (an ELF-contained bootloader inside the Mac OS ROM file) hands control to the NanoKernel, it populates some data structures that describe many levels of the Mac OS environment. The NanoKernel pays the most attention to ConfigInfo, a structure that was formerly part of the RISC ROM but became a dynamic structure on NewWorld Macs. One of the jobs of ConfigInfo is to tell the NanoKernel how to lay out the address space of the blue task. ConfigInfo instructs the NanoKernel to make the main area, starting at address zero, as large as possible given the RAM available. If "Virtual Memory" is enabled then this area will be expanded later in the boot process, at the command of the 68k boot code. But ConfigInfo also instructs the NanoKernel to create these small special-purpose areas:

Code: [Select]

5fffe000-5fffefff InfoRecord page: contains pointers to structures shared by kernel and userspace
64000000-6417ffff Open Firmware device tree, and Mac II-style "Universal" structures
68fef000-68feffff ConfigInfo
68ffe000-68ffefff Kernel Data Page (kernel actually expands this to ~68ff5000-68ffefff)
68fff000-68ffffff Emulator Data Page

Other areas in the 6 segment are configured differently. The addresses listed might be familiar to anyone who's looked at the NanoKernel log before. But notice in the attached screenshot that the 5fffe000 area is missing, and a 6fffe000 area is present instead. That is because I have inserted code into the NanoKernel that reads the Trampoline's PageMap and edits it. For the curious, here is that code:

Code: [Select]

macro
PutPMDT &w1, &w2
lisori r31, &w1
stwx r31, r20, r21
addi r21, r21, 4
lisori r31, &w2
stwx r31, r20, r21
addi r21, r21, 4
endm


macro
StartSeg &s
stw r21, NKConfigurationInfo.SegMap32SupInit + 8*&s(r3)
stw r21, NKConfigurationInfo.SegMap32UsrInit + 8*&s(r3)
stw r21, NKConfigurationInfo.SegMap32CPUInit + 8*&s(r3)
stw r21, NKConfigurationInfo.SegMap32OvlInit + 8*&s(r3)
endm

macro
SpecialArea &a
stw r21, NKConfigurationInfo.PageMap&a.Offset(r3)
endm


if RedRover & !Vanilla

; INIT
lwz r20, NKConfigurationInfo.PageMapInitOffset(r3)
add r20, r20, r3
li r21, 0

lwz r29, NKConfigurationInfo.SegMaps + 8*6(r3)
add r29, r29, r20 ; r20 being the base of the pagemap
subi r29, r29, 8

@nt6 lwzu r28, 8(r29)
srwi r28, r28, 28
cmpwi r28, 4
bne @nt6
lwz r28, 0(r29)
lwz r29, 4(r29)
; Now r28/r29 are the 64000000 entry!
; (Important because its physical address and size vary)


; Vanilla page table (probably doesn't vary much between NW Macs)

; StartSeg 0x0
; PutPMDT 0x0000ffff, 0x00000a00
; PutPMDT 0x0000ffff, 0x00000a00
;
; StartSeg 0x1
; PutPMDT 0x0000ffff, 0x00000a00
; PutPMDT 0x0000ffff, 0x00000a00
;
; StartSeg 0x2
; PutPMDT 0x0000ffff, 0x00000a00
; PutPMDT 0x0000ffff, 0x00000a00
;
; StartSeg 0x3
; PutPMDT 0x0000ffff, 0x00000a00
; PutPMDT 0x0000ffff, 0x00000a00
;
; StartSeg 0x4
; PutPMDT 0x0000ffff, 0x00000a00
; PutPMDT 0x0000ffff, 0x00000a00
;
; StartSeg 0x5
; PutPMDT 0x0000fffd, 0x00000a00
; PutPMDT 0x0000fffd, 0x00000a00
; SpecialArea IRP
; PutPMDT 0xfffe0000, 0x00000012
; PutPMDT 0x0000ffff, 0x00000a00
; PutPMDT 0x0000ffff, 0x00000a00
;
; StartSeg 0x6
; stwx r28, r20, r21
; addi r21, r21, 4
; stwx r29, r20, r21
; addi r21, r21, 4
; PutPMDT 0x8fef0000, 0x00003013
; SpecialArea KDP
; PutPMDT 0x8ffe0000, 0x00000011
; SpecialArea EDP
; PutPMDT 0x8fff0000, 0x00000012
; PutPMDT 0x0000ffff, 0x00000a01
; PutPMDT 0x0000ffff, 0x00000a01
; PutPMDT 0x0000ffff, 0x00000a00
;
; StartSeg 0x7
; PutPMDT 0x0000ffff, 0x00000a01
; PutPMDT 0x0000ffff, 0x00000a01
; PutPMDT 0x0000ffff, 0x00000a00
;


; Don't forget to patch logical address fields:
lisori r31, 0x6fffe000
stw r31, NKConfigurationInfo.LA_InfoRecord(r3)


StartSeg 0x0
PutPMDT 0x0000ffff, 0x00000a00
PutPMDT 0x0000ffff, 0x00000a00

StartSeg 0x1
PutPMDT 0x0000ffff, 0x00000a00
PutPMDT 0x0000ffff, 0x00000a00

StartSeg 0x2
PutPMDT 0x0000ffff, 0x00000a00
PutPMDT 0x0000ffff, 0x00000a00

StartSeg 0x3
PutPMDT 0x0000ffff, 0x00000a00
PutPMDT 0x0000ffff, 0x00000a00

StartSeg 0x4
PutPMDT 0x0000ffff, 0x00000a00
PutPMDT 0x0000ffff, 0x00000a00

StartSeg 0x5
PutPMDT 0x0000ffff, 0x00000a00
PutPMDT 0x0000ffff, 0x00000a00

StartSeg 0x6
PutPMDT 0x00003fff, 0x00000a00
PutPMDT 0x00003fff, 0x00000a00

stwx r28, r20, r21 ; Name Registry
addi r21, r21, 4
stwx r29, r20, r21
addi r21, r21, 4

PutPMDT 0x8fef0000, 0x00003013 ; ConfigInfo

SpecialArea KDP
PutPMDT 0x8ffe0000, 0x00000011
SpecialArea EDP
PutPMDT 0x8fff0000, 0x00000012
SpecialArea IRP
PutPMDT 0xfffe0000, 0x00000012

PutPMDT 0x0000ffff, 0x00000a01
PutPMDT 0x0000ffff, 0x00000a01
PutPMDT 0x0000ffff, 0x00000a00

StartSeg 0x7
PutPMDT 0x0000ffff, 0x00000a01
PutPMDT 0x0000ffff, 0x00000a01
PutPMDT 0x0000ffff, 0x00000a00


; Later parts of the table
; (definitely vary between machines)

; StartSeg 0x8
; PutPMDT 0x0000ffff, 0x8000003a
; PutPMDT 0x0000ffff, 0x80000a01
; PutPMDT 0x0000ffff, 0x00000a00

; StartSeg 0x9
; PutPMDT 0x0000000f, 0x9000003a
; PutPMDT 0x0020001f, 0x9002003a
; PutPMDT 0x80007fff, 0x98000032
; PutPMDT 0x0000ffff, 0x90000a01
; PutPMDT 0x0000ffff, 0x00000a00

; StartSeg 0xa
; PutPMDT 0x0000ffff, 0xa0000a01
; PutPMDT 0x0000ffff, 0x00000a00

; StartSeg 0xb
; PutPMDT 0x0000ffff, 0xb0000a01
; PutPMDT 0x0000ffff, 0x00000a00

; StartSeg 0xc
; PutPMDT 0x0000ffff, 0xc0000a01
; PutPMDT 0x0000ffff, 0x00000a00

; StartSeg 0xd
; PutPMDT 0x0000ffff, 0xd0000a01
; PutPMDT 0x0000ffff, 0x00000a00

; StartSeg 0xe
; PutPMDT 0x0000ffff, 0xe0000a01
; PutPMDT 0x0000ffff, 0x00000a00

; StartSeg 0xf
; PutPMDT 0x0000ffff, 0xf000003a
; PutPMDT 0x0000ffff, 0xf0000a01
; PutPMDT 0x0000ffff, 0x00000a00

;stw r21, NKConfigurationInfo.PageMapInitSize(r3)
li r20, 0
li r21, 0
li r31, 0

endif

The effect of the patch is to move the InfoRecord page from the 5 segment to the equivalent location in the 6 segment, and to expand the blue area up to the bottom of the next obstacle, at 64000000. This is a difference of 64 MB.

On its own, this patch crashes my Power Mac (even though it only has 256 MB). This is because the 68k Emulator, which lives in ROM just a few KB north of the NanoKernel, contains a hardcoded constant, 5fffe000, which it uses to fetch the LogicalMemorySize from the NanoKernel's SystemInfo structure. I changed that, and all was well. (Hopefully there are no more subtle crashes waiting to strike. Well-written software should use the NKLocateInfoRecord kernel call instead of hardcoding that location.)

Interestingly, the patch alone did not crash QEMU. QEMU is currently my only way to start Mac OS 9 with the full 2 GB of RAM. (By the way, anyone want to send me 2 GB that will work in my FW800 MDD? I'm hopeless with hardware.) QEMU does not start the Dynamic Recompiling function of the emulator, which is the part that contains the hardcoded constant. I am not yet sure why.

So that's one down and

So, I need testers with 2 GB Macs running Mac OS 9.2.2. Attached is a Mac OS ROM file and a copy of NKLogReader. First, dump the log output of your normal kernel. Then, install the patched Mac OS ROM file. If you get a crash on boot, please take a picture of the kernel log. If it boots up, please get a screenshot of the Finder About window and a kernel dump.

Soon we might bust the 1.5 GB limit completely. Good luck, and have fun.

[Protools5LEGuy]

Are this 64Mb able to be used as a RAM disk?

[ELN]

The system should see it as normal, usable RAM.

[MacOS Plus]

  Not smooth sailing so far.  One of the DIMMs I borrowed from an Xserve to max out the RAM in my MDDs caused random hard freezes.  The usual moral of the story applies - if possible stick to a fully matched set of DIMMs!  Completing a matched set fixed that issue.

  I had a bit of trouble with saving the attached file.  Classilla doesn't use the correct file name for the archive, no big deal, but I don't think the ROM file came out right.  A fresh 9.2.2 boot drive wasn't recognized as bootable at startup with the replacement ROM.  Also I had to pull NKLogReader from your original post about it because it wasn't present in the current attachment.  I was able to run NKLogReader when booted from the original ROM and then save the log to a file.  The modified ROM will have to wait until this issue is sorted out.

  Once I can get 'step one' working, could you apply the "unsupported G4s" patch so I can try this on FW800 and Xserve?

[MacTron]

Great!!! 

All right, here goes. In the spirit of working in the open:

I think I have broken the 1.5 GB limit...

...by 64 MB.

... 64MB...
... this is a small step for a man... but a big step for ...
 

[ELN]

I am surprised to hear that! Sorry you've had trouble. I tried downloading the archive from the site. It worked fine for me, log dumper and all.

Here's a new archive. I rebuilt the image with some of the changes I've made in the last couple of hours, and compressed it with a different version of Stuffit. You should get an extra 128 MB recognised now.

My builds should actually boot fine on an "unsupported" G4. (But I reflashed my MDD back in 2015, so I haven't been testing that.)

Thanks for trying it out already! Let me know how this goes.

[ELN]

Thanks, MacTron! Could I trouble you for the contents of your kernel log?

[MacTron]

Thanks, MacTron! Could I trouble you for the contents of your kernel log?

No problem but I have to solve an issue with the "Second limit-breaking ROM" because it cant startup the MDD I was using for testing, and in this machine I only have installed one System folder  ... LOL

[MacTron]

Attached is a screenshot of the crash I obtain when booting with "Second limit-breaking ROM".