Part of your aggravation (not the actual rez issue) is that you're using PC components on an old, Classic Mac system without being aware of all of the little differences that will trip you up.
When you tried to reset NVRAM, you got OSX, but you don't know why. Here's why:
The reset requires holding [Apple-Option-N-V] at startup. The operative part here is "Apple"
You don't have an "Apple" key on your keyboard - you have a PC "Windows" key. When you booted the system the very first time with that keyboard, it had to be identified to the system so it could be interpreted and used. The differences are stored in the keyboard preferences file that loads later during the boot - too late to do the reset. BUT, you might be able to do it in OSX…iffy, but worth a try. The results vary depending on the particular keyboard.
If you must use a PC keyboard, you should keep an actual Apple keyboard around to do this kind of stuff at least.
Meanwhile, if you can get the rez you want without problems in OSX, that means that rez is supported by the monitor but not available in the OS9 monitor driver. Why? Because the monitor(s) connected to the system are identified at boot (just like the keyboard) and your monitors, manufactured long after OS9 may not speaking a language that OS9 understands. OSX does, but not OS9. Also, the rez you're seeing that you like in OSX may be close, but not identical to the selection in OS9. What graphics card is in your computer? You might get better results with a different one.
Set the rez back to normal in OSX then reboot OSX and try a NVRAM reset. If that doesn't work, you may have to beg, borrow, steal or buy an actual Mac keyboard.
Lastly, a disclaimer: I am NOT the #1 monitor / display guru here by any means. Anyone better schooled on this subject is requested to jump in…please!
