I've done a lot of testing with all the video cards I have, including flashing various ROMs. I have one of the very same GeForce cards as you do, which came to me in an MDD FW800, and have concluded that the issue is likely the video card's ROM itself that is purposely not properly compliant with OS 9. Given that those particular MDD machines were never intended to support booting OS 9, this would make sense.
nVidia appears to handle the device reporting to the system on these, and the previous 2 MX, by enabling the presence of "Port A" or "Port B" in the device tree in Open Firmware. If a monitor isn't connected then the device is absent as far as the motherboard is concerned. This is really annoying if you want or need to re-flash a ROM without a monitor connected and using a second card for video because the system can't see the device to flash it! When I flashed the wrong ROM once I was forced to put it in a PC and use a DOS-based flasher to correct it. Additionally, I've found that under OS 9 the DVI port will detect a monitor and native resolution properly but will never display video. I'm using an adapter on the ADC port instead. I'm also convinced that neither of these cards was ever intended to be used for dual monitors (under OS 9 at least) unless they came stock from the factory as "TwinView" variants. (Flashing a non-TwinView card with a TwinView ROM won't change that, and can have bad effects on basic function.)
I hadn't known about the DVD Player issue, but I can hazard a guess that the OEM GeForce4 MX is not being handled/accepted by nVidia's OS 9 drivers and therefore the DVD Acceleration component of the drivers is not engaging and reporting to the system. I can confirm on my own system that the error you received upon starting DVD Player happens here also. Perhaps the drivers can be hacked to accept this card's PCI device ID, but I don't know.
The ROM I have in my 4 MX card is version 2032. Does anyone know if there is another version available, newer or older, that would address the issue? This card, and my OEM 2 MX for that matter, have mostly sat on the shelf over the years because of the various annoyances they create on one or the other operating system, in particular negating the usefulness in a dual-boot machine. I've never been a big fan of ATI, but if memory serves me correctly I've had the least trouble overall with the Radeon 9000 under OS 9, with the Radeon 7000 PCI a close second. (I just recently picked up a GeForce4 Ti 4600 so it's too early for me to conclusively comment on that one.)