Interesting questions…
Neither the 5200 or the 440 cards were Apple equipment. They've been re-flashed from Windows.
That said, here are some facts:
The 5200 is a 128Mb card @ 8x AGP as opposed to the 440 64Mb @ 4x AGP.
The 5200 has evidently been neutered by cutting traces to make it run at 4x AGP or it wouldn't be working in the MDD at all.
AFAIK, the only 3D acceleration extension for OS9 was an ATI for use with the Radeon 9000.
So, it would seem that although your 5200 should work pretty darn well in the MDD overall, there may be no way to optimize it for 3D.
So So, an original equipment ATI Radeon 9000, despite being "only" a 64Mb card, may work better/faster since there are OS9 3D drivers for it.
So so so, maybe someone else more into this area knows a hack or such that I don't, but this is where you're starting from. You may just have to trial-and-error your way to the best 3D you can get.