An update: the new speakers I got, despite being plenty loud in general, simply cannot provide enough gain. Cranked all the way up, you practically have to sit right in front of them for output to be audible. I decided to change gears, and try something else. I thought I'd try a phono preamp, because the signal out of the headphone jack is about at that level. I grabbed a cheap one (Depusheng PP500) off Amazon that had some volume control knobs. I'm running a stereo 3.5mm to RCA splitter into the preamp, then taking the 6.35mm out into my desktop speakers. Finally, I have enough gain! However, there's a ton of noise, mostly because I have to figure out how to ground it to the Mac Mini.
At this point, I'm out of my depth, I think. Anyone know where I can ground to the Mini?