The compression is exactly defined by the PEF opcodes. I suppose PEF Viewer being a viewer means it doesn't have an option to edit? The only method I tried was to convert the disassembly of a PEF to an assembly file to be compiled by MPW or CodeWarrior PowerPC assembler into a new PEF.
I used Jasik's The Debugger for disassembly and used a script to convert that to assembly. I also used DumpPEF and a script to convert that to assembly. I don't remember how well they work. Jasik's the Debugger produces inconsistent results between the two formats it creates: .list and .asm. list includes hex data and offsets for each line so it might be more useful since you can check the offsets for all the labels to find the bugs.
Other options I haven't tried:
NationalSecurityAgency/ghidra
DBJ314/dePEF-and-disarm
The MPW DumpPEF command calls the compressed data "pidata"
The pidata opcodes are listed in IOGraphics-305.14/IONDRVSupport/IOPEFInternals.h
kPEFPkDataZero = 0, /* Zero fill "count" bytes. */
kPEFPkDataBlock = 1, /* Block copy "count" bytes. */
kPEFPkDataRepeat = 2, /* Repeat "count" bytes "count2"+1 times. */
kPEFPkDataRepeatBlock = 3, /* Interleaved repeated and unique data. */
kPEFPkDataRepeatZero = 4 /* Interleaved zero and unique data. */
To be Mac OS X compatible, an ndrv must import only from a specific set of libraries. CarbonLib is definitely not one of them. Does it say CarbonLib somewhere in the ndrv? Maybe you could post a link to the ndrv so we know what you're looking at.
See IOGraphics-305.14/IONDRVSupport/IONDRVLibraries.cpp for the list of libraries and functions from each library.
Drivers should only use: PCILib, VideoServicesLib, NameRegistryLib, and DriverServicesLib
Apple also allowed some functions from InterfaceLib and PrivateInterfaceLib to make some of its ndrv's compatible with OS X.
Actually, IOGraphics-305.14 is for 10.5.8. If you look at IOGraphics-179.2.1 (for 10.4.11), then you see there's also some functions from the AtiUtils library which is used by the Beige G3's GPU.
XPostFacto may have some kexts that define some other libraries and functions to allow some other ndrv's to be usable. I added some libraries and functions to make my modified sixty6 ndrv usable (for video out of Power Mac 8600).
I'm not sure it's possible to create a ndrv that is compatible with OS X but not Mac OS 9?
An ndrv is suppose to have two exports:
TheDriverDescription - data
DoDriverIO - code
https://developer.apple.com/library/archive/documentation/Hardware/DeviceManagers/pci_srvcs/pci_cards_drivers/Designing_PCI_Cards_Drivers.pdf