I've found Incantatrix (from Player's Guide to Faerun), Fochlucan Lyrist (from Complete Adventurer), Ultimate Magus (from Complete Mage), and Abjurant Champion (also Complete Mage) to be problematic. Reasons below.
Incantatrix - Ever wanted to see a pc mage tear through your bad guys like so much tissue paper? This is the guy who can do it. His ability to add metamagic after the fact practically guarantees him extended (and higher levels persistent) defenses. And his 10th level ability to lower metamagic costs by one is grotesque. It's an epic feat that this guy gains at 15th level. And if you allow pcs to take multiple prestige classes, this nightmare gets worse with the addition of archmage or argent savant (both fine on their own).
Fochlucan Lyrist - I think this has a misprint, they shouldn't have fighter attack bonus. If you turn that down to match a wizard the class isn't so bad.
Ultimate Magus - This one is under constant debate. It seems to be too powerful but I haven't tested it yet. I think when I allow it I might make some changes.
Abjurant Champion - Only dangerous when stacked with existing fighter/ arcane caster type classes. By itself it's rather harmless, but slightly unbalanced. Also note, the only spells its abjurant armor ability affect is shield and repelling shield (not sure what they were thinking with that).