Ok, so it seems you want a guitar with a Floyd Rose and humbuckers, but other than that, the 2 guitars you mention are very different.
My advice would be to go Jackson, but not the RR3,(as a huge Randy fan, I feel very guilty saying this)they are very uncomfortable and only really good for live performance(you dont wanna play that thing when sitting down...it's worse than a Gibson v in terms of comfort. I would look for a Dinky or soloist Jackson, or something like that, with a more conventional shape. As for the pickups, Duncan Design means that although the pickups and their wiring systems were designed by Seymour Duncan(not personally, obviously) the pickups are built under license by Jackson or any other company, so I guess they are probably a bit better than the stock pickups on the Ibanez...unless the Ibanez has V7/V8 pickups...those are nice.
Ibanez's Prestige range is amazing, but hellishly expensive...Just as Jackson Usa is.

I cant be anymore specific I'm afraid, if you are set on getting one of those2, i'd go with the Rg, but purely on body shape.
If you look around for more conventional Jacksons,maybe they are the way(tho having said that, I own two Ibanez guitars,no Jacksons, and love both my Ibanez's)