I am having difficulty with that transition as well. The move from D major to E minor is fine, but when I move in the other direction from E minor to D major, I can't seem to put my three fingers on the frets simultaneously. I keep getting behind at that point.
