Yep that XYZ switch sounds like it's corroded inside or not adjusted. From what I've read it's only held on by a few 10mm bolts/nuts, a 13mm for the shifter linkage, and of coarse the electrical connection. It appears most people drill the 3 rivets holding it together (putting it back together with small bolt/nuts), and then use brake clean on the contacts.
I was thinking my 04 D2 XYZ switch was probably dirty inside as when I bought it and pulled the codes it listed one for the XYZ switch, but upon futher inspection my battery cable connections were very dirty and caused the lights to flicker, and the XYZ switch to fault. I was lucky and after cleaning the battery terminals and battery cables the issue hasn't returned.
I also read that some people had issues just like your describing and they loosened the mounting bolts slightly and moved it one way or the other and the issue went away (small amount of play in the mounting holes). Just something else to try if you get a chance without having to remove it.