When you say mounted does this mean glued or taped to the outside of the casing?
Also, would making the antennae on the transceiver 18" long help?
A few years back, I did extensive testing using an RF receiver that output the received signal strength. I tested with motion sensors, Stick-a-Switch and security door/window sensors and security motion sensors and with varying orientations of the passive radiator.
I found no meaningful difference with orientation either relative to the sensor or horizontal/vertical. All of the hair-splitting, voodoo, magical recommendations otherwise are pure speculation. I suspect it merely needs to be within the
near-field of the internal antenna. So, mount it anyway that makes sense appearance-wise. Early on, shortly after I first published the web article, a user wrote that he hid it behind dry wall with no ill effect on range.
For the receiver, try 9" and 18".
If there is an FCC ID, it may enable us to find a schematic of the receiver and that might lead to better suggestions for improving/replacing the antenna and/or RF receiver module.