I don't think this is an AGC issue because the default setting works well for most people. In a low noise environment you can turn the high threshold off to be able to pick up very weak signals.
From your description, I suspect you are running into the command storm shutdown. The default setting allows a continuous burst of about 120 commands, or 20 commands per minute average. That is sufficient for most people (including myself). If you have a lot of motion detectors, you may be exceeding that amount of activity. Try the high command storm threshold (mode option 7 ON). That allows a continuous burst of about 400 commands, or 60 commands per minute average.
Whoops - missed that last line this morning. I only had a few minutes to respond before I had to head out. You can confirm the high command storm shutdown threshold is set by checking that the XTB-IIR transmits "P7 Status ON" in response to the mode command. Also, you could check whether it is actually going into the command storm shutdown by the LED doing a rapid flash until the excessive traffic clears. It will also send "P16 Status OFF" out the digital port when it shuts down.
Another possibility is that you are seeing collisions. If you needed to set the high command storm threshold, that is a LOT of X10 traffic. There is a strong likelyhood that you are seeing collisions, particularly where the source can be random transmissions from motion detectors. If indeed you are seeing collisions, the XTB-IIR would not repeat those commands, and you would get the triple LED flash that indicates a receiving error.
Jeff