... I'm getting significant interference on my wifi network and the only thing left is the x10.
Does that mean you eliminated all the other 2.4 ghz devices.... like phones, baby monitors, intercoms, bluetooth devices, or other devices that may use bluetooth [2.4ghz] like cell phones, BlueRay players with wireless internet connections, and even a handful of digital picture frames.
Also
"significant interference" is a slight bit vague. There is also a bunch of problems a router can experience that can can slow or cause dropped connections.
I am NOT saying your problem isn't being caused by an X10 camera [or other audio/video device] but
more info is needed. Although.. to be honest I have seen X10 wireless cameras be interfered with by a wifi router....
I have never seen a camera mess with a internet connection.