I don't profess to be an expert with Codecs, but I do have two wireless cameras that record onto my PC when motion is detected. The quality of the picture is acceptable. And it is quite reliable. Interference can cause "lines" and even distort the picture. There are many threads that discuss line noise and RF interference as well as how to correct it. Suggest you do some search/research on these forum topics if you haven't already done so.
I'm assuming you have a wireless camera. You didn't say what kind of camera you have. I have the XX11A. So my experience is limited to that camera.
I have 640 x 480 selected for Capture Resolution under General Video. Under Video Settings I have Quality selected at 100% and Microsoft Video 1 selected for Codec. With frame rate at 6fps. The quality of your recording will improve if you increase the fps to higher numbers. However, this will decrease the amount of recording time. Example I'm recording for about 5 1/2 minutes with the 6fps. If I were to increase that to 30fps, my recording time would be about 1 minute.
My personal opinion is that the quality of the picture will not change by changing the Codec. You can do some minor adjusting with the lenses. You can select different channels (A,B,C,D) on the camera and video receiver to improve picture quality and reduce interference. You can adjust the antenna on both the receiver and the camera. Experimenting with different locations may improve picture quality too.
You may have already tried these things, but don't expect HD type of images. These are low cost cameras that are susceptible to lots of interference. Hope this helps some.