A defective switch module or the harness could be the cause; both are common suspects. You should examine the door panel wiring for corrosion or loose connections.
Double-check that harness and give those fuses and relays a once-over too. Sometimes, a sneaky bad fuse can act just like a harness problem and throw you off!