An easy solution is to simply remove that portion of the armrest. This is possible. Nothing needs to be cut, and the process is 100% reversible. You can use one of the "Oh Jesus" handles from the rear passenger compartment to act as something with which to close the door.

The electric mirror switch can be swedged in between the upper attachment point for the grab handle and the door panel. It sits in there quite securely. The whole thing ends up looking almost factory. If desired, the depression in the door panel (from the old armrest) can be mostly removed with a heat gun or hair drier. But the door panel vinyl also springs back over time, these pictures were taken right after I removed the armrest.