The governor or speakers may remove a member of the commission appointed by them for inefficiency, neglect of duty or misconduct in office.