The appointing authority may at any time remove any commissioner for neglect of duty or malfeasance in office.