A member of the board of adjustment shall be removed from office, upon notice and hearing, for neglect of duty or malfeasance in office, but for no other reason. The hearing shall be conducted jointly by the Commission and the supervisors of the district or districts.