The trustees shall hold office, one-third for one year, one-third for two years and one-third for three years from the first of July following their appointment, and at their first regular meeting shall cast lots for the respective terms; and annually thereafter the mayor or other proper official, before the first of July of each year, shall appoint three trustees, who shall hold office for three years. The mayor or other proper official, by and with the consent of the legislative branch of the city government, may remove any trustee for misconduct or neglect of duty.