Vacancies in the board may be filled by such judge at any time, the commissioners appointed to hold [office] for the residue of the unexpired term. The removal of any commissioner from the county or counties in which lands of such district are situated shall render his office vacant.