Section 2. Any town which has an infirmary may annually choose three, five, seven or more directors to have the management thereof, who may appoint a superintendent and assistants. If such directors are not chosen, the selectmen shall be the directors.