No license issued as aforesaid shall be valid to protect the holder thereof in a building or place other than that designated in the license, unless consent to removal is granted by the mayor and aldermen or selectmen.