All territory annexed to a city or town shall become a part of the district or districts of the city or town adjoining the territory so annexed.