If a partition fence is to be built on a line between towns, or partly in one town and partly in another, two supervisors, one from each town, shall be the fence viewers.