The state commissioner of public institutions shall have authority, under such rules and regulations as the commissioner may prescribe, to grant, on the recommendation of the superintendent, a discharge or parole to any inmate of any of the various facilities; but while such inmate is on parole, and until he is discharged according to law, he shall remain in the legal custody of the commissioner of public institutions and subject at any time to be returned to physical custody, if in the judgment of the commissioner the interests of such paroled inmate will best be served thereby. The written order of said commissioner, countersigned by the superintendent, shall be sufficient warrant for any officer or person named therein to arrest and return to the facility the youth so paroled; and it shall be the duty of any such officer or person to arrest and return such youth to the facility. All actual expenses incurred in returning such youth to the facility shall be paid out of the funds appropriated for the maintenance of the facility.