At any time after the entry thereof, the commission, in the manner provided for in the making thereof, may alter, amend, annul or otherwise modify any order, but no such modification shall in any way affect the validity of any act done or omitted by any utility on the faith of such order, or in pursuance of its terms, prior to notice of the modification thereof.