The order of suspension shall operate to relieve the officer from all the duties of the office until the matter is finally determined, and he must forthwith vacate the office and turn over all moneys, books, papers and property belonging thereto to the party appointed to serve until such suspension is removed.