Passengers, by Robert Silverberg (epub/Kindle/pdf)

For three years, people on Earth have been subject to the will of the "Passengers"—intangible beings who usurp human bodies temporarily and without warning, and do nothing but play and cause havoc. People being "ridden" are ignored by others, and when they are freed, the experience, by social convention, is ignored by all. When the Passenger leaves the host body, the person is left with no memories of his time being ridden.