Do not lose grip on reality. Though the light of an object takes a long time to reach its destination, the object that produced that light is long gone, therefore there is no past to go back to. So, you can arrive at where the light originated but what produced that light is no longer existent...