'Castaway' gamma-ray bursts come from distant early galaxies