An interesting proposal. But wouldn't we see things appearing out the other side and coming towards the center of the universe just as much as moving away from it? We would see things essentially moving back and forth.
Think of how Pac-man can off the screen and then appears on the other side, that is how I understand your theory. But imagine if Pacman was a 3D first person game. You would look down that hall and see the back of pacman's head's. (I believe the ghost spawn is actually blocking the view in the real game, but for the purpose of the analogy, just pretend it's a straight hall way.) This is because light reflecting off pacman's head would bounce back behind him until it would hit the wormhole. Then, like everything else, it would appear on the other side in front of pacman, allowing him to see the back of his own head.
Now imagine if the ghost spawn is still in the middle and 'Pinky' and 'Blinky' head left and 'Inky' and 'Clyde' head right, one after the other. Say Pinky hits the left wormhole and goes through, and Inky hits the right wormhole, going through. They each appear on the other side, now heading the opposite direction. You now have Blinky and Inky on the left and Pinky and Clyde on the right. All the ghosts are now heading the same direction as before, but they are now no longer all heading away from spawn, Pinky and Inky are heading back! If you were an astronomer at the ghost's spawn, you would see two ghosts heading towards you and two heading away. This isn't consistent with the data on the universe, as in our universe mostly everything is heading away from the center, not and even amount. Perhaps Pinky and Inky haven't reached the wormholes yet? Please let me know what you think.