Which direction did the Big Bang come from?

If the universe is expanding, wouldn't matter from behind and in front of the Milky Way, be moving differently from the matter on all other sides in accordance to the direction of the Big Bang?