The Sumerians would seem to be a likely candidate for those who built the Tower of Babel.
The Sumerian Empire was superseded and ended by Sargon of Akkad (ie. The rising Akkadian empire) in 2320 BC