Is your server on your local box? If so, your internet connection might just suck, so after the first few players the rest get stuck at the gamestate or connecting screen because of severe one- or two-way latency. This has happened to me when I start a server on my college campus because their firewall simply blows chunks and murders my outbound ping.