Well the thing is I do not know if anyone else can connect, I doubt it though being the times I tried it was empty.
And I'm not exactly sure what client I'm using either, I got it from these forums never really checked, and, it worked perfectly fine before, have no idea why it would cause a problem now.
Wait, I found the zip file, it's: backport-svn931-4-win32.zip
p.s. I'm on vista, have played trem for a long time with out any problems but this backport actually fixed some gamma ray issues I had before. I can also connect to other servers so I still think it's a server side problem.
Edit -
For future reference: The problem was a saved layout in the atcs map. Got rid of the layout now the server runs fine.