Just wonderinf whether you have cable or ADSL, as with cable you'd get a hiccup when someone logs onto your shared line.
Apart from that i'd suspect hardware somewhere along the line, either a dodgey router along your path to the server or your computer being unable to keep up with the data transfers.