Ok, another proof that it is packet loss :
On the server browser, if I spam the refresh list button, quite often the information for those servers are missing, as if the request for info ( and ping ) didn't get through. If I get that problem before even joining the game, it's not a misconfiguraiton on my side.
Unfortunately, the most probable cause is a router problem between our networks and the network hosting the server so there isn't much we can do.