It is probably caused by ping.
I'm not sure about this, but building might be client-sided. If it is, then here's a solution:
One way to solve this is to reject a build request from the second client that builds when the first one is already building.