Client connection bundling, inter-server connections?

[Risujin]

SST is hurting because they have to pay a great deal of money to host a large server for a large game (40+ players or something). My question is this, could we modify the client/server model slightly in order to allow one large game to be hosted on multiple, smaller servers?

The idea could work like this:
1) 16 clients connect to local server A at 50 ms ping,
2) 16 clients connect to local server B at 50 ms ping,
3) server A and B sync changes to each other at the cost of one client connection (!), possibly with a slave/master setup where the master server controls the game and collisions etc and the slave only forwards compacted data
4) players play a 32 client game at 100 ms ping

I just thought of this off the top of my head but could bandwidth saving schemes like this be plausible? Taking the idea one step further, could we build an information flow tree through the client machines that would require any one client to hold only two or three connections?

[Thorn]

If the servers were sitting on the same LAN. Probably OR you could just go with the crowd and say happy ending to that large server.

[Risujin]

If the servers were sitting on the same LAN. Probably OR you could just go with the crowd and say happy ending to that large server.

a) This has nothing to do with LAN games.
b) This is not the bash SST thread, don't post that here.
c) I already went with the crowd in the SST thread.

[IJsje]

bandwidth saving schemes

The instead of 32 client server connections, use 2x16 client-server connection and one server-server connection which sends the info for 2x16 client-server connections, bandwidth saving scheme?

Nice math you got there.

[vcxzet]

assume
X players are on A server
Y players are on B server
A server will pass the player/entity data to B server
B server will pass the player/entity data to A server
so load on
server A will be X+1 clients
server B will be Y+1 clients

I think this will just improve game speed not required slots

[kevlarman]

it will also cause some nasty latency (at least 1 server frame worth (50ms), if they were on a lan or something, well over 100 if they aren't)with relation to players on the other server.

[Risujin]

bandwidth saving schemes

The instead of 32 client server connections, use 2x16 client-server connection and one server-server connection which sends the info for 2x16 client-server connections, bandwidth saving scheme?

Nice math you got there.

Good lord, this isnt a save-the-world reduce-carbon-emissions by cutting bandwidth plan. Its a plan for using multiple LOW CAPACITY servers to host one large game. There are more connections, but they are spread out so server load (per server connections) is low enough to allow say three DSL hosts to host a 64 player game (possibly...).

And yes, it will have the one server frame type of lag but for many HPBs, thats nothing new.

[Caveman]

For this you need to rewrite the net-code of the Engine.
Go ahead. If you find someone willing to do that.

But why would one do that?
You will not save any bindwidth with it, as it is irrelevant to where you send the data as long as the size of it gets not smaller :)
It does not matter if you send the data for 16 clients to a single recipient or to 16 different ones. The amount is still the same.

This server-sharing would only make sense to take advantage of the additional computing-power you get with chaining several machines.

Remember... the number of connections says nothing about the actual traffic you send over a single one.

To get to the point where you could actually save some traffic you'd need to compress the data, that in turn would mean getting a fast CPU and (maybe) more ram.
For SST this is no solution as that would mean upgrading and that would then finally cost more than the $2.000/year they are at now.

Dont take it wrong, but you've got to face it that the hosting cost is way to high. Only businesses with the right deduction-setup and money-making plan throw out that kind of money.

Has anyone at SST been looking into the possibility of changing the hoster to a much cheaper one? imho that's the only way to go. (scraping $50/month is way easier than coming up with $180)

PS: if you are thinking of joining several different DSL-lines, you'd better forget it right away. Just think man.
You'd need more than 2 DSL-Lines to get the _upstream_ you need and that means you'd also need a controlling server that sits somewhere somehow in the middle and does nothing more than keeping track of the packets and packet-loss and that is totally out of the question. Because this server would be needing the _combined_ upstream capability of all DSL-Lines used together.

PPS: I like your engagement for SST. But please do SST the favor and think yor plans more thorougly thru. Such half thought plans only damage SST further.

[Undeference]

There are ways to do (fairly) efficient distributed serving. With a game that uses UDP, it might cause some problems.

Not to mention the issue that when you start daisy chaining servers, you are encouraging Tremulous to become MMOFPS, which may be significantly more than 16 players (and would require rethinking the goals of the game -- can you have a match based MMO game?).

[Risujin]

Not to mention the issue that when you start daisy chaining servers, you are encouraging Tremulous to become MMOFPS, which may be significantly more than 16 players (and would require rethinking the goals of the game -- can you have a match based MMO game?).

Absolutely right. Tremulous is best with 6-8 player teams anyway which you can host on cable/DSL.

There are ways to do (fairly) efficient distributed serving. With a game that uses UDP, it might cause some problems.

A packet lost to the head of a cluster is lost to everyone chaining off of that cluster. UDP is more reliable than it sounds though. Packets are not *guaranteed* to arrive but overwhelmingly do.

For this you need to rewrite the net-code of the Engine.
Go ahead. If you find someone willing to do that.

Are you familiar with my Balance Mod? I'm not afraid to turn stupid ideas into even stupider code.

But why would one do that?
You will not save any bindwidth with it, as it is irrelevant to where you send the data as long as the size of it gets not smaller
It does not matter if you send the data for 16 clients to a single recipient or to 16 different ones. The amount is still the same.

Not quite true. When you share information with a client you are sending to them the world state, i.e. all (well...many at least) of the game entities and all facets of their current state.

The volume of data shared between two daisy chained servers would not be much greater than that between a server and client. The bottleneck here is sending out the data to all of the clients.

For instance, many people (myself included) host cable servers. These have enourmous downstream capacities (nearly 1 MB/s!) but their upstream capacity is severely handicapped in comparison (48 KB/s for me). Even daisy chaining clients to propagate received information among each other would allow a much much larger number of clients to play.

The drawback is of course the overhead of passing this information along. For most games however this overhead will not amount to more than one frame's worth of lag (50 ms).

This server-sharing would only make sense to take advantage of the additional computing-power you get with chaining several machines.

Computational costs are not a bottleneck.

To get to the point where you could actually save some traffic you'd need to compress the data, that in turn would mean getting a fast CPU and (maybe) more ram.

Thats actually a good question. Does Tremulous compress network data? Is (can?) this done at the hardware level?

For SST this is no solution as that would mean upgrading and that would then finally cost more than the $2.000/year they are at now.

For every cable player (with the stats given above) we can daisy chain 6 other people comfortably. SST's 54 player server, if filled with cable players with others chaining to them, would hold 378 clients with 100 ms ping. These are numbers viewed through rose-tinted glasses but there is potential here.

Dont take it wrong, but you've got to face it that the hosting cost is way to high. Only businesses with the right deduction-setup and money-making plan throw out that kind of money.

We can distribute media content at much higher bandwidth than games require over peer to peer systems already. Its about time we took steps to decentralize the server burden in games.

PS: if you are thinking of joining several different DSL-lines, you'd better forget it right away. Just think man.
You'd need more than 2 DSL-Lines to get the _upstream_ you need and that means you'd also need a controlling server that sits somewhere somehow in the middle and does nothing more than keeping track of the packets and packet-loss and that is totally out of the question. Because this server would be needing the _combined_ upstream capability of all DSL-Lines used together.

One of these DSL servers would be the controller. The communication channel between them would use a little over a single player's slot worth of data volume.

PPS: I like your engagement for SST. But please do SST the favor and think yor plans more thorougly thru. Such half thought plans only damage SST further.

a) this isnt a plan, but an idea
b) my intention is not to save SST
c) SST is not in any way affected by ideas kicked around in this thread

[kevlarman]

For SST this is no solution as that would mean upgrading and that would then finally cost more than the $2.000/year they are at now.

For every cable player (with the stats given above) we can daisy chain 6 other people comfortably. SST's 54 player server, if filled with cable players with others chaining to them, would hold 378 clients with 100 ms ping. These are numbers viewed through rose-tinted glasses but there is potential here.

um... are you seriously suggesting this? let me point out what i said before...

it will also cause some nasty latency (at least 1 server frame worth (50ms), if they were on a lan or something, well over 100 if they aren't)with relation to players on the other server.

if you start chaining more than 1 server, then the latency will make the game almost unplayable, if you chain the numbers you are talking about, then you are talking about latency on the order of seconds.

[vcxzet]

just do it and come back here and tell us how you wasted your one month
 

[techhead]

Maybe change the game sizes and make it a 40 player game.
Or have two 30 player SST servers.
Either way, you would probably save a ton of bandwidth associated with a 50 player game.

[WolfWings ShadowFlight]

The other deal-breaker for this idea is fairly straight-forward actually.

The number of connections and bandwidth used is not what causes server load. Processing a megabit worth of connection traffic every server frame, way more than even a 64-player server would generate, isn't difficult even on a very low-spec machine.

What causes server load is the number of unique entities and players moving around the server. If you punted half the clients to each of two servers, then tried to have the servers cross-communicate, you would still have to have each server calculate all the physics and movement for all the entities to avoid conflicts. For example, what if server A handles player X shooting a shotgun at player Y, but server B has player Z moving between the other two. Server A still needs to process player Z's movement to be able to have player Z eat the shotgun blast instead of player Y.

And that's a very straight-forward situation. Things only get more complicated exponentially as more players are active in a firefight.

[Risujin]

For SST this is no solution as that would mean upgrading and that would then finally cost more than the $2.000/year they are at now.

For every cable player (with the stats given above) we can daisy chain 6 other people comfortably. SST's 54 player server, if filled with cable players with others chaining to them, would hold 378 clients with 100 ms ping. These are numbers viewed through rose-tinted glasses but there is potential here.

um... are you seriously suggesting this? let me point out what i said before...

it will also cause some nasty latency (at least 1 server frame worth (50ms), if they were on a lan or something, well over 100 if they aren't)with relation to players on the other server.

if you start chaining more than 1 server, then the latency will make the game almost unplayable, if you chain the numbers you are talking about, then you are talking about latency on the order of seconds.

Wait a sec... its not 378 clients daisy chained in a line.

There are 54 cable clients connected to SST directly which feed back their received entity state information etc to 6 clients that leech off of them. The server does not communicate with the chained clients directly. A chained client is thus one hop away at 50 ms overhead lag.

What causes server load is the number of unique entities and players moving around the server. If you punted half the clients to each of two servers, then tried to have the servers cross-communicate, you would still have to have each server calculate all the physics and movement for all the entities to avoid conflicts.

Well I never really heard of a better solution to this kind of thing (and this applies to unlagged too somewhat) than to calculate all collisions etc at one point. So only one of the two servers would handle collisions etc and the other would only feed information in.

The argument for it is that information coming from the players is lesser in volume than updated world state info coming back to each player. If player A of 100 makes one change to the world, 99 other players need to hear about it whereas the server only gets player A's message once. If we chain 10 clients to one then we form a kind of information flow tree where the server sends out the received information to 10 clients which then do the same thing, sending out the same information to their 10 clients and no single machine is required to have more capacity than that.

[WolfWings ShadowFlight]

What causes server load is the number of unique entities and players moving around the server. If you punted half the clients to each of two servers, then tried to have the servers cross-communicate, you would still have to have each server calculate all the physics and movement for all the entities to avoid conflicts.

Well I never really heard of a better solution to this kind of thing (and this applies to unlagged too somewhat) than to calculate all collisions etc at one point. So only one of the two servers would handle collisions etc and the other would only feed information in.

The argument for it is that information coming from the players is lesser in volume than updated world state info coming back to each player. If player A of 100 makes one change to the world, 99 other players need to hear about it whereas the server only gets player A's message once. If we chain 10 clients to one then we form a kind of information flow tree where the server sends out the received information to 10 clients which then do the same thing, sending out the same information to their 10 clients and no single machine is required to have more capacity than that.

The problem is, the 'heavy lifting' that slows a server down is not the transmission and reception of data. It's the calculations of all the physics and collissions that cause high CPU load. If you're trying to reduce the bandwidth used, you're going to have marginal savings at best at the cost of a dramatic increase in latency primarilly. Seperately, any sub-server can be proven to have to have the full capacity of a single-server configuration no matter how you partition things because eventually all the players could be in the same area, funlilying the savings.

[Risujin]

The problem is, the 'heavy lifting' that slows a server down is not the transmission and reception of data. It's the calculations of all the physics and collissions that cause high CPU load.

CPU load isn't a bottleneck, servers are configured by default to run at a puny 20 frames per second. Bandwidth is the bottleneck. My home cable server is limited by upstream bandwidth, at 48 kb/s and typical 7.5 kb/s rate per player I can host a whopping 6 players.

If you're trying to reduce the bandwidth used, you're going to have marginal savings at best at the cost of a dramatic increase in latency primarilly. Seperately, any sub-server can be proven to have to have the full capacity of a single-server configuration no matter how you partition things because eventually all the players could be in the same area, funlilying the savings.

Not true, especially with the client-pyramid scheme.

If all players are in the same room making changes to the world, all players are receiving snapshots of the whole world. Each server sends the world state to each of its clients (half the players) and to the other server. If the bandwidth cost to sync two servers is smaller than the cost to sync half of the players we have increased overall bandwidth usage but saved bandwidth for each of the servers (as is our goal).

The basic idea here is that many people have broadband internet now and it has become commoditized to an extent. High power servers arent cheap comparatively. Many households would have broadband internet even if no one played games. Its much easier to get a bunch of these lower-bandwidth servers and string em together to match the bandwidth of a high-bandwidth server, and can be done at no extra cost to anyone involved.

With the client chain scheme we do this transparently. Every broadband player connects to the main server and handles part of the information distribution back to its child clients.

[Paradox]

Thorn, while you are entitled to have your own beliefs, and even talk about others in that minority, flaming every single post about sst is not good. I cant do anything about it if you want to be a troll, but i am just saying that instead of being a dumbass, just dont post flamebait.

[vcxzet]

Thorn, while you are entitled to have your own beliefs, and even talk about others in that minority, flaming every single post about sst is not good. I cant do anything about it if you want to be a troll, but i am just saying that instead of being a dumbass, just dont post flamebait.

I wonder which word highlightened me

[pyrax]

Hardware issues aside, any "parent" user could modify the code sent to its children to give themselves an advantage (thus bypassing the security of the host system).  I suppose this is a minor point given that the game isn't bulletproof as is.

[techhead]

The sending and receiving of the data to the 40-50 clients does in fact cause lag.
Actually playing in the game just makes a lot more.

[Risujin]

Hardware issues aside, any "parent" user could modify the code sent to its children to give themselves an advantage (thus bypassing the security of the host system).  I suppose this is a minor point given that the game isn't bulletproof as is.

In the client-chain system, yeah, you can pass along altered entity states to your children. Also you are getting information you may not be entitled to receive but only to pass along. This would allow you to affect other clients directly unlike now.

[KnightBear]

hmmm... its an interesting theory.  

as far as "modding" the parent that an advantage may be gained(if i read correctly) in an open sourced game, why wouldnt someone just take less time and make an aimbot?  there are simpler and easier ways to cheat.  i think that someone willing to go to such lengths deserves the right, lol.

eh, but im messed up i guess.  i havent the energy, nor lack of honor to cheat myself, so the psychology is foriegn to me.

[Undeference]

Daisy chaining servers can be useful in a cluster configuration, or even making it threaded so it takes full advantage of available resources. But daisy chaining isn't scalable beyond the local network (adding 10ms overhead is nothing, but adding an extra 50-100ms for each extra hop is insane).

I think the future of online gaming is in p2p. It would require a highly complex, highly efficient, and dynamically reconfigurable system. Each client is a node in the network making up the abstract "server". Processing and bandwidth distribution would be handled in a distributed and redundant manner.
The major goal of such a system would be to take advantage of any given configuration (e.g., there's a quad CPU system with 64GB RAM on a 56k line -- let's let it do CPU and memory intensive tasks that don't have to be done within the next few frames) and efficiently send the results to everyone who needs it.
Two necessities for such a system to exist is modular handling of events (so a dozen different systems can handle a particularly complex problem) and highly efficient bandwidth utilization (the client needs to have only what it needs, but when it needs that, it needs it instantly).

When you think of a system like that, it seems completely preposterous that you would have information when you need it. But in reality, the Internet is designed very much like that. There are systems in place to prevent node A from entirely losing communication with node B, and enabling data to be sent ASAP.


Anyway, there are problems with both network and CPU. The network usage is not entirely inefficient, but could be somewhat better. The CPU usage is, for the most part, acceptable, but does not scale well. (Hint, hint: if you want to start fixing things that aren't really broken, work on these instead. If you just try to jump on top of a much larger problem without having any idea how to go about it, you'll give up long before any useful code has been written.)

[[Kcorp]Noobius]

please don't flame me (too much) if i'm wrong, but isn't there this thing called openmosix that can help you share cpu load over more servers. or was the problem bandwidth and not processing power?


anyway, postcount++

[Undeference]

OpenMosix is for clustering and it runs at kernel level. Clustering would not work well because tremded only uses a single thread and there are often times when one thread has to wait for another (having to wait 50ms for something that would normally take 1ms might be painful). Furthermore, even if tremded were a threaded program, OpenMosix distributes processes, not threads. So if tremded were threaded, it could run on multiple CPUs on a single system, but not on multiple nodes.

(Potentially tremded could listen on the different NICs on the different systems, using the cluster just as a network of proxies. But that would be silly.)

The problem is both bandwidth and processing. tremded should be able to run perfectly fine on a minimal system (33MHz 486) with a reasonably low latency, low throughput network

[Risujin]

Daisy chaining servers

Oops, I think I completely misunderstood what you meant by daisy chaining before. I just realized you meant like an old coaxial network literally chained in a loop. As you pointed out many times, this is a bad idea. This isnt really what I had in mind at all!

I think the future of online gaming is in p2p.

Jackpot. Thats what I was getting at. There is a lot of talk about this around but Ive yet to see a real p2p game. Perhaps FPS isnt the best suited for this either.

It would seem that the ideal p2p game is a slow-paced game. Using a massive tree of players globbed together, sharing the processing and bandwidth burdens, you would be distributing the data slowly, so probably a turn based game would work well (turn ends when all players catch up for example).

On the other hand, I argue above that you can have a single layer of this and be in touch with the server only one frame behind.
When you think of a system like that, it seems completely preposterous that you would have information when you need it. But in reality, the Internet is designed very much like that. There are systems in place to prevent node A from entirely losing communication with node B, and enabling data to be sent ASAP.

(Hint, hint: if you want to start fixing things that aren't really broken, work on these instead. If you just try to jump on top of a much larger problem without having any idea how to go about it, you'll give up long before any useful code has been written.)

I just wanted to bounce the idea around and see what people thought.

[Undeference]

Daisy chaining servers

Oops, I think I completely misunderstood what you meant by daisy chaining before. I just realized you meant like an old coaxial network literally chained in a loop. As you pointed out many times, this is a bad idea. This isnt really what I had in mind at all!

Not quite. Obviously with few enough servers, they would all be constantly in communication with each other (not A to B to C, but A to B and A to C). This is not very scalable and reduces the effectiveness of any sort of compression you may consider using.

As you stated before, the actual likelihood of receiving a packet is much greater than that of losing one. However, packets are lost. Packets are lost frequently. Each hop between your computer and the server increases the likelihood of a lost packet. If the servers then relay this information to each other, it further increases that likelihood.

The thought that this could be solved by simply using TCP is self-defeating. TCP requires additional bandwidth and additional CPU time for each packet generated and each packet received.

Using a massive tree of players globbed together, sharing the processing and bandwidth burdens, you would be distributing the data slowly, so probably a turn based game would work well (turn ends when all players catch up for example).

This is where the challenge comes in. Using a tool like traceroute (tracert on Windows) can indicate its effectiveness.

FYI, all computer games are turn based.

[Risujin]

As you stated before, the actual likelihood of receiving a packet is much greater than that of losing one. However, packets are lost. Packets are lost frequently. Each hop between your computer and the server increases the likelihood of a lost packet. If the servers then relay this information to each other, it further increases that likelihood.

The packets travel a different but not necessarily an inferior route. The problem of packet loss doesnt sound so scary to me.

The thought that this could be solved by simply using TCP is self-defeating. TCP requires additional bandwidth and additional CPU time for each packet generated and each packet received.

So don't mention it.

FYI, all computer games are turn based.

Then so is your brain. When those "turns" happen to be 50 msec each and they dont wait for you to make a move, we call it (more or less) real time.

[Caveman]

FYI, all computer games are turn based.

Then so is your brain. When those "turns" happen to be 50 msec each and they dont wait for you to make a move, we call it (more or less) real time.

You just might want to check up on that. The brain is the only "device" that is capable of real multitasking and does not use the time-frame-slot structure