TremSnaps 2

[Syntac]

Say hello to incarnation #2 of my venerable project, TremSnaps. It takes a snapshot of the entire Tremulous "network" and saves it for future viewing.

Requirements
* A web server running PHP (tested on version 5.2.5)
* A cron job pointed at private/generate.php (for automatic snapshot creation; also, make sure a time limit isn't set)
* Read/write permissions in private/dumps/

Downloads
2.0 (highly defective)
2.1
  2.1.1

[Hendrich]

Good job Syntac, I can use this little App to show my children someday of how active Tremulous used to be before....

[Archangel]

http://www.aodclan.us/tremsnaps/

Still same 'bug' as last time. I don't see a lot of servers on there, ones that I know exist and are popular.

[David]

He's truncating each packet to 1024 bytes.
The master sends ~1400 per-packet.
If he's reading more than one packet he's mangling them all too.

[Undeference]

Color handling is wrong.
for($i = 0; $i < count($sections); $i++) {
if($i % 2) {
if($addr && $port) {
$servers[] = array("addr" => $addr, "port" => $port);
}
} else {
$addr = @inet_ntop(substr($sections[$i], 0, 4));
$port = @unpack("n", substr($sections[$i], 4, 2));
$port = $port[1];
}
}Maybe I'm missing something, but that looks wrong to me. Is that ignoring every other server?

He's truncating each packet to 1024 bytes.

It doesn't look to me like that should be happening, though from what I see at least 750 bytes were received, so maybe it isn't getting anything beyond the first 1024.

http://www.aodclan.us/tremsnaps/

Knowing what, if any, errors there are might help Syntac...

[Syntac]

Archangel: I don't know what you're talking about. Tell me what the problem is!

Undeference: I need to check up on the loop there; I wrote it a long time ago. Also, what do you mean by the color handling being wrong?

[ed] Ugh, you may be right about skipping every other server...? This is probably what Archangel is going on about.
[ed2] Holy fuck, you were right. How could I have been so stupid??? (New version coming out by the way, once you tell me about the color thing.)

[Undeference]

Code Select Expand

#define Q_COLOR_ESCAPE  '^'
#define Q_IsColorString(p)      ( p && *(p) == Q_COLOR_ESCAPE && *((p)+1) && isalnum(*((p)+1)) ) // ^[0-9a-zA-Z]
A caret followed by an alphanumeric character is a valid color code.

Code Select Expand

#define ColorIndex(c)   ( ( (c) - '0' ) & 7 )
function ColorIndex ($c) {
return ord ($c) & 7;
}

[Syntac]

Okay, cool. Can do.

[ed] Um, that always returns 0. Duh, I was using the wrong function.

[David]

He's truncating each packet to 1024 bytes.

It doesn't look to me like that should be happening, though from what I see at least 750 bytes were received, so maybe it isn't getting anything beyond the first 1024.

I was getting confused between read and recv.  (The latter bins everything that wont fit in the buffer).

[Archangel]

actually, recv just keeps it in the 'waiting room' (dumbing this down here)
recv returns the number of bytes ACTUALLY read, so let's say there's 850 bytes in the buffer and you request 1024, recv returns 850
at least this is from my perl knowledge of the function

Code Select Expand

       recv SOCKET,SCALAR,LENGTH,FLAGS
               Receives a message on a socket.  Attempts to receive LENGTH characters of data into variable SCALAR from the specified SOCKET filehandle.
               SCALAR will be grown or shrunk to the length actually read.  Takes the same flags as the system call of the same name.  Returns the
               address of the sender if SOCKETâs protocol supports this; returns an empty string otherwise.  If thereâs an error, returns the undefined
               value.  This call is actually implemented in terms of recvfrom(2) system call.  See "UDP: Message Passing" in perlipc for examples.

               Note the characters: depending on the status of the socket, either (8-bit) bytes or characters are received.  By default all sockets oper-
               ate on bytes, but for example if the socket has been changed using binmode() to operate with the ":utf8" I/O layer (see the "open" pragma,
               open), the I/O will operate on UTF-8 encoded Unicode characters, not bytes.  Similarly for the ":encoding" pragma: in that case pretty
               much any characters can be read.


[David]

From man 2 recv:
All three routines return the length of the message on successful completion.  If a message  is too  long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from.

In my experience, for UDP recv returns one packet per call, and discards any overflow.

[Syntac]

4096 is large enough, hopefully?

[David]

1400 is the max the master will send.  1500 to be safe.

[Syntac]

Okay, here's the current version of the color escape parser.

Code Select Expand

function get_color_index($chr)
{
$chr = substr($chr, 0, 1);
return ord($chr) & 7;
}

function parse_color_escapes($str)
{
$num = preg_match_all("/\^[^\n]/", $str, $null);

$replace = array(
0 => "<span style=\"color: #222;\">",
1 => "<span style=\"color: #F00;\">",
2 => "<span style=\"color: #0F0;\">",
3 => "<span style=\"color: #FF0;\">",
4 => "<span style=\"color: #00F;\">",
5 => "<span style=\"color: #0FF;\">",
6 => "<span style=\"color: #F0F;\">",
7 => "<span style=\"color: #FFF;\">"
);

$str = str_replace(array("<", ">"), array("&lt;", "&gt;"), $str);
$str = str_replace(array("^&lt;", "^&gt;"), array("^<", "^>"), $str);

for($i = 0; $i < strlen($str); $i++) {
if($str[$i] == "^" && isset($str[$i+1]) && preg_match("/[^\n]/", $str[$i+1])) {
$c = get_color_index($str[$i+1]);
$str = substr_replace($str, $replace[$c], $i, 2);
}
}

for($i = 0; $i < $num; $i++) {
$str .= "</span>";
}

return trim($str);
}

[Syntac]

O rly?

--> 2.1 released, check the OP.

[Archangel]

http://www.aodclan.us/tremsnaps/

Still same 'bug' as last time. I don't see a lot of servers on there, ones that I know exist and are popular.

running 2.1

[Syntac]

And...?

--> 2.1.1 released. Fixes a few minor issues.

[Bissig]

Going to aods tremsnaps url and seeing this for "house of nuts":

----> ouse of uts

Doesn't look quite right.

[Syntac]

I think I fixed that in 2.1.1. The problem was that I was stripping out unprintables before parsing the color escapes, so it ended up as "^House of ^Nuts".

[Oblivion]

I will be adding this to Gamez-Host as well. 
Good work.

[Syntac]

Thanks.

If anyone particularly wants it, I can add a more thorough statistics system.

[Undeference]

If anyone particularly wants it, I can add a more thorough statistics system.

I thought that was its purpose initially

[Syntac]

The purpose of TremSnaps is to take a snapshot of Tremulous; that is, to archive the state of every server for future retrieval. Perhaps you're thinking of TremStats?

[Undeference]

The purpose of TremSnaps is to take a snapshot of Tremulous; that is, to archive the state of every server for future retrieval.

What is the purpose of that without generating statistics based on that data?

Perhaps you're thinking of TremStats?

No.

[Syntac]

You seem to have a completely different idea of the purpose of archival. You're free to fork TremSnaps if you'd like, but until more people want a feature, I'm not adding it. My time is scarce enough as it is.

[David]

How do you store the data?  I assume mysql or whatever?
Assuming a sane scheme, then people should be able to extract stats without issue.

[Syntac]

It's stored in a serialized PHP array (private/dumps/<snapshot>, <snapshot> being a Unix timestamp corresponding to when the dump was generated).

[David]

Ah.
How big is a snapshot?  Data-analysis may be "fun" once you have a few tens of thousands of snapshots.

[Syntac]

Somewhere around 210 KB for 200 servers. Given that most of those are empty, it's a waste of space.

generate.php has a few switches you can set, most notably --getservers-args. "--getservers-args full" will cause the master to only report populated servers. (A snapshot generated with that option is 68 KB.)