For a very long time, tremstats were running just fine. But with more and more games, the database filled up extremly and slowed down with every game more and more.
Well, there's an easy solution. I could pre-render all graphs just after a parsing and store everything, which is calculated in realtime a the moment, in the database. This would increase the speed of the output dramaticly, but also take a longer time of precalculation on the one hand, and also more diskspace on the other hand.
A plus of this would be, that the cpu time goes down, because there will be less calculation left for the ouput (which is called more often than the pre-calculation process).
So, would you accept such a change or do you prefer a short pre-calculation time?
One obvious solution is to give people a lot of control over tremstats's output (e.g., do we want graphs or is text fine?). Since most people want all the bells and whistles, that doesn't really help.
Your solution to the database getting larger is to put more data in the database? In that case, the performance hit will move from image rendering to database queries.
How about this? Don't generate graphs when updating the database. Only render graphs once and when they are needed. And avoid blobs in MySQL databases at all costs! Retrieving graphs directly from the FS is faster and has less overhead than retrieving them from the DB. Since the graphs are generated from information readily available, they can be transparently generated when needed.
This takes the lazy approach: do things at the last possible moment and avoid ever doing them again.
The advantages are that the same graph does not have to be repeatedly generated (so a 6 month old graph can be used if the data hasn't changed), there is no DB performance hit, the graphs can be regenerated if something went wrong or if there is a problem with the cache, and caching can be trivially disabled (e.g., for benchmarking).
So I vote for faster output and pre-calculation.
Poll