A group of academic researchers have obtained the complete server logs for the Everquest 2 MMORPG. It’s four years of data for over 400,000 players – the resulting dataset is nearly 60TB. That’s right, terabytes. Combined with some demographic surveys there is interesting datamining potential here.
This is also interesting because apparently the standard tools don’t quite scale to the task of analyzing this data:
Regardless of format, many one-pass, exhaustive algorithms simply choke on a dataset this large, which is forcing his group to use some incremental analysis methods or to work with subsets of the data.
Some items in the results that I found interesting:
Posted by Karl W. Reinsch 