Log files are huge

[wheelmac92]

While playing some Demos of games, 1 megabyte of log is written every 2 seconds. After playing for a while, the log is 700 megabytes large. When the game crashes, it feels inconvenient to upload it, because it is so large. Is there any way to narrow down the log to only the most relevant information? For example, if a game crashes after playing for an hour, is there any way to determine the last few minutes of gameplay? Does this happen in all games?

It happened for me while using the Dec 4 release of Xenia (latest) while playing Diablo 3, Pixar Up, and Sonic Unleashed. I can test more games if you'd like me to

The rapid speed of writing to log was happening during 3D scenes.
During 2D intro cutscenes, the logs were increasing at a slower rate

[Razzile]

setting the command line argument log_level can help with this. using --log_level=0 will only log errors. Also it's possible to zip (or otherwise compress) the log file which can shrink its size by sometimes as much as 10x.

[Margen67]

Xenia should automatically compress logs like rpcs3 does to avoid this.

[jpb18]

Xenia should automatically compress logs like rpcs3 does to avoid this.

Think using gzip to compress the logs would be enough?
EDIT: gonna checkout if you can use gzip with streaming data

https://gist.github.com/jgarzik/2716209 <--- an 8 year old example, but the Boost library seems to have some tools to help with that

https://github.com/geromueller/zstream-cpp <--- found this library that seems to make the implementation of the zipped log easier.

[jpb18]

Also, another sugestion I've been thinking: instead of having a single monolithic file, why don't we split the log into several files as it goes along? Probably, if there's an issue, we don't need the entire 700 megabytes (example given before) log to track a problem. Maybe the last 100000 lines could be enough...

[halotroop2288]

Maybe think about tracking the number of times a line or chunk gets repeated instead of printing them to the log repeatedly, as well?

A very simple implementation could look like this...
Here's some pseudocode:

string log_buffer;
uint repetition_count = 0;

Main() {
	// An example
	Log("HELLO WORLD!");
	Log("HELLO WORLD!");
	Log("HELLO WORLD!");
	Log("HI UNIVERSE!");
	Log("GREETINGS GITHUB");
	Log("GREETINGS GITHUB");
	Log("HI UNIVERSE!");
}

Log(string message) {
	if (message == log_buffer.last(message.length)) { // Check if the message has already been logged once
		repetition_count++; // Increase the counter
		// Don't log yet
	} else {
		// If the message was repeated, add the counter to the line
		if (repetition_count > 0) {
			  // Add the message count to the line
			  log_buffer +=  " [x" + repetition_count + "]"; // Append counter string to the message line
		} else {
			// Add the message to the line
			log_buffer += "\n" + message; // Append the message to a new line
		}
		// NOW log
		Logger.Log(log_buffer); // Call the regular log function
		log_buffer = ""; // clear the log buffer
	}
}

The output should look something like

HELLO WORLD! [x3]
HI UNIVERSE!
GREETINGS GITHUB [x2]
HI UNIVERSE!

Which is less than ideal, but that's the best I can come up with.

[Razzile]

I personally think this feature isn't needed because users can combat log file sizes in other ways:

1. set their log level to something lower (e.g. 0 for errors only)
2. if they need to share a log they can zip the log file up afterwards and save ~90% file size

If we were to tackle log size I think the best approach would be to compress the log in-memory, but then that makes logging much more computationally expensive than it already is. Not to mention compressing the log as you go will never be able to reach the level of compression you could get from compressing (e.g. zipping the log) after the fact.

[Triang3l]

The order of the log message is also often important (like eDRAM resolve, then texture load), but it would probably be good to move per-frame versions to the "info" level. Though the downside is that we won't even see texture formats in users' logs at all in this case. Many messages, however, have variables in them, so deduplication won't directly help this way, while deduplication by the ID won't be revealing the most interesting infromation — the variable values themselves.

What would be really interesting for compatibility reporting is telemetry of the features used in the game (such as texture/framebuffer/resolve/memexport formats, OS functions) without any host-specific information, so database lookups could be done to locate games that require some feature. However, there's no way the protocol can be made secure probably — modified clients would be able to report any info, which would require manual cleanup somehow.

[KateHanami]

still an issue, I re-downloaded it last night and played a game for 2-3 hours, and wake up this morning with a 15GB log that was big as the game I was playing