Log details
Here is a list of things I would like to see work differently in the log:
- There is currently a bug-log.txt and a full-log.txt, I would actually like to split the current contents of bug-log.txt into sut-bug-log.txt and xsmith-bug-log.txt (feel free to improve the names if you want). The point being that I want issues where xsmith crashed or the test crashed on all implementations (IE the generated code was obviously bad) to go in the xsmith-bug-log, while different output or tests that crash on some but not all implementations should go in the sut-bug-log. This would make it much easier to find the (much more important!) language implementation bugs that we want to find when there are still errors in the fuzzer itself.
- When all implementations crash on an input, the input itself is clearly bad. In this case I want the log to show me the seed and just one error output. I maybe already implemented this improvement, but I really just want to see one instance of the crash log with its seed/options, not one for each implementation.
- Xsmith generation timeouts - I would prefer the log just shows the seed/options and the fact that it timed out with no stack trace or anything -- the stack trace of a timeout is uninteresting and clutters the log. Also, I only want to see these in the full log rather than the bug log, just so I can get a sense of what percentage of calls are timing out rather than finishing overall.
- System-under-test timeouts are more interesting. If some finish and some don't, I absolutely want to see that in the bug log. If they all time out I also want to see it in the bug log for now, but at some point I may want them in a different log file.
These changes are mostly about making it easy to find the most important data with ease, and not losing any bug needles in a log haystack. After fuzzing I want to review as quickly as possible 1: what actual bugs are found (sut-bug-log), 2: what bugs have been uncovered in my fuzzer (xsmith-bug-log), 3: how is my efficiency (full-log comparing number of normal runs, crashes, bugs, timeouts, etc).