Triereme: Speeding up hybrid fuzzing through efficient query scheduling
Hybrid fuzzing, the combination between fuzzing and concolic execution, holds great promise in theory, but has so far failed to deliver all the expected advantages in practice due to its high overhead. The cause is the large amount of time spent in the SMT solver. As a result, hybrid fuzzers often lose out to simpler, yet faster techniques. This issue remains despite novel query pruning techniques that reduce the number and complexity of solver queries as they preclude other crucial optimizations like incremental solving.
We introduce Triereme, a method to speed up the hybrid fuzzer's concolic engine by reducing the time spent in the SMT solver. Triereme uses a trie (or prefix tree) data structure to schedule and cache solver queries, exploiting common prefixes. This design is made possible by decoupling concolic tracing from concolic solving. As a result, Triereme manages to reconcile pruning with incremental solving, reaping their combined benefits. In our tests, Triereme speeds up concolic executions by 6.1x on average in FuzzBench and improves coverage progress in 79% of the benchmarks.