• Xin Tong's avatar
    implementing victim TLB for QEMU system emulated TLB · 88e89a57
    Xin Tong authored
    QEMU system mode page table walks are expensive. Taken by running QEMU
    qemu-system-x86_64 system mode on Intel PIN , a TLB miss and walking a
    4-level page tables in guest Linux OS takes ~450 X86 instructions on
    average.
    
    QEMU system mode TLB is implemented using a directly-mapped hashtable.
    This structure suffers from conflict misses. Increasing the
    associativity of the TLB may not be the solution to conflict misses as
    all the ways may have to be walked in serial.
    
    A victim TLB is a TLB used to hold translations evicted from the
    primary TLB upon replacement. The victim TLB lies between the main TLB
    and its refill path. Victim TLB is of greater associativity (fully
    associative in this patch). It takes longer to lookup the victim TLB,
    but its likely better than a full page table walk. The memory
    translation path is changed as follows :
    
    Before Victim TLB:
    1. Inline TLB lookup
    2. Exit code cache on TLB miss.
    3. Check for unaligned, IO accesses
    4. TLB refill.
    5. Do the memory access.
    6. Return to code cache.
    
    After Victim TLB:
    1. Inline TLB lookup
    2. Exit code cache on TLB miss.
    3. Check for unaligned, IO accesses
    4. Victim TLB lookup.
    5. If victim TLB misses, TLB refill
    6. Do the memory access.
    7. Return to code cache
    
    The advantage is that victim TLB can offer more associativity to a
    directly mapped TLB and thus potentially fewer page table walks while
    still keeping the time taken to flush within reasonable limits.
    However, placing a victim TLB before the refill path increase TLB
    refill path as the victim TLB is consulted before the TLB refill. The
    performance results demonstrate that the pros outweigh the cons.
    
    some performance results taken on SPECINT2006 train
    datasets and kernel boot and qemu configure script on an
    Intel(R) Xeon(R) CPU  E5620  @ 2.40GHz Linux machine are shown in the
    Google Doc link below.
    
    https://docs.google.com/spreadsheets/d/1eiItzekZwNQOal_h-5iJmC4tMDi051m9qidi5_nwvH4/edit?usp=sharing
    
    In summary, victim TLB improves the performance of qemu-system-x86_64 by
    11% on average on SPECINT2006, kernelboot and qemu configscript and with
    highest improvement of in 26% in 456.hmmer. And victim TLB does not result
    in any performance degradation in any of the measured benchmarks. Furthermore,
    the implemented victim TLB is architecture independent and is expected to
    benefit other architectures in QEMU as well.
    
    Although there are measurement fluctuations, the performance
    improvement is very significant and by no means in the range of
    noises.
    Signed-off-by: 's avatarXin Tong <trent.tong@gmail.com>
    Message-id: 1407202523-23553-1-git-send-email-trent.tong@gmail.com
    Reviewed-by: 's avatarPeter Maydell <peter.maydell@linaro.org>
    Signed-off-by: 's avatarPeter Maydell <peter.maydell@linaro.org>
    88e89a57
cputlb.c 11.9 KB