TL;DR: There are a total of 5 uops in the unfused domain (See: Micro fusion and addressing modes). The loop stream detector on Ivy Bridge cannot allocate uops across the loop body boundaries (See: Is performance reduced when executing loops whose uop count is not a multiple of processor width?), so it takes two cycles to allocate one iteration. The loop actually runs at 2.3c/iter on dual socket Xeon E5-2680 v2 (10 cores per socket vs. your 12), so that is close to the best that can be done given the front-end bottleneck.
The prefetchers have performed very well, and most of the time the loop is not memory-bound. Copying 1 byte per 2 cycles is very slow. (gcc did a poor job, and should have given you a loop that could run at 1 iteration per clock. Without profile-guided optimization, even -O3 doesn't enable -funroll-loops, but there are tricks it could have used (like counting a negative index up toward zero, or indexing the load relative to the store and incrementing a destination pointer) that would have brought the loop down to 4 uops.)
The extra .3 cycles per iteration slower than the front-end bottleneck on average is probably coming from stalls when prefetching fails (maybe at page boundaries), or maybe from page faults and TLB misses in this test that runs over statically-initialized memory in the .data section.
There are two data dependencies in the loop. First, the store instruction (specifically the STD uop) depends on the result of the load instruction. Second, both the store and load instructions depend on add rax, 0x1. In fact, add rax, 0x1 depends on itself as well. Since the latency of add rax, 0x1 is one cycle, an upper bound on performance of the loop is 1 cycle per iteration.
Since the store (STD) depends on the load, it cannot be dispatched from the RS until the load completes, which takes at least 4 cycles (in case of an L1 hit). In addition, there is only one port that can accept STD uops yet up to two loads can complete per cycle on Ivy Bridge (especially in the case the two loads are to lines that are resident in the L1 cache and no bank conflict occurs), resulting in additional contention. However, RESOURCE_STALLS.ANY shows that the RS actual never gets full. IDQ_UOPS_NOT_DELIVERED.CORE counts the number of issue slots that were not utilized. This is equal to 36% of all slots. The LSD.CYCLES_ACTIVE event shows that the LSD is used to deliver uops most of the time. However, LSD.CYCLES_4_UOPS/LSD.CYCLES_ACTIVE =~ 50% shows that in about 50% of the cycles, less than 4 uops are delivered to the RS. The RS will not get full because of the sub-optimal allocation throughput.
The stalled-cycles-frontend count corresponds to UOPS_ISSUED.STALL_CYCLES, which counts allocation stalls due to both frontend stalls and backend stalls. I don't understand how UOPS_ISSUED.STALL_CYCLES is related to the number of cycles and other events.
The LLC-loads count includes:
- All demand load requests to the L3 irrespective of whether the request hits or misses in the L3 and, in case of a miss, irrespective of the source of data. This also includes demand load requests from the page walking hardware. It's not clear to me whether load requests from the next-page prefetcher are counted.
- All hardware prefetch data read requests generated by an L2 prefetcher where the target line is to be placed in the L3 (i.e., in the L3 or both in the L3 and L2, but not only in the L2). Hardware L2 prefetcher data read requests where the line is to be placed only in the L2 are not included. Note that the L1 prefetchers' requests go to the L2 and influence and may trigger the L2 prefetchers, i.e., they don't skip the L2.
LLC-load-misses is a subset of LLC-loads and includes only those events that missed in the L3. Both are counted per core.
There is an important difference between counting requests (cache-line granularity) and counting load instructions or load uops (using MEM_LOAD_UOPS_RETIRED.*). Both the L1 and L2 caches squash load requests to the same cache line, so multiple misses in the L1 may result in a single request to the L3.
Optimal performance can be achieved if all stores and loads hit in the L1 cache. Since the size of a buffer you used is 1GB, the loop can cause a maximum of 1GB/64 =~ 17M L3 demand load requests. However, your LLC-loads measurement, 83M, is much larger, probably due to code other than the loop you've shown in the question. Another possible reason is that you forgot to use the :u suffix to count only user-mode events.
My measurements on both IvB and HSW show that LLC-loads:u is negligible compared to 17M. However, most of the L3 loads are misses (i.e., LLC-loads:u =~ LLC-loads-misses:u). CYCLE_ACTIVITY.STALLS_LDM_PENDING shows that the overall impact of loads on performance is negligible. In addition, my measurements show that the loop runs at 2.3c/iter on IvB (vs. 1.5c/iter on HSW), which suggests that one load is issued every 2 cycles. I think that the sub-optimal allocation throughput is the main reason for this. Note that 4K aliasing conditions (LD_BLOCKS_PARTIAL.ADDRESS_ALIAS) are almost non-existent. All of this means that the prefetchers have done a pretty good job at hiding the memory access latency for most loads.
Counters on IvB that can be used to evaluate the performance of hardware prefetchers:
Your processor has two L1 data prefetchers and two L2 data prefetchers (one of them can prefetch both into the L2 and/or the L3). A prefetcher may not be effective for the following reasons:
- A triggering condition has not been satisfied. This is typically because an access pattern has not been recognized yet.
- The prefetcher has been triggered but the prefetch was to a useless line.
- The prefetcher has been triggered to a useful line but the line got replaced before being used.
- The prefetcher has been triggered to a useful line but the demand requests have already reached the cache and missed. This means that the demand requests were issued faster than the ability of the prefetcher to react in a timely manner. This can happen in your case.
- The prefetcher has been triggered to a useful line (that doesn't exist in the cache), but the request had to be dropped because no MSHR was available to hold the request. This can happen in your case.
The number of demand misses at the L1, L2, and L3 are good indicators of how well the prefetchers have performed. All the L3 misses (as counted by LLC-load-misses) are also necessarily L2 misses, so the number of L2 misses is larger than LLC-load-misses. Also all of the demand L2 misses are necessarily L1 misses.
On Ivy Bridge, you can use the LOAD_HIT_PRE.HW_PF and CYCLE_ACTIVITY.CYCLES_* performance events (in addition to the miss events) to know more about how the prefetchers have performed and evaluate their impact on performance. It's important to measure CYCLE_ACTIVITY.CYCLES_* events because even if the miss counts were seemingly high, that doesn't necessarily mean that misses are the main cause of performance degradation.
Note that the L1 prefetchers cannot issue speculative RFO requests. Therefore, most writes that reach the L1 will actually miss, requiring the allocation of an LFB per cache line at the L1 and potentiality other levels.
The code I used follows.
BITS 64
DEFAULT REL
section .data
bufdest: times COUNT db 1
bufsrc: times COUNT db 1
section .text
global _start
_start:
lea rdi, [bufdest]
lea rsi, [bufsrc]
mov rdx, COUNT
mov rax, 0
.loop:
movzx ecx, byte [rsi+rax*1]
mov byte [rdi+rax*1], cl
add rax, 1
cmp rdx, rax
jnz .loop
xor edi,edi
mov eax,231
syscall
memcpysince thecountargument has typeintinstead ofsize_t. It should at least generate a test such asif (count <= 0) return;or some other code to account for negative size values. – chqrliecountis signed negative, and on 64-bit ISAs, zero-extend it tosize_t. PSKocik just meant that they don't actually auto-vectorize the loop, they use memcpy for the real work instead. godbolt.org/z/NGJi1e has gcc and clang output for x86-64, ARM, AArch64 and MIPS. – Peter Cordes