Too large perf.data with too few events

Question

Recorded some stats using:

perf record -a -F 20 -o perf.data -e major-faults sleep 1800

and got perf.data ~ 1GiB with samples: 355, event count: 7592:

# Total Lost Samples: 0
#
# Samples: 355  of event 'major-faults'
# Event count (approx.): 7592

Why a few samples took lots of space? Is there a tool to look inside perf.data to find out what it actually contains?

Using command:

perf report -i perf.data -D

I've found these events in perf.data: (cut hex dumps of each event)

0 0 0x63a0 [0x30]: PERF_RECORD_COMM: gmain:1000/1010
0x63d0 [0x38]: event: 7
0 0 0x63d0 [0x38]: PERF_RECORD_FORK(1004:1004):(1:1)
0x6408 [0x30]: event: 3
0 0 0x6408 [0x30]: PERF_RECORD_COMM: cron:1004/1004
0x6438 [0x70]: event: 10
0 0 0x6438 [0x70]: PERF_RECORD_MMAP2 1004/1004: [0x5586d93d0000(0xb000) @ 0 fd:01 3285696 93896821003936]: r-xp /usr/sbin/cron

But I didnt asked perf to record those events with -e selector. How to avoid recording this?

tool to look inside perf.data is perf script. It will dump every event recorded in the file. Also supports -D option to add hex dump: perf script -D. You asked perf to do system-wide recording with -a options, and to be able to decode events in any program it must to record every program starting and exec segments mmaping for all 1800 seconds on every cpu. Don't know how to avoid recording these events in perf with option (perf by default asks OS to record with perf_event_open syscall args). You should check tracing tools like bpf (brendangregg.com/ebpf.html) or lttng. — osgx

Arnabjyoti Kalita Arnabjyoti Kalita · Accepted Answer · 2020-04-06T13:32:49

As @osgx has already mentioned, the tool to look inside the perf.data file is perf script. perf script -D dumps raw events from the perf.data file in the hex format.

The perf.data file contains all the events generated by the Performance Monitoring Units, plus some metadata. The on-disk perf.data file usually begins with a perf_header struct of the form below -

struct perf_header {
    char magic[8];      /* PERFILE2 */
    uint64_t size;      /* size of the header */
    uint64_t attr_size; /* size of an attribute in attrs */
    struct perf_file_section attrs;
    struct perf_file_section data;
    struct perf_file_section event_types;
    uint64_t flags;
    uint64_t flags1[3];
};

The bulk of the perf.data file includes perf events, which can include any of the event types mentioned here that contain metadata about each of the events, like the 32-bit process id and thread id, instruction pointer, information about the CPU being used etc. Storage of this metadata is subject to various flags being passed, like PERF_SAMPLE_PID or PERF_SAMPLE_TID etc. Look into the perf_event_open manpage. You can disable recording some of the metadata and reduce the size of each event data being written to the file.

The PERF_RECORD_COMM, PERF_RECORD_FORK and PERF_RECORD_MMAP etc. are sideband events recorded by the kernel, to help in further post-processing and detailed analysis. They are enabled by default in the kernel source code, which can be seen here.

struct perf_event_attr {

                ........
                mmap           :  1, /* include mmap data     */
                comm           :  1, /* include comm data     */
                freq           :  1, /* use freq, not period  */
                inherit_stat   :  1, /* per task counts       */
                enable_on_exec :  1, /* next exec enables     */
                task           :  1, /* trace fork/exit       */

Having 1 in these fields means they are enabled by default, and to disable logging of these events, you'd have to make them 0 in the source code and recompile only the userspace perf module of the kernel. If set to 0, these events will not be recorded as can be seen here.

There are no command line switches or options with perf record, that would enable these events to be disabled.

Too large perf.data with too few events

1 Answers