This tutorial is good example on how to use it.
To install it on ubuntu, sudo apt-get install linux-tools
A quick example on how to find the bottleneck of a program.
- Run the program as usual to collect performance record, for example " perf record iperf -c 192.168.1.1 -d", This command will ask perf to execute command "iperf -c 192.168.1.1 -d" and collect the performance number in "perf.data" file.
- View the performance record: perf record, this will show you how much time is spent in a userspace function or kernel function. In my example, it looks like this:
31.95% iperf [kernel.kallsyms] [k] md5_transform 16.99% iperf [aesni_intel] [k] _aesni_enc1 3.14% iperf [kernel.kallsyms] [k] do_csum 2.50% iperf [kernel.kallsyms] [k] memcpy 2.16% iperf [kernel.kallsyms] [k] __ticket_spin_lock 1.78% iperf [aesni_intel] [k] _aesni_dec4 1.10% iperf [kernel.kallsyms] [k] nf_iterate 0.97% iperf [nf_conntrack] [k] ____nf_conntrack_find 0.91% iperf [kernel.kallsyms] [k] __slab_free 0.81% iperf [kernel.kallsyms] [k] skb_release_data 0.80% iperf [kernel.kallsyms] [k] fib_table_lookup 0.80% iperf [kernel.kallsyms] [k] memset 0.80% iperf [kernel.kallsyms] [k] __copy_user_nocache 0.77% iperf [cxgb4] [k] process_responses 0.77% iperf [ip_tables] [k] ipt_do_table 0.69% iperf [nf_conntrack] [k] __nf_conntrack_find_get 0.68% iperf [kernel.kallsyms] [k] md5_update 0.67% iperf [nf_conntrack] [k] nf_conntrack_in 0.56% iperf [nf_conntrack] [k] hash_conntrack_raw 0.52% iperf [kernel.kallsyms] [k] dst_release
This is because I'm running tcp benchmark using iperf with ipsec turned on in ESP and AH mode, so a lot of cpu time is spent on md5(for checksum) and aes(for encryption). Now that we know where the performance bottlenecks are, it's time to spend time to fix them.
