Skip to content

PAPI

PAPI provides an interface to the hardware counters of CPUs. These counters hint at potential performance issues in running programs.

PAPI's library and companion utilities are available on Apocrita on loading a module.

Using PAPI

For users on Apocrita the PAPI libraries would generally be used indirectly, as part of some other profiling or performance tool. However, the library is available for user code and the library interface is documented here.

More useful for users of Apocrita are the companion utilities which provide information about the available hardware counters. For example, the utility papi_avail details the counters (with the option -a only those counters which may be queried are listed). Counters which may be used will vary by CPU and, for example, the output of papi_avail -a on an NXV node looks like:

  PAPI Preset Events
================================================================================
    Name        Code    Deriv Description (Note)
PAPI_L1_DCM  0x80000000  No   Level 1 data cache misses
PAPI_L1_ICM  0x80000001  No   Level 1 instruction cache misses
PAPI_L2_DCM  0x80000002  Yes  Level 2 data cache misses
PAPI_L2_ICM  0x80000003  No   Level 2 instruction cache misses
PAPI_L1_TCM  0x80000006  Yes  Level 1 cache misses
PAPI_L2_TCM  0x80000007  No   Level 2 cache misses
PAPI_L3_TCM  0x80000008  No   Level 3 cache misses
PAPI_CA_SNP  0x80000009  No   Requests for a snoop
PAPI_CA_SHR  0x8000000a  No   Requests for exclusive access to shared cache line
PAPI_CA_CLN  0x8000000b  No   Requests for exclusive access to clean cache line
PAPI_CA_INV  0x8000000c  No   Requests for cache line invalidation
PAPI_CA_ITV  0x8000000d  No   Requests for cache line intervention
PAPI_L3_LDM  0x8000000e  No   Level 3 load misses
PAPI_TLB_DM  0x80000014  Yes  Data translation lookaside buffer misses
PAPI_TLB_IM  0x80000015  No   Instruction translation lookaside buffer misses
PAPI_L1_LDM  0x80000017  No   Level 1 load misses
PAPI_L1_STM  0x80000018  No   Level 1 store misses
PAPI_L2_LDM  0x80000019  No   Level 2 load misses
PAPI_L2_STM  0x8000001a  No   Level 2 store misses
...

Here the first column gives the name of the counter and the final column a brief description of the counter. This name will be accepted by the PAPI library to refer to the counter and will generally be used by third-party tools.

Hardware counters presented may be either measured directly or derived. Derived results (indicated by the third column in the output above) are calculated considering a number of directly measured counters.

Multiple counters may be recorded during a single profiling run of a program. However, it is not possible to watch all counters at once and some combinations of counters are not observable. The utility papi_command_line can be used to see whether a particular combination of counters may be selected. For example, on an NXV node the counters PAPI_L1_DCM, PAPI_L3_DCR and PAPI_L1_TCM are compatible:

$ papi_command_line PAPI_L1_DCM PAPI_L3_DCR PAPI_L1_TCM

This utility lets you add events from the command line interface to see if they work.

Successfully added: PAPI_L1_DCM
Successfully added: PAPI_L3_DCR
Successfully added: PAPI_L1_TCM

PAPI_L1_DCM :   2329
PAPI_L3_DCR :   52
PAPI_L1_TCM :   2486

----------------------------------

whereas the counters PAPI_L1_DCM, PAPI_L3_DCR, PAPI_L1_TCM, PAPI_SP_OPS and PAPI_DP_OPS are not compatible:

$ papi_command_line PAPI_L1_DCM PAPI_L3_DCR PAPI_L1_TCM PAPI_SP_OPS PAPI_DP_OPS

This utility lets you add events from the command line interface to see if they work.

Successfully added: PAPI_L1_DCM
Successfully added: PAPI_L3_DCR
Successfully added: PAPI_L1_TCM
Successfully added: PAPI_SP_OPS
Failed adding: PAPI_DP_OPS
because: Invalid argument

PAPI_L1_DCM :   2369
PAPI_L3_DCR :   51
PAPI_L1_TCM :   2549
PAPI_SP_OPS :   0

----------------------------------

Incompatible combinations of counters may be resolved by separating collection between multiple profiling runs of the program (again using papi_command_line or papi_event_chooser to assist).

Reference