Skip to content
kmabbasi edited this page Jan 11, 2021 · 16 revisions

How to find Intel® RDT hardware and software capabilities?

To display detected Intel® RDT capabilities please use -d option or -D for more verbose output including cache information. Those options display Intel® RDT capabilities supported via MSR (legacy) and the OS/Kernel interfaces.

Example pqos -d command output:

Hardware capabilities
    Monitoring
        Cache Monitoring Technology (CMT) events:
            LLC Occupancy (LLC)
        Memory Bandwidth Monitoring (MBM) events:
            Total Memory Bandwidth (TMEM)
            Local Memory Bandwidth (LMEM)
            Remote Memory Bandwidth (RMEM) (calculated)
        PMU events:
            Instructions/Clock (IPC)
            LLC misses
    Allocation
        Cache Allocation Technology (CAT)
            L3 CAT
                CDP: disabled
                Num COS: 16
        Memory Bandwidth Allocation (MBA)
            Num COS: 8
OS capabilities (Linux kernel 4.13.0-rc5)
    Monitoring
        Cache Monitoring Technology (CMT) events:
            LLC Occupancy (LLC)
        Memory Bandwidth Monitoring (MBM) events:
            Total Memory Bandwidth (TMEM)
            Local Memory Bandwidth (LMEM)
            Remote Memory Bandwidth (RMEM) (calculated)
        PMU events:
            Instructions/Clock (IPC)
            LLC misses
    Allocation
        Cache Allocation Technology (CAT)
            L3 CAT
                CDP: disabled
                Num COS: 8
        Memory Bandwidth Allocation (MBA)
            Num COS: 8

Note: The pqos -s command combined with the -v or -V option can also be used to display detected Intel® RDT capabilities and cache information. Please see the next FAQ below for more information.

How to display current Intel® RDT configuration?

To print the current Intel® RDT configuration, please use pqos -s command.

Example pqos -s command output:

L3CA/MBA COS definitions for Socket 0:
    L3CA COS0 => MASK 0x7ff
    L3CA COS1 => MASK 0x7ff
…
    L3CA COS14 => MASK 0x7ff
    L3CA COS15 => MASK 0x7ff
    MBA COS0 => 100% available
    MBA COS1 => 100% available
…
    MBA COS6 => 100% available
    MBA COS7 => 100% available
Core information for socket 0:
    Core 0, L2ID 0, L3ID 0 => COS0, RMID0
    Core 1, L2ID 1, L3ID 0 => COS0, RMID0
…
    Core 46, L2ID 14, L3ID 0 => COS0, RMID0
    Core 47, L2ID 15, L3ID 0 => COS0, RMID0

The -v option will print additional log information messages containing the detected Intel® RDT capabilities and cache information.

Example pqos –s -v command output:

INFO: CACHE: type 1, level 1, max id sharing this cache 2 (1 bits)
INFO: CACHE: type 2, level 1, max id sharing this cache 2 (1 bits)
INFO: CACHE: type 3, level 2, max id sharing this cache 2 (1 bits)
INFO: CACHE: type 3, level 3, max id sharing this cache 32 (5 bits)
INFO: Monitoring capability detected
INFO: CPUID.0x7.0: L3 CAT supported
INFO: CDP is disabled
INFO: L3 CAT details: CDP support=1, CDP on=0, #COS=16, #ways=11, ways contention bit-mask 0x600
INFO: L3 CAT details: cache size 23068672 bytes, way size 2097152 bytes
INFO: L3CA capability detected
INFO: CPUID 0x10.0: L2 CAT not supported!
INFO: L2CA capability not detected
INFO: MBA details: #COS=8, linear, max=90, step=10
INFO: MBA capability detected
INFO: resctrl detected
INFO: OS support for CMT detected
INFO: OS support for L3 CAT detected
INFO: OS support for MBA detected
INFO: resctrl not mounted
L3CA/MBA COS definitions for Socket 0:
    L3CA COS0 => MASK 0x7ff
    L3CA COS1 => MASK 0x7ff
…

The -voption can be replaced with -V to include more verbose debug log messages.

When I try to run the utility I get the following error messages:

WARN: Error opening file '/dev/cpu/0/msr'!
ERROR: CDP detection error!
ERROR: Fatal error encounter in CAT discovery!
ERROR: discover_capabilities() error 1
Error initializing PQoS library!

This is usually seen when the utility is run without root privileges. Another possible cause is when the msr driver has not been auto-loaded.
For some modular kernels the driver may need to be loaded manually by running:
modprobe msr

When I try to run the utility I get the following error message:

pqos: error while loading shared libraries: libpqos-0.1.5.so: cannot open shared object file: No such file or directory

This occurs when the library default installation directory is not included in the linker search path.
To add the path, as root (not with sudo), run:

# echo /usr/local/lib > /etc/ld.so.conf.d/<FILENAME>.conf

Then to update the linker cache run:

sudo ldconfig

When I try to run the utility I get the following error message:

sudo: pqos: command not found

This occurs when the default installation directory has not been added to the secure path section in the sudoers file.
To fix this, open the sudoers file as root with:
visudo
and append the default installation path to the list of secure paths:
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

When I try to start monitoring I get the following error messages:

ERROR: IPC and/or LLC miss performance counters already in use!
Use -r option to start monitoring anyway.
Monitoring start error on core(s) 0, status 6

This is a warning that another application may be using PMU counters to monitor resources. To force the PQoS utility to start monitoring anyway, add the -r option to the command.
Example: pqos -r -p llc:34534 or pqos -r -m :[0-47]

When I try to monitor processes/tasks I get the following error message:

PID 1 monitoring start error,status 1

This is usually caused when the system is using a kernel that does not support PQoS Perf events such as:

  • llc_occupancy - llc (LLC occupancy)
  • local_mb - mbl (Local Memory B/W)
  • total_mb - mbr (Used with MBL to calculate Remote Memory B/W)

NOTE: Kernel version 4.1 or newer is required for PID monitoring.

Does CAT extend to kernel/ring 0 code?

Q: For example, if I have a kernel module run on a particular core, will it be limited to the LLC allocation for that particular core or are the limitations just for userspace applications?
A: Currently, the utility and library offer static CAT configuration. This means cores are assigned to classes of service and this configuration doesn't change when tasks are scheduled in and out on these cores. Consequently Linux kernel threads running on this core will be also affected. Depending on the application, this may be a good or a bad thing. There is work in progress to enable CAT configuration on task basis to address this gap.

Can the portion of cache (i.e. number of cache ways) be changed dynamically?

Yes it can, user applications can modify COS bitmasks at runtime. However, currently CAT can only be managed through the MSR interface. The downside of the MSR interface is that the technologies can only be managed on logical core basis - no task or process tracking capability. There is work in progress to enable CAT configuration on task basis to address this gap.

Different bit masks for different cache sizes

One bit in a COS bitmask corresponds to one cache way. Cache way size can be calculated manually with (Total LLC size divided by number of ways). This information can also be retrieved from the utility by running pqos -s -v and looking for the following lines:
...
INFO: LLC cache size 36700160 bytes, 20 ways
INFO: LLC cache way size 1835008 bytes
...
Note: These lines will only be logged if CAT support is detected.

How fine-grained can the cache assignment be for each COS?

Each COS bitmask must have at least 2 contiguous bits set (i.e. 2 cache ways).
For example, for a CPU with 20MB of LLC and 20 cache ways, each way is ~1MB in size. In this case, at least 2MB of cache must be assigned to each COS, which can be incremented by 1MB thereafter.

Is there a limit for the number of COS on my system?

The number of classes of service is product dependent. To find out how many COS are supported by your product run pqos -s -v and look for the CAT details message:
...
INFO: CAT details: CDP support=1, CDP on=0, #COS=16, #ways=20, ways contention bit-mask 0xc0000
...
In this example, the number of COS supported by the product is 16.
Note: These lines will only be logged if CAT support is detected.

What Operating System's is the software packaged for?

Linux Distribution Release
Ubuntu 16.04 "The Xenial Xerus"
16.10 "The Yakkety Yak" (Devel)
Debian 9 "Stretch" (Testing)
"Sid" (Unstable)
Fedora 24/25/26 (Devel)
EPEL 7
CentOS 7
OpenSuse 13
Leap 42
Tumbleweed
SLES 11/12
Clear Linux 5370 - latest

How do I install the package using a package manager?

To install using the "Advanced Package Tool", or apt do:
$ sudo apt-get install intel-cmt-cat

To install using the "Yellowdog Updater, Modified", or yum do:
$ sudo yum install intel-cmt-cat

To install using the "Dandified Yum", or dnf do:
$ sudo dnf install intel-cmt-cat

How do I build and install from a RPM Spec file?

Note: RPM name has an OS extension so the final name will be OS dependent.
The below examples are based on Linux CentOS.

Basic steps to build source and binary packages in your home directory:

Create the rpmbuild directory structure:
$ rpmdev-setuptree

Next, download the intel-cmt-cat tar file and copy into ~/rpmbuild/SOURCES directory:
$ wget https://github.com/01org/intel-cmt-cat/archive/v0.1.4.tar.gz
$ cp ./v0.1.4.tar.gz ~/rpmbuild/SOURCES

To build, do:
$ rpmbuild -ba path/to/intel-cmt-cat.spec

To install, do:
$ sudo rpm -i ~/rpmbuild/RPMS/x86_64/intel-cmt-cat-0.1.4-3.el7.centos.x86_64.rpm

How do I rebuild the package from a Source RPM file?

To rebuild the package from a SRPM file in your home directory, do:
$ rpmbuild --rebuild path/to/intel-cmt-cat-0.1.4.src.rpm

To install, do:
$ sudo rpm -i ~/rpmbuild/RPMS/x86_64/intel-cmt-cat-0.1.4-3.el7.centos.x86_64.rpm

How to install libpqos Python wrapper?

If you would like to install Python wrapper as a dependency in your application and you are using Pipfile for specifying dependencies, use the following code:

[packages]  
...your dependencies here...  

[packages.pqos]  
editable = true  
path = "path-to-intel-cmt-cat/lib/python"

Alternatively, you may use the following commands:

cd path-to-intel-cmt-cat/lib/python   
python -m pip wheel --wheel-dir=/tmp .   
python -m pip install --no-index --find-links=/tmp pqos  
rm -rf /tmp/pqos*.whl

Note: use python3 instead of python in above commands if your system runs Python 2.x under python command.
You may also use python specific to your Python environment created by virtualenv.

How to run a Python script that uses libpqos Python wrapper?

Python wrapper for libpqos requires built version of libpqos.
If libpqos is installed in system, Python wrapper will find it.

Alternatively, if the library is not installed in the system or you would like to use a different version of the library,
Python wrapper can look for the library in a directory specified by LD_LIBRARY_PATH environment variable,
for example:

LD_LIBRARY_PATH=./intel-cmt-cat/lib python my_script.py