QATlib: 23.11.0 release

Changes from 23.08.0 to 23.11.0: * Support DC NS (NoSession) APIs * Support Symmetric Crypto SM3 & SM4 * Support Asymmetric Crypto SM2 * DC compressBound APIs * Bug Fixes. See Resolved section in README.md Signed-off-by: Firos Sadarul <firos.sadarul@intel.com>
intel · Nov 15, 2023 · 142e305 · 142e305
1 parent 7429ee2
commit 142e305
Show file tree

Hide file tree

Showing 96 changed files with 15,528 additions and 1,858 deletions.
diff --git a/INSTALL b/INSTALL
@@ -1,7 +1,4 @@
 ===============================================================================
-
-
-August 2023
 ===============================================================================
 
 
@@ -94,10 +91,9 @@ Check System Prerequisites
         intel_qat
         qat_4xxx
         They should load by default if using any of the following:
-         * Linux kernel v5.11+ (This is for crypto, for compression use v5.17+,
-         for sym-only asym-only, sym;dc and asym;dc use v6.6+, for dcc use v6.7+)
-         * Fedora 34+ (for compression use 36+)
-         * RHEL 8.4+ (for compression use 9.0+)
+        * A recent Linux kernel (see https://intel.github.io/quickassist/qatlib/requirements.html#kernel-firmware-requirements )
+        * Fedora 36+
+        * RHEL 8.4+ (for compression use 9.0+)
      * each PF device must be bound to the 4xxx driver
         Use "ls /sys/bus/pci/drivers/4xxx/" to show the BDFs of each bound PF
      * BIOS settings
@@ -169,7 +165,7 @@ Compilation and installation - quickstart instructions
 
         # Install dependencies
         sudo dnf install -y gcc systemd-devel automake autoconf libtool
-        sudo dnf install -y openssl-devel zlib-devel nasm
+        sudo dnf install -y pkg-config openssl-devel zlib-devel nasm
 
         # Clone QATlib into ~/qatlib, i.e. in your home dir
         cd ~
@@ -227,6 +223,7 @@ Compilation and installation - detailed instructions
             gcc
             make
             autotools (automake, autoconf, libtool)
+            pkg-config
             systemd-devel
             nasm
         Note: If nasm compiler is unavailable see
@@ -720,7 +717,7 @@ Common issues
         bound to qat_4xxx:
         sudo lspci -vvd:4940 | grep "Kernel driver in use".
         sudo lspci -vvd:4942 | grep "Kernel driver in use"
-        If no driver in use, upgrade to Linux kernel 5.11 or greater.
+        upgrade to a recent Linux Kernel.
         (2) No VFs available. Check VFs are available and bound to vfio-pci
         sudo lspci -vvd:4941 | grep "Kernel driver in use"
         sudo lspci -vvd:4943 | grep "Kernel driver in use"

diff --git a/Makefile.am b/Makefile.am
@@ -71,7 +71,8 @@ libadf_la_SOURCES = \
 	quickassist/lookaside/access_layer/src/qat_direct/vfio/qat_log.c \
 	quickassist/lookaside/access_layer/src/qat_direct/vfio/vfio_lib.c \
 	quickassist/lookaside/access_layer/src/qat_direct/vfio/adf_pfvf_proto.c \
-	quickassist/lookaside/access_layer/src/qat_direct/vfio/adf_pfvf_vf_msg.c
+	quickassist/lookaside/access_layer/src/qat_direct/vfio/adf_pfvf_vf_msg.c \
+	quickassist/lookaside/access_layer/src/qat_direct/vfio/adf_vfio_pf.c
 libadf_la_CFLAGS = -I$(srcdir)/quickassist/utilities/libusdm_drv \
 		   -I$(srcdir)/quickassist/utilities/osal/include \
 		   -I$(srcdir)/quickassist/utilities/osal/src/linux/user_space/include \
@@ -93,7 +94,8 @@ qatmgr_SOURCES = \
 	quickassist/lookaside/access_layer/src/qat_direct/vfio/qat_log.c \
 	quickassist/lookaside/access_layer/src/qat_direct/vfio/adf_pfvf_proto.c \
 	quickassist/lookaside/access_layer/src/qat_direct/vfio/adf_pfvf_vf_msg.c \
-	quickassist/lookaside/access_layer/src/qat_direct/vfio/vfio_lib.c
+	quickassist/lookaside/access_layer/src/qat_direct/vfio/vfio_lib.c \
+	quickassist/lookaside/access_layer/src/qat_direct/vfio/adf_vfio_pf.c
 qatmgr_CFLAGS = -I$(srcdir)/quickassist/lookaside/access_layer/src/qat_direct/vfio \
 	        -I$(srcdir)/quickassist/lookaside/access_layer/include \
 	        -I$(srcdir)/quickassist/include \
@@ -126,7 +128,7 @@ libcrc_SOURCES_DIR = quickassist/lookaside/access_layer/src/common/compression/
 %.lo:
 	@echo "  CCAS     $(libcrc_SOURCES_DIR)$@"
 	@$(LIBTOOL) --mode=compile --quiet \
-	nasm -f elf64 -D LINUX -X gnu $(libcrc_SOURCES_DIR)$(@:.lo=.S) -I$(libcrc_SOURCES_DIR) -o $@ -prefer-non-pic
+	nasm -f elf64 -D LINUX -X gnu $(top_srcdir)/$(libcrc_SOURCES_DIR)$(@:.lo=.S) -I$(top_srcdir)/$(libcrc_SOURCES_DIR) -o $@ -prefer-non-pic
 endif
 
 lib_LTLIBRARIES += lib@LIBQATNAME@.la
@@ -158,6 +160,7 @@ lib@LIBQATNAME@_la_SOURCES = \
 	quickassist/lookaside/access_layer/src/common/crypto/asym/ecc/lac_ec_nist_curves.c \
 	quickassist/lookaside/access_layer/src/common/crypto/asym/ecc/lac_ecdh.c \
 	quickassist/lookaside/access_layer/src/common/crypto/asym/ecc/lac_ecdsa.c \
+	quickassist/lookaside/access_layer/src/common/crypto/asym/ecc/lac_ecsm2.c \
 	quickassist/lookaside/access_layer/src/common/crypto/asym/ecc/lac_kpt_ecdsa.c \
 	quickassist/lookaside/access_layer/src/common/crypto/asym/large_number/lac_ln.c \
 	quickassist/lookaside/access_layer/src/common/crypto/asym/large_number/lac_ln_interface_check.c \
@@ -261,6 +264,7 @@ pkginclude_HEADERS = \
 	quickassist/include/lac/cpa_cy_dsa.h \
 	quickassist/include/lac/cpa_cy_ecdh.h \
 	quickassist/include/lac/cpa_cy_ecdsa.h \
+	quickassist/include/lac/cpa_cy_ecsm2.h \
 	quickassist/include/lac/cpa_cy_ec.h \
 	quickassist/include/lac/cpa_cy_im.h \
 	quickassist/include/lac/cpa_cy_key.h \

diff --git a/README.md b/README.md
@@ -26,6 +26,7 @@
 
 | Date      |     Doc Revision      | Version |   Details |
 |----------|:-------------:|------:|:------|
+| November 2023 | 011 | 23.11 | - Support DC NS (NoSession) APIs.  <br> - Support  DC compressBound APIs. <br> - Support Symmetric Crypto SM3 & SM4. <br> - Support Asymmetric Crypto SM2. <br> - Bug Fixes. See [Resolved Issues](#resolved-issues). |
 | August 2023 | 010 | 23.08 | - Removal of following insecure algorithms: Diffie-Hellman and Elliptic curves less than 256-bits. <br> - Additional configuration profiles, including sym which facilitates improved symmetric crypto performance. <br> - DC Chaining (Hash then compress) <br> - Bug Fixes. See [Resolved Issues](#resolved-issues). |
 | February 2023 | 009 | 23.02 | - Added configuration option --enable-legacy-algorithms to use these insecure crypto algorithms and disabled them by default (AES-ECB, SHA-1, SHA2-224, SHA3-224, RSA512/1024/1536, DSA)<br>- Refactored code in quickassist/utilities/libusdm_drv<br>- Bugfixes<br>- Updated documentation with configuration and tuning information |
 | November 2022 | 008 | 22.07.2 | - Changed from yasm to nasm for assembly compilation<br> - Added configuration option to use C implementation of soft CRC implementation instead of asm<br>- Added support for pkg-config<br>- Added missing lock around accesses to some global data in qatmgr |
@@ -56,9 +57,10 @@ sample codes.
 The following services are available in qatlib via the QuickAssist API:
 * Symmetric (Bulk) Cryptography
   * Ciphers ([AES-ECB](#insecure-algorithms), AES-CBC, AES-CTR (no partials support),
-    AES-XTS (no partials support), AES-GCM, AES-CCM (192/256)
+    AES-XTS (no partials support), AES-GCM, AES-CCM (192/256), [SM4-ECB](#insecure-algorithms),
+    SM4-CBC, SM4-CTR)
   * Message digest/hash ([SHA1](#insecure-algorithms), SHA2 ([224](#insecure-algorithms)/256/384/512),
-    SHA3 ([224](#insecure-algorithms)/256/384/512) (no partials support) and
+    SHA3 ([224](#insecure-algorithms)/256/384/512) (no partials support), SM3) and
     authentication (AES-CBC-MAC, AES-XCBC-MAC)
   * Algorithm chaining (one cipher and one hash in a single operation)
   * Authenticated encryption (CCM-128 (no partials support),
@@ -77,12 +79,15 @@ The following services are available in qatlib via the QuickAssist API:
   * [DSA](#insecure-algorithms) parameter generation and digital signature generation/verification
   * Elliptic Curve Cryptography: ECDSA, ECDHE, Edwards Montgomery curves
   * Generic point multiply
+  * SM2
 * Compression
   * Deflate
   * lz4/lz4s
   * Compress and Verify (CnV)
   * Compress and Verify and Recover (CnVnR)
   * End-to-end (E2E) integrity check
+  * DC compressBound APIs
+  * DC NS (No Session) APIs
 * Compression Chaining (Deflate only)
   * Hash then compress
 
@@ -96,12 +101,13 @@ This package includes:
 The following algorithms are considered insecure and are disabled by default.
 * AES-ECB
 * SHA-1
-* SHA2-224 
+* SHA2-224
 * SHA3-224
 * RSA512/1024/1536
 * DSA
 * Diffie-Helman
 * Elliptic Curve Cryptography algorithms with less 256 bits
+* SM4-ECB
 
 To enable these algorithms, use the following configuration option:
    * `--enable-legacy-algorithms`
@@ -122,6 +128,9 @@ supported.
 ## Limitations
 * If an error occurs on the host driver (Heartbeat, Uncorrectable error) it
   will not be communicated to the library.
+* For simplicity, only one configuration file is used by qatlib. For guidance
+  on how to use this to allocate resources for processes, please refer to
+  Configuration and Tuning section in [QATlib User’s Guide](https://intel.github.io/quickassist/qatlib/index.html).
 
 The following features are not currently supported:
 * Dynamic instances
@@ -147,8 +156,6 @@ The following assumptions are made concerning the deployment environment:
   discovered and initialized the device, exposing the VFs. This driver is
   included in the Linux kernel, see [INSTALL](INSTALL) for information about which kernel
   to use.
-* The library can be used by unprivileged users if that user is included in
-  the 'qat' group.
 
 ## Examples
 Example applications that showcase usage of the QAT APIs are included in the
@@ -180,8 +187,7 @@ where: \<Component\> is one of the following:
 | QATE-41707 | [CY - Incorrect digest returned when performing a plain hash operation on input data of size 4GB or larger.](#qate-41707) |
 | QATE-76073 | [GEN - If PF device configuration is modified without restarting qatmgr, undefined behavior may occur.](#qate-76073) |
 | QATE-76698 | [GEN - Multi-process applications running in guest will fail when running with default Policy settings.](#qate-76698) |
-| QATE-94286 | [CY - Compression services not detected when crypto-capable VFs are added to VM.](#qate-94286) |
-| QATE-94369 | [GEN - SELinux Preventing QAT Service Startup](#qate-94369) |
+| QATE-94369 | [GEN - SELinux Preventing QAT Service Startup.](#qate-94369) |
 
 ## QATE-3241
 | Title      |       CY - cpaCySymPerformOp when used with parameter checking may reveal the amount of padding.        |
@@ -224,20 +230,10 @@ where: \<Component\> is one of the following:
 | Affected OS | Linux |
 | Driver/Module | CPM-IA - General |
 
-## QATE-94286
-| Title      |       GEN - Compression services not detected when crypto-capable VFs are also added to VM.        |
-|----------|:-------------
-| Reference # | QATE-94286 |
-| Description | When configuring a system with different services on different QAT end-points, e.g. asym;sym on one and dc on another, and exposing only one of those Virtual Function (VF) types to the Virtual Machine (VM), the application works as expected. However, when VFs of more than one type are passed to the same VM, the application may only recognize one service-type, e.g. it may detect crypto instances, but not compression instances. There is an assumption that all VFs provide the same services if they come from the same PF. However, detecting which PF they come from is based on domain+bus, which is not always a valid assumption on a VM. |
-| Implication | This issue prevents the detection of compression services in a virtualized environment when the default kernel configuration is used, and crypto and dc VFs are passed to the VM, potentially impacting the proper functioning of the system. |
-| Resolution | When passing VFs to a guest, the BDFs on the guest should facilitate qatlib recognizing whether VFs are from the same PF or not. See RUNNING IN A VIRTUAL MACHINE / GUEST section of INSTALL for details. |
-| Affected OS | Linux |
-| Driver/Module | CPM-IA - General |
-
 ## QATE-94369
 | Title      |       GEN - SELinux Preventing QAT Service Startup        |
 |----------|:-------------
-| Reference # | QATE-94286 |
+| Reference # | QATE-94369 |
 | Description | The qat service fails to start due to SELinux preventing the qat_init.sh script and qatmgr from accessing resources. The issue occurs when the system is running with SELinux enabled, causing insufficient permissions for the qat_init.sh script and qatmgr to function correctly. |
 | Implication | This issue affects the proper functioning of the qat service on systems with SELinux enabled, potentially preventing QAT virtual functions (VFs) from functioning. |
 | Resolution | None available. |
@@ -250,23 +246,45 @@ in this section.
 
 | Issue ID | Description |
 |-------------|------------|
-| QATE-90845 | [GEN - QAT service fails to start, issue #38](#qate-90845) |
+| QATE-94286 | [GEN - Compression services not detected when crypto-capable VFs are added to VM.](#qate-94286) |
+| QATE-95905 | [GEN - Fix build when building outside of main directory, issue #56](#qate-95905) |
+| QATE-93844 | [DC - cpaDcLZ4SCompressBound is not returning correct value, which could lead to a buffer overflow.](#qate-93844)
 | QATE-93278 | [GEN - sample_code potential seg-fault, issue #46](#qate-93278) |
+| QATE-90845 | [GEN - QAT service fails to start, issue #38](#qate-90845) |
+| QATE-78459 | [DC - cpaDcDeflateCompressBound API returns incorrect output buffer size when input size exceeds 477218588 bytes.](#qate-78459) |
 | QATE-76846 | [GEN - Forking and re-initializing use-cases do not work](#qate-76846) |
-| QATE-78459 | [DC - cpaDcDeflateCompressBound API returns incorrect output buffer size when input size exceeds 477218588 bytes.](#qate-74786) |
 | QATE-12241 | [CY - TLS1.2 with secret key lengths greater than 64 are not supported.](#qate-12241) |
 
+## QATE-94286
+| Title      |       GEN - Compression services not detected when crypto-capable VFs are also added to VM.        |
+|----------|:-------------
+| Reference # | QATE-94286 |
+| Description | When configuring a system with different services on different QAT end-points, e.g. asym;sym on one and dc on another, and exposing only one of those Virtual Function (VF) types to the Virtual Machine (VM), the application works as expected. However, when VFs of more than one type are passed to the same VM, the application may only recognize one service-type, e.g. it may detect crypto instances, but not compression instances. There is an assumption that all VFs provide the same services if they come from the same PF. However, detecting which PF they come from is based on domain+bus, which is not always a valid assumption on a VM. |
+| Implication | This issue prevents the detection of compression services in a virtualized environment when the default kernel configuration is used, and crypto and dc VFs are passed to the VM, potentially impacting the proper functioning of the system. |
+| Resolution | Fixed in 23.11. <br>Temporary solution: use a custom libvirt XML file like QATE-76698 here: https://github.com/intel/qatlib/tree/main#qate-76698 . |
+| Affected OS | Linux |
+| Driver/Module | CPM-IA - General |
 
-## QATE-90845
-| Title      |         GEN - QAT service fails to start, issue #38 |
+## QATE-95905
+| Title      |       GEN - Fix build when building outside of main directory, issue #56        |
 |----------|:-------------
-| Reference # | QATE-90845 |
-| Description | QAT service fails to start. The qat service may fail if the kernel driver's initialization is not fully finished when the service starts. See [issue 38](https://github.com/intel/qatlib/issues/38). |
-| Implication | The qatmgr may not detect any or all of the vfio devices. |
-| Resolution | Fixed in 23.08. The service waits until the kernel driver has completed initialization of all PFs before starting the service. |
+| Reference # | QATE-95905 |<F3>
+| Description | Fix build when building outside of main directory. Added changes to autoconfig to be able to build outside main directory. See [issue 56](https://github.com/intel/qatlib/issues/56). |
+| Implication | A fatal error occurs when trying to build outside main directory. |
+| Resolution | Fixed in 23.11. |
 | Affected OS | Linux |
 | Driver/Module | CPM-IA - General |
 
+## QATE-93844
+| Title      |        DC - cpaDcLZ4SCompressBound is not returning correct value, which could lead to a buffer overflow.     |
+|----------|:-------------
+| Reference # | QATE-93844 |
+| Description | CompressBound API (cpaDcLZ4SCompressBound()) is intended to return the maximum size of the output buffer. However, this API is not returning the correct value, which can lead to a lz4s buffer overflow. |
+| Implication | Applications may experience buffer overflows even when using the output of compressBound API to allocate output buffers. |
+| Resolution | Fixed in 23.11 |
+| Affected OS | Linux |
+| Driver/Module | QAT IA - Compression |
+
 ## QATE-93278
 | Title      |         GEN - sample_code potential seg-fault, issue #46     |
 |----------|:-------------
@@ -277,13 +295,13 @@ in this section.
 | Affected OS | Linux |
 | Driver/Module | CPM-IA - General |
 
-## QATE-76846
-| Title      |         GEN - Forking and re-initializing use-cases do not work     |
+## QATE-90845
+| Title      |         GEN - QAT service fails to start, issue #38 |
 |----------|:-------------
-| Reference # | QATE-76846 |
-| Description | Forking and re-initializing use-cases do not work:<br>-icp_sal_userStart()/icp_sal_userStop()/icp_sal_userStart() in single process<br>-icp_sal_userStart()/fork()/icp_sal_userStart() in child.<br> This is the use case in openssh + QAT_Engine. |
-| Implication | The process will have undefined behavior in these use-cases. |
-| Resolution | This issue is resolved with the 21.08 release. If using release prior to this release and using these flows, call qaeMemDestroy() immediately after icp_sal_userStop() to prevent this issue. |
+| Reference # | QATE-90845 |
+| Description | QAT service fails to start. The qat service may fail if the kernel driver's initialization is not fully finished when the service starts. See [issue 38](https://github.com/intel/qatlib/issues/38). |
+| Implication | The qatmgr may not detect any or all of the vfio devices. |
+| Resolution | Fixed in 23.08. The service waits until the kernel driver has completed initialization of all PFs before starting the service. |
 | Affected OS | Linux |
 | Driver/Module | CPM-IA - General |
 
@@ -297,13 +315,23 @@ in this section.
 | Affected OS | Linux |
 | Driver/Module | CPM-IA - Data Compression |
 
+## QATE-76846
+| Title      |         GEN - Forking and re-initializing use-cases do not work     |
+|----------|:-------------
+| Reference # | QATE-76846 |
+| Description | Forking and re-initializing use-cases do not work:<br>-icp_sal_userStart()/icp_sal_userStop()/icp_sal_userStart() in single process<br>-icp_sal_userStart()/fork()/icp_sal_userStart() in child.<br> This is the use case in openssh + QAT_Engine. |
+| Implication | The process will have undefined behavior in these use-cases. |
+| Resolution | Fixed in 21.08. If using release prior to this release and using these flows, call qaeMemDestroy() immediately after icp_sal_userStop() to prevent this issue. |
+| Affected OS | Linux |
+| Driver/Module | CPM-IA - General |
+
 ## QATE-12241
 | Title      |         CY - TLS1.2 with secret key lengths greater than 64 are not supported     |
 |----------|:-------------
 | Reference # | QATE-12241 |
 | Description | Algorithms, as with Diffie-Hellman using 8K parameters that can use a secret key length greater than 64 bytes is not supported.|
 | Implication | Key generation would fail for TLS1.2 algorithms that use more than 64 bytes secret length keys. |
-| Resolution | This is resolved with the 22.07 release. |
+| Resolution | Fixed in 22.07. |
 | Affected OS | Linux |
 | Driver/Module | CPM-IA - Crypto |