transform: luaxform transform script

doc/userguide/configuration/suricata-yaml.rst

@@ -2861,8 +2861,9 @@ Lua
 ~~~
 
 Suricata 8.0 sandboxes Lua rules by default. The restrictions on the sandbox for Lua rules can be
-modified in the ``security.lua`` section of the configuration file.  Additionally, Lua rules 
-can be completely disabled the same as the Suricata 7.0 default:
+modified in the ``security.lua`` section of the configuration file. This section also applies to
+Lua transforms. Additionally, Lua rules can be completely disabled in the same way as for as the
+Suricata 7.0 default:
 
 ::
 

doc/userguide/lua/lua-functions.rst

@@ -7,6 +7,7 @@ Differences between `output` and `detect`:
 ------------------------------------------
 
 Currently, the ``needs`` key initialization varies, depending on what is the goal of the script: output or detection.
+The Lua script for the ``luaxform`` transform **does not use ``needs``**.
 
 If the script is for detection, the ``needs`` initialization should be as seen in the example below (see :ref:`lua-detection` for a complete example of a detection script):
 
@@ -812,7 +813,7 @@ Example:
           return 0
       end
   end
-  
+
 HasshServerGet
 ~~~~~~~~~~~~~~
 
@@ -828,7 +829,7 @@ Example:
           return 0
       end
   end
-  
+
 HasshServerGetString
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -998,7 +999,7 @@ index so in our case we need to use 0.
          SCFlowintSet(0, a + 1)
      else
          SCFlowintSet(0, 1)
-     end 
+     end
 
 SCFlowintGet
 ~~~~~~~~~~~~
@@ -1031,7 +1032,7 @@ SCFlowvarSet
 Set a Flowvar. First parameter is the index, second is the data
 and third is the length of data.
 
-You can use it to set string 
+You can use it to set string
 
 ::
 
@@ -1041,7 +1042,7 @@ You can use it to set string
      needs["flowvar"] = {"cnt"}
      return needs
  end
- 
+
  function match(args)
      a = SCFlowvarGet(0);
      if a then
@@ -1050,7 +1051,7 @@ You can use it to set string
      else
          a = tostring(1)
          SCFlowvarSet(0, a, #a)
-     end 
+     end
 
 Misc
 ----

doc/userguide/lua/lua-usage.rst

@@ -1,20 +1,29 @@
 Lua usage in Suricata
 =====================
 
-Lua scripting can be used in two components of Suricata. The first is in
-output and the second one in rules in the detection engine.
+Lua scripting can be used in two components of Suricata:
+
+  * Output
+  * Detection: ``lua`` keyword and ``luaxform`` transform
 
 Both features are using a list of functions to access the data extracted by
 Suricata. You can get the list of functions in the :ref:`lua-functions` page.
 
-.. note:: Currently, there is a difference in the ``needs`` key in the ``init`` function, depending on what is the usage: ``output`` or ``detection``. The list of available functions may also differ.
+.. note:: Currently, there is a difference in the ``needs`` key in the ``init`` function,
+   depending on what is the usage: ``output`` or ``detection``. The list of available
+   functions may also differ. The ``luaxform`` doesn't use the ``needs`` key.
 
 Lua output
 ----------
 
-Lua can be used to write arbitrary output. See :ref:`lua-output` for more information.
+Lua scripts can be used to write arbitrary output. See :ref:`lua-output` for more information.
 
 Lua detection
 -------------
 
-Lua script can be used as a filter condition in signatures. See :ref:`lua-detection` for more information.
+Lua scripts can be used as a filter condition in signatures. See :ref:`lua-detection` for more information.
+
+Lua transform
+-------------
+
+The ``luaxform`` transform can be used in signatures. See :ref:`lua-transform` for more information.

doc/userguide/rules/lua-detection.rst

@@ -3,10 +3,18 @@
 Lua Scripting for Detection
 ===========================
 
+There are 2 ways that Lua can be used with detection. These are
+
+* ``lua`` rule keyword.
+* ``luaxform`` transform.
+
 .. note:: Lua is disabled by default for use in rules, it must be
           enabled in the configuration file. See the ``security.lua``
           section of ``suricata.yaml`` and enable ``allow-rules``.
 
+Lua Rule Keyword
+----------------
+
 Syntax:
 
 ::
@@ -103,8 +111,13 @@ Entire script:
 
   return 0
 
-Sandbox and Available functions
--------------------------------
+Lua Transform: ``luaxform``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+More details in :ref:`lua-transform`.
+
+Lua Sandbox and Available functions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Lua rule scripts are run in a sandbox environment the applies the
 following restrictions:
@@ -140,7 +153,7 @@ Of note, the following standard libraries are not available:
 
 This behavior can be modified via the ``security.lua`` section of :ref:`suricata-yaml-lua-config`
 
-.. note:: Suricata 8.0 has moved to Lua 5.4 and has builtin support for bitwise and utf8 operations now.
+.. note:: Suricata 8.0 has moved to Lua 5.4 and now has builtin support for bitwise and utf8 operations.
 
 A comprehensive list of existing lua functions - with examples - can
 be found at :ref:`lua-functions` (some of them, however, work only for

doc/userguide/rules/transforms.rst

@@ -243,3 +243,112 @@ This example transforms `"Zm 9v Ym Fy"` to `"foobar"`::
 
        content:"/?arg=Zm 9v Ym Fy"; from_base64: offset 6, mode rfc2045; \
        content:"foobar";
+
+.. _lua-transform:
+
+luaxform
+--------
+
+This transform allows a Lua script to apply a transformation
+to a buffer.
+
+Lua scripts that are used for transformations *must* contain a function
+named ``transform``.
+
+Lua transforms can be passed optional arguments -- see the examples below -- but they
+are not required to do so. Arguments are comma-separated.
+
+A Lua transform function is not invoked if the buffer is empty or the Lua framework is
+not accessible (rare).
+
+Lua transform functions must return two values (see below) or the buffer is not modified.
+
+Note that the arguments and values are passed without validation
+nor interpretation. There is a maximum of 10 arguments.
+
+The Lua transform function is invoked with these parameters:
+
+  * `input-length` The number of bytes in the buffer provided to the transform
+  * `input` The buffer provided to the transform
+  * `argument` The number of arguments provided in the following parameters. If there are
+    no arguments to the Lua transform, this value will be `0`.
+  * `arguments` The list of arguments.
+
+Lua transform functions must return two values [Lua datatypes shown]:
+
+  * `buffer` [Lua string] The return buffer containing the original input buffer or buffer modified by the transform.
+  * `bytes` [Lua integer] Number of bytes in return buffer.
+
+This example supplies the HTTP data to a Lua transform and the transform
+results are checked with `content`.
+
+Example::
+
+    alert http any any -> any any (msg:"Lua Xform example"; flow:established;  \
+    file.data; luaxform:./lua/lua-transform.lua; content: "abc"; sid: 2;)
+
+
+This example supplies the HTTP data to a Lua transform with with arguments
+that specify the offset and byte count for the transform. The resulting
+buffer is then checked with a `content` match.
+
+Example::
+
+    alert http any any -> any any (msg:"Lua Xform example"; flow:established; \
+    file.data; luaxform:./lua/lua-transform.lua, bytes 12, offset 13; content: "abc"; sid: 1;)
+
+
+The following Lua script shows a transform that handles arguments: `bytes` and `offset` and uses
+those values (or defaults, if there are no arguments) for applying the uppercase transform to
+the buffer.
+
+.. code-block:: lua
+
+   function init (args)
+        local needs = {}
+        return needs
+   end
+
+   local function get_value(item, key)
+       if string.find(item, key) then
+           local _, value = string.match(item, "(%a+)%s*(%d*)")
+           if value ~= "" then
+               return tonumber(value)
+           end
+       end
+
+       return nil
+   end
+
+   -- Arguments supported
+   local bytes_key = "bytes"
+   local offset_key = "offset"
+   function transform(input_len, input, argc, args)
+       local bytes = input_len
+       local offset = 0
+
+       -- Look for optional bytes and offset arguments
+       for i, item in ipairs(args) do
+           local value = get_value(item, bytes_key)
+           if value ~= nil then
+               bytes = value
+           else
+               local value = get_value(item, offset_key)
+               if value ~= nil then
+                   offset = value
+               end
+           end
+       end
+       local str_len = #input
+       if offset < 0 or offset > str_len then
+           print("offset is out of bounds: " .. offset)
+           return nil
+       end
+       str_len = str_len - offset
+       if bytes < 0 or bytes > str_len then
+           print("invalid bytes " ..  bytes .. " or bytes > length " .. bytes .. " length " .. str_len)
+           return nil
+       end
+       local sub = string.sub(input, offset + 1, offset + bytes)
+       return string.upper(sub), bytes
+   end

doc/userguide/upgrade.rst

@@ -82,6 +82,8 @@ Major changes
 - Unknown requirements in the ``requires`` keyword will now be treated
   as unmet requirements, causing the rule to not be loaded. See
   :ref:`keyword_requires`.
+- New transform ``luaxform`` that uses a Lua script for sticky buffer transformation.
+  More details in :ref:`lua-transform`.
 
 Removals
 ~~~~~~~~

rust/src/detect/transforms/casechange.rs

@@ -41,7 +41,7 @@ fn tolower_transform_do(input: &[u8], output: &mut [u8]) {
 }
 
 #[no_mangle]
-unsafe extern "C" fn tolower_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn tolower_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {
@@ -106,7 +106,7 @@ fn toupper_transform_do(input: &[u8], output: &mut [u8]) {
 }
 
 #[no_mangle]
-unsafe extern "C" fn toupper_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn toupper_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {

rust/src/detect/transforms/compress_whitespace.rs

@@ -51,7 +51,7 @@ fn compress_whitespace_transform_do(input: &[u8], output: &mut [u8]) -> u32 {
 }
 
 #[no_mangle]
-unsafe extern "C" fn compress_whitespace_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn compress_whitespace_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {

rust/src/detect/transforms/dotprefix.rs

@@ -43,7 +43,7 @@ fn dot_prefix_transform_do(input: &[u8], output: &mut [u8]) {
 }
 
 #[no_mangle]
-unsafe extern "C" fn dot_prefix_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn dot_prefix_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input_len = InspectionBufferLength(buffer);
     if input_len == 0 {
         return;

rust/src/detect/transforms/hash.rs

@@ -51,7 +51,7 @@ fn md5_transform_do(input: &[u8], output: &mut [u8]) {
 }
 
 #[no_mangle]
-unsafe extern "C" fn md5_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn md5_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {
@@ -107,7 +107,7 @@ fn sha1_transform_do(input: &[u8], output: &mut [u8]) {
 }
 
 #[no_mangle]
-unsafe extern "C" fn sha1_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn sha1_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {
@@ -163,7 +163,7 @@ fn sha256_transform_do(input: &[u8], output: &mut [u8]) {
 }
 
 #[no_mangle]
-unsafe extern "C" fn sha256_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn sha256_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {

rust/src/detect/transforms/http_headers.rs

@@ -54,7 +54,7 @@ fn header_lowertransform_do(input: &[u8], output: &mut [u8]) {
 }
 
 #[no_mangle]
-unsafe extern "C" fn header_lowertransform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn header_lowertransform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {
@@ -120,7 +120,7 @@ fn strip_pseudo_transform_do(input: &[u8], output: &mut [u8]) -> u32 {
 }
 
 #[no_mangle]
-unsafe extern "C" fn strip_pseudo_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn strip_pseudo_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {

rust/src/detect/transforms/mod.rs

@@ -37,7 +37,7 @@ pub struct SCTransformTableElmt {
     pub flags: u16,
     pub Setup: unsafe extern "C" fn(de: *mut c_void, s: *mut c_void, raw: *const c_char) -> c_int,
     pub Free: Option<unsafe extern "C" fn(de: *mut c_void, ptr: *mut c_void)>,
-    pub Transform: unsafe extern "C" fn(inspect_buf: *mut c_void, options: *mut c_void),
+    pub Transform: unsafe extern "C" fn(det: *mut c_void, inspect_buf: *mut c_void, options: *mut c_void),
     pub TransformValidate:
         Option<unsafe extern "C" fn(content: *const u8, len: u16, context: *mut c_void) -> bool>,
 }

rust/src/detect/transforms/strip_whitespace.rs

@@ -48,7 +48,7 @@ fn strip_whitespace_transform_do(input: &[u8], output: &mut [u8]) -> u32 {
 }
 
 #[no_mangle]
-unsafe extern "C" fn strip_whitespace_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn strip_whitespace_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {

rust/src/detect/transforms/urldecode.rs

@@ -88,7 +88,7 @@ fn url_decode_transform_do(input: &[u8], output: &mut [u8]) -> u32 {
 }
 
 #[no_mangle]
-unsafe extern "C" fn url_decode_transform(buffer: *mut c_void, _ctx: *mut c_void) {
+unsafe extern "C" fn url_decode_transform(_det: *mut c_void, buffer: *mut c_void, _ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {

rust/src/detect/transforms/xor.rs

@@ -82,7 +82,7 @@ fn xor_transform_do(input: &[u8], output: &mut [u8], ctx: &DetectTransformXorDat
 }
 
 #[no_mangle]
-unsafe extern "C" fn xor_transform(buffer: *mut c_void, ctx: *mut c_void) {
+unsafe extern "C" fn xor_transform(_det: *mut c_void, buffer: *mut c_void, ctx: *mut c_void) {
     let input = InspectionBufferPtr(buffer);
     let input_len = InspectionBufferLength(buffer);
     if input.is_null() || input_len == 0 {