Dynamic deserialization proposal #861

addisoncrump · 2022-10-24T13:42:09Z

Using downcast-rs, inventory, and named serialization.

addisoncrump · 2022-10-24T13:42:58Z

libafl/src/inputs/bytes.rs

+ fn deserialize_generalised_to_bytes() {
+ let generalised = GeneralizedInput::new(b"hello".to_vec());
+ let mut buf = Vec::new();
+ generalised.serialize_dynamic(&mut buf).unwrap();
+ let bytes = BytesInput::deserialize_dynamic(&buf).unwrap().unwrap();
+ assert_eq!(bytes.target_bytes().as_slice(), b"hello");
+ }


Example usage of the dynamic deserialisation. #858

addisoncrump · 2022-10-24T13:52:57Z

libafl/src/inputs/bytes.rs

+ fn failed_deserialize_from_nop() {
+ // note that NopInput implements HasTargetBytes, but because we have not submitted the
+ // conversion BytesInput cannot be converted from NopInput
+
+ let nop = NopInput {};
+ let mut buf = Vec::new();
+ nop.serialize_dynamic(&mut buf).unwrap();
+ assert!(BytesInput::deserialize_dynamic(&buf).unwrap().is_none());
+ }


See comments here: if the deserialisation for this type cannot occur, it is skipped and not deserialised.

andreafioraldi · 2022-10-24T13:56:25Z

libafl/src/inputs/bytes.rs

@@ -24,6 +27,8 @@ pub struct BytesInput {
 }

 impl Input for BytesInput {
+ const NAME: &'static str = "BytesInput";


I think here type_name can be used instead of an associated const https://doc.rust-lang.org/std/any/fn.type_name.html

Oh, awesome -- didn't know about that. I'll patch it out.

Actually, this needs to be an associated const because type_name is not const stable.

I think it will be stabilized soon, see rust-lang/rust#63084

So keep the const for now but put a comment to remind us to switch to type_name once it is stable

The exact contents and format of the string returned are not specified [...] amongst the strings that type_name::<Option<String>>() might return are "Option<String>" and "std::option::Option<std::string::String>".

Could be an issue, probably a NAME field is safer

maye it should be called TYPE else it's confusing with our named trait(?)

libafl/src/inputs/bytes.rs

domenukk · 2022-10-24T14:31:28Z

libafl/src/inputs/mod.rs

 /// Generate a name for this input
 fn generate_name(&self, idx: usize) -> String;

 /// An hook executed if the input is stored as `Testcase`
 fn wrapped_as_testcase(&mut self) {}
 }

+/// Utility trait for downcasting inputs for conversion
+#[cfg(feature = "std")]


the libs seem to support no_std

still inventory doesn't so this stuff is anyway std-only

https://github.com/dtolnay/inventory/blob/969a8994fc25519cabb698687d9289e1c8dd4a85/src/lib.rs#L106

why not

Ah it uses ctor I guess. Nevermind

So in theory yes, it will work for an ELF with no_std for instance but not for no_std in general. Maybe we should use the ctor feature here to enable it in specific no_std cases.

Now you can enable it on no_std with the input_conversion feature, on by default with std

Also our registry will check now for ctor and not std, so we can enable ctor on no_std when supported

andreafioraldi · 2022-10-24T15:25:20Z

Here the issue is for some types serialization and deserialization to other types can be complex. For instance, NautilusInput to serialize to bytes (unparse method) needs a reference to NautilusContext.

andreafioraldi · 2022-10-24T15:38:14Z

Btw downcast-rs is interesting, I was not aware of it, but it seems to bound the type to Any https://github.com/marcianx/downcast-rs/blob/master/src/lib.rs#L175 and so I think it cannot work with non 'static types (?), we should try with a minimal example, atm it may just work because all the inputs we have are static too

addisoncrump · 2022-10-24T15:39:31Z

Btw downcast-rs is interesting, I was not aware of it, but it seems to bound the type to Any https://github.com/marcianx/downcast-rs/blob/master/src/lib.rs#L175 and so I think it cannot work with non 'static types (?), we should try with a minimal example, atm it may just work because all the inputs we have are static too

My example above shows that the requirement for 'static is avoided.

addisoncrump · 2022-10-24T15:43:40Z

Here the issue is for some types serialization and deserialization to other types can be complex. For instance, NautilusInput to serialize to bytes (unparse method) needs a reference to NautilusContext.

This can be specialised by the implementation, if you're comfortable with passing around a NautilusContext 😁

domenukk · 2022-10-24T15:44:41Z

Like an AnyMap for state?

addisoncrump · 2022-10-24T15:47:43Z

libafl/src/inputs/mod.rs

+ fn serialize_dynamic(&self, buf: &mut Vec<u8>) -> Result<(), postcard::Error> {
+ buf.extend_from_slice(postcard::to_allocvec(Self::NAME)?.as_slice());
+ buf.extend_from_slice(postcard::to_allocvec(self)?.as_slice());
+ Ok(())
+ }
+
+ /// Deserializes this input type from the dynamic serialization format, if possible
+ fn deserialize_dynamic(
+ buf: &[u8],
+ ) -> Result<Option<Self>, <&mut Deserializer<Slice> as serde::de::Deserializer>::Error> {
+ convert_named(buf)
+ }


These two methods can be overriden, so it's possible to serialize context and pass it at the end of buf here; just append the context to the end of the message. If the target already has it, it can opt to simply not deserialize it.

I'm not sure how big context is, but if it's reasonably sized this could be effective.

context doesn't need to be serialized, but it is needed to do the serialization as it contains some info to convert the AST to bytes

How does that work with LLMP now? I don't see its unparse getting invoked except in specific places.

It doesn't. To serialize with serde there is no need of it, while it is needed to convert the AST stored in the input to bytes. You see it only in the harness.

Also EncodedInput has something similar, as it is an array of u32 id representing tokens and the mapping id -> token is ofc not stored in the input itself so to convert to bytes you need to call https://github.com/AFLplusplus/LibAFL/blob/main/libafl/src/inputs/encoded.rs#L75

So the conversion would have to happen at the sending client, not the receiving?

But I see why it is convenient to do in the receiver, we can reuse the NewTestcase event and avoid double memory usage

I have to think about it, but I don't see other solutions, the tokens map in the EncodedInput decoder can be large even gigabytes when using a huge initial corpus

addisoncrump · 2022-10-26T19:19:01Z

So what's the verdict? Are we sticking with this approach or will we use something else? (e.g., making the clients aware of local transformations they need to complete first?)

andreafioraldi · 2022-10-27T08:20:59Z

I'm thinking about an additional field to NewTestcase

something like

NewTestcase {
     input_type_name: String,
     serialized_input: Option<Vec<u8>>,
     bytes_input: Option<Vec<u8>>,
     ...
}

Then the event manager can take two optional parsing and unparsing lambdas to convert an input to bytes and back.
When the event is received, we compare the type name and if it match serialized_input it is deserialized, like we already do.
If not, and if bytes_input is not None, we invoke the parsing lambda if present.

domenukk · 2022-11-17T10:53:03Z

Any decisions @andreafioraldi ?

domenukk · 2022-12-17T01:01:54Z

Do we have a verdict here? @andreafioraldi ?

addisoncrump · 2023-01-12T19:48:16Z

Closing in favour of #997

dynamic deserialisation example

1727b00

addisoncrump requested a review from andreafioraldi October 24, 2022 13:42

addisoncrump commented Oct 24, 2022

View reviewed changes

add failing case

d3ba8ae

addisoncrump commented Oct 24, 2022

View reviewed changes

andreafioraldi reviewed Oct 24, 2022

View reviewed changes

libafl/src/inputs/bytes.rs Outdated Show resolved Hide resolved

fix box import

e3f6098

domenukk reviewed Oct 24, 2022

View reviewed changes

input_conversion feature

d3521e1

addisoncrump commented Oct 24, 2022

View reviewed changes

prevent the need for a self-translation if from is to

7155437

Merge branch 'main' into dynamic_deserialisation

cda248b

addisoncrump closed this Jan 12, 2023

domenukk deleted the dynamic_deserialisation branch March 18, 2024 07:45

Dynamic deserialization proposal #861

Dynamic deserialization proposal #861

Conversation

addisoncrump commented Oct 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domenukk Oct 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreafioraldi commented Oct 24, 2022

andreafioraldi commented Oct 24, 2022

addisoncrump commented Oct 24, 2022

addisoncrump commented Oct 24, 2022

domenukk commented Oct 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addisoncrump commented Oct 26, 2022

andreafioraldi commented Oct 27, 2022

domenukk commented Nov 17, 2022

domenukk commented Dec 17, 2022

addisoncrump commented Jan 12, 2023

domenukk Oct 24, 2022 •

edited

Loading