Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed bugs with different enum formats, closes #3 #4

Merged
merged 16 commits into from
May 14, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ Supported types and values for the deserialization:
- `char` from a JavaScript string containing a single codepoint.
- `String` from any JavaScript string.
- Rust map (`HashMap`, `BTreeMap`, ...) from any JavaScript iterable producing `[key, value]` pairs (including but not limited to ES2015 `Map`).
> One exception being [internally tagged](https://serde.rs/enum-representations.html#internally-tagged) and [untagged](https://serde.rs/enum-representations.html#untagged) enums. These representations currently do not support deserializing map-like iterables. They only support deserialization from `Object` due to their special treatment in `serde`.
>
> This restriction may be lifted at some point in the future if a `serde(with = ...)` attribute can define the expected Javascript representation of the variant, or if serde-rs/serde#1183 gets resolved.
- `HashMap<String, _>` from any plain JavaScript object (`{ key1: value1, ... }`).
- Rust sequence (tuple, `Vec`, `HashSet`, ...) from any JavaScript iterable (including but not limited to `Array`, ES2015 `Set`, etc.).
- Rust byte buffer (see [`serde_bytes`](https://github.com/serde-rs/bytes)) from JavaScript `ArrayBuffer` or `Uint8Array`.
Expand Down
22 changes: 21 additions & 1 deletion src/de.rs
Original file line number Diff line number Diff line change
Expand Up @@ -236,9 +236,29 @@ impl<'de> de::Deserializer<'de> for Deserializer {
} else if let Some(v) = self.value.as_bool() {
visitor.visit_bool(v)
} else if let Some(v) = self.value.as_f64() {
visitor.visit_f64(v)
if js_sys::Number::is_safe_integer(&self.value) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these conversions covered by new tests somehow?

Generally I'd be wary of extending deserialize_any with casts, and prefer that users specify explicit destination types, unless there is a compelling reason to use dynamic heuristics.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this gets into the weeds of how serde handles internally tagged enums. Basically what happens is serde buffers the variant into an intermediate Content structure. When it encounters a JS number, it stores it as an f64, which ends up breaking deserialization for integer types when it converts this intermediate structure to the one we actually want to deserialize.

The same is true for sequences and objects, which is why I added checks for the Array and Object types.

In the test cases, I test one with type B as a float, and one with type B as an int. I believe this also covers the Object type, since the struct has to be deserialized as a map first, but I'm not certain. We will probably want to add a test for a sequence type (I guess I only verified that this worked in my project).

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ughh this is unfortunate, but makes sense...

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coming back to this - can you please make sure to cover these branches by tests (outside of internally tagged enums)? I want to make sure that assumptions about representation of new types are clearly defined for future regressions. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm running into a few problems with this. I added two variants to the test_enum! macro, making it look like this:

macro_rules! test_enum {
    ($(# $attr:tt)* $name:ident) => {{
        #[derive(Debug, PartialEq, Serialize, Deserialize)]
        $(# $attr)*
        enum $name<A, B> where A: Debug + Hash + Eq {
            Unit,
            Newtype(A),
            Tuple(A, B),
            Struct { a: A, b: B },
            Sequence(Vec<A>),
            Map(HashMap<A, B>),
        }

        test_via_json($name::Unit::<String, i32>);
        test_via_json($name::Newtype::<_, i32>("newtype content".to_string()));
        test_via_json($name::Tuple("tuple content".to_string(), 42));
        test_via_json($name::Struct {
            a: "struct content".to_string(),
            b: 42,
        });
        test_via_json($name::<i32, i32>::Sequence(vec![52, 1, -124, 23, -65]));
        test_via_json($name::<String, i32>::Map(
            vec![("a".to_string(), 12), ("abc".to_string(), -1161), ("b".to_string(), 64)].into_iter().collect()
        ));
    }};
}

This should at least touch on some sequences and maps in non-internally tagged enums. However, I ran into the following error with this setup:

failures:

---- serde::enums output ----
    error output:
        panicked at 'assertion failed: `(left == right)`
          left: `"{\"Map\":{}}"`,
         right: `"{\"Map\":{\"abc\":-1161,\"b\":64,\"a\":12}}"`', tests/serde.rs:37:9
        
        Stack:
        
        Error
            at module.exports.__wbg_new_59cb74e423758ede (serde-wasm-bindgen/target/wasm32-unknown-unknown/wbg-tmp/wasm-bindgen-test.js:524:21)
            at console_error_panic_hook::Error::new::h274f015c600dfb3e (wasm-function[6169]:0x2018c8)
            at console_error_panic_hook::hook_impl::h3ca87fb44cb9eea6 (wasm-function[1239]:0x151671)
            at console_error_panic_hook::hook::h2b7999185aaf7486 (wasm-function[6797]:0x20b038)
            at core::ops::function::Fn::call::hc246242b419002f9 (wasm-function[6193]:0x201f6b)
            at std::panicking::rust_panic_with_hook::h8540df5fb5cc4da7 (wasm-function[1847]:0x181b27)
            at std::panicking::continue_panic_fmt::h3082507e3bf1767e (wasm-function[3830]:0x1cf7a9)
            at std::panicking::begin_panic_fmt::hd9925b25492d4f84 (wasm-function[6026]:0x1ff2bf)
            at serde::assert_json::h2af830e0b2abbb33 (wasm-function[186]:0x6ba78)
            at serde::test_via_json::hd30427c9dd2fd6e6 (wasm-function[1698]:0x1777a1)
        
        
    
    JS exception that was thrown:
        RuntimeError: unreachable
            at __rust_start_panic (wasm-function[7985]:0x2160dc)
            at rust_panic (wasm-function[7342]:0x211ef4)
            at std::panicking::rust_panic_with_hook::h8540df5fb5cc4da7 (wasm-function[1847]:0x181b4e)
            at std::panicking::continue_panic_fmt::h3082507e3bf1767e (wasm-function[3830]:0x1cf7a9)
            at std::panicking::begin_panic_fmt::hd9925b25492d4f84 (wasm-function[6026]:0x1ff2bf)
            at serde::assert_json::h2af830e0b2abbb33 (wasm-function[186]:0x6ba78)
            at serde::test_via_json::hd30427c9dd2fd6e6 (wasm-function[1698]:0x1777a1)
            at serde::enums::h8c39b7b5568167b3 (wasm-function[89]:0x1eb44)
            at core::ops::function::FnOnce::call_once::hf429f4f99c997298 (wasm-function[7207]:0x21078e)
            at wasm_bindgen_test::__rt::Context::execute_sync::{{closure}}::hc76b1a4e656b2ee3 (wasm-function[5241]:0x1f0cde)

failures:

    serde::enums

test result: FAILED. 10 passed; 1 failed; 0 ignored

Not sure why the HashMap is being skipped over here, but it appears to be another bug (unless I'm missing something). Note that serializing structs works perfectly fine, but not HashMap, which I believe works fine for serde_json.

Another thing I saw was due to the way assert_json checks the serialization results. When I add another test case to test_enum! for sequences of floats:

test_via_json(InternallyTagged::<f32, ()>::Sequence(vec![12.4, 11.0, -124.5, 232.2, -65.56]));

I get this error due to floating points not being serialized exactly the same by serde_json and js_sys::JSON:

---- serde::enums output ----
    error output:
        panicked at 'assertion failed: `(left == right)`
          left: `"{\"Sequence\":[12.399999618530273,11,-124.5,232.1999969482422,-65.55999755859375]}"`,
         right: `"{\"Sequence\":[12.4,11.0,-124.5,232.2,-65.56]}"`', tests/serde.rs:37:9
...

This indicates that we need a more robust method of asserting that values are serialized and deserialized properly in order to provide more comprehensive coverage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So after looking into df09c8b, I'm a little unclear as to why iterables get excluded there. I agree that we should have a serialization switch for Map vs Object, but if we don't at the moment, then isn't it ok just to allow deserialization from iterables?

It might not be the best policy, but it wouldn't be unreasonable until a serde(with = ...) switch is implemented or serde-rs/serde#1183 gets resolved in my opinion. At the very least, it might save someone a gotcha if they just naively replace their serde_json code with serde_wasm_bindgen.

I suspended that changeset and ran this test and it seemed to do fine:

assert_eq!(
    InternallyTagged::Map(
        vec![
            ("a".to_string(), 12), 
            ("abc".to_string(), -1161), 
            ("b".to_string(), 64)
        ].into_iter().collect()
    ), 
    from_value::<InternallyTagged<String, i32>>(
        js_sys::eval(r"
            var m = new Map(); 
            m.set('tag', 'Map'); 
            m.set('a', 12);
            m.set('abc', -1161);
            m.set('b', 64);
            m
        ").unwrap()
    ).unwrap()
);

I'll upload the deserialization test and the README change so you can look at it, though.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My hesitation with allowing such conversions that is that someone will most likely start relying on that behaviour, and removing it afterwards in a favour of "the right way" will be a breaking change.

I'd rather be as restrictive as possible and throw an explicit error, until someone comes up with a use-case where this is important, rather than silently allow some conversions through and deal with the challenge of deprecating them later.

Copy link
Contributor Author

@awestlake87 awestlake87 May 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thing to consider is that excluding Iterables also affects untagged enums since they go through deserialize_any as well. So this test fails with df09c8b as well:

    #[derive(Debug, PartialEq, Serialize, Deserialize)]
    #[serde(untagged)]
    enum Untagged<A, B>
    where
        A: Ord,
    {
        Unit,
        Sequence { seq: Vec<A> },
        Map(BTreeMap<A, B>),
    }
    assert_eq!(
        Untagged::Map(
            vec![
                ("a".to_string(), 12), 
                ("abc".to_string(), -1161), 
                ("b".to_string(), 64)
            ].into_iter().collect()
        ), 
        from_value::<Untagged<String, i32>>(
            js_sys::eval(r"
                var m = new Map(); 
                m.set('a', 12);
                m.set('abc', -1161);
                m.set('b', 64);
                m
            ").unwrap()
        ).unwrap()
    );

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oof and basically maps at any level in the variant. This also fails:

    #[derive(Debug, PartialEq, Serialize, Deserialize)]
    #[serde(tag = "tag")]
    enum InternallyTagged<A, B>
    where
        A: Ord,
    {
        Unit,
        Struct { a: A, b: B },
        Sequence { seq: Vec<A> },
        Map(BTreeMap<A, B>),
        DeeperMap {
            map: BTreeMap<A, B>,
        }
    }

    assert_eq!(
        InternallyTagged::DeeperMap {
            map: vec![
                ("a".to_string(), 12), 
                ("abc".to_string(), -1161), 
                ("b".to_string(), 64)
            ].into_iter().collect()
        }, 
        from_value::<InternallyTagged<String, i32>>(
            js_sys::eval(r"
                var m = new Map(); 
                m.set('a', 12);
                m.set('abc', -1161);
                m.set('b', 64);
            
                ({ 'tag': 'DeeperMap', 'map': m })
            ").unwrap()
        ).unwrap()
    );

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I uploaded 793f42b with the changes to tests and an update to the README. I think this should cover what we talked about before I brought up df09c8b, but let me know if you think we should add some more info or tests.

visitor.visit_i64(v as i64)
} else {
visitor.visit_f64(v)
}
} else if let Some(v) = self.value.as_string() {
visitor.visit_string(v)
} else if js_sys::Array::is_array(&self.value) {
self.deserialize_seq(visitor)
} else if self.value.is_object() &&
// The only reason we want to support objects here is because serde uses
// `deserialize_any` for internally tagged enums
// (see https://github.com/cloudflare/serde-wasm-bindgen/pull/4#discussion_r352245020).
//
// We expect such enums to be represented via plain JS objects, so let's explicitly
// exclude Sets, Maps and any other iterables. These should be deserialized via concrete
// `deserialize_*` methods instead of us trying to guess the right target type.
//
// Hopefully we can rid of these hacks altogether once
// https://github.com/serde-rs/serde/issues/1183 is implemented / fixed on serde side.
!js_sys::Reflect::has(&self.value, &js_sys::Symbol::iterator()).unwrap_or(false)
{
self.deserialize_map(visitor)
} else {
self.invalid_type(visitor)
}
Expand Down
111 changes: 100 additions & 11 deletions tests/serde.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
use js_sys::Reflect;
use serde::de::DeserializeOwned;
use serde::{Deserialize, Serialize};
use serde_wasm_bindgen::{from_value, to_value};
use std::collections::{HashMap, HashSet};
use std::fmt::Debug;
use std::{
collections::{BTreeMap, HashMap, HashSet},
hash::Hash,
};
use wasm_bindgen::{JsCast, JsValue};
use wasm_bindgen_test::*;

Expand All @@ -24,14 +28,48 @@ where
test(value, value);
}

fn recurse_and_replace_maps(val: JsValue) -> Option<JsValue> {
if val.is_object() {
let obj = if js_sys::Map::instanceof(&val) {
js_sys::Object::from_entries(&js_sys::Array::from(&val)).unwrap()
} else {
val.unchecked_into()
};

for key in js_sys::Object::keys(&obj).values() {
let key = key.unwrap();
let val = Reflect::get(&obj, &key).unwrap();

if let Some(replacement) = recurse_and_replace_maps(val) {
Reflect::set(&obj, &key, &replacement).unwrap();
}
}

Some(JsValue::from(obj))
} else {
None
}
}

fn assert_json<R>(lhs_value: JsValue, rhs: R)
where
R: Serialize + DeserializeOwned + PartialEq + Debug,
{
assert_eq!(
js_sys::JSON::stringify(&lhs_value).unwrap(),
serde_json::to_string(&rhs).unwrap(),
);
let lhs_value = if let Some(replacement) = recurse_and_replace_maps(lhs_value.clone()) {
replacement
} else {
lhs_value
};

if lhs_value.is_undefined() {
assert_eq!("null", serde_json::to_string(&rhs).unwrap())
} else {
assert_eq!(
js_sys::JSON::stringify(&lhs_value).unwrap(),
serde_json::to_string(&rhs).unwrap(),
);
}

let restored_lhs: R = from_value(lhs_value.clone()).unwrap();
assert_eq!(restored_lhs, rhs, "from_value from {:?}", lhs_value);
}
Expand Down Expand Up @@ -78,20 +116,31 @@ macro_rules! test_float {
macro_rules! test_enum {
($(# $attr:tt)* $name:ident) => {{
#[derive(Debug, PartialEq, Serialize, Deserialize)]
enum $name<A, B> {
$(# $attr)*
enum $name<A, B> where A: Debug + Ord + Eq {
Unit,
Newtype(A),
Tuple(A, B),
Struct { a: A, b: B },
Map(BTreeMap<A, B>),
Seq { seq: Vec<B> } // internal tags cannot be directly embedded in arrays
}

test_via_json($name::Unit::<(), ()>);
test_via_json($name::Newtype::<_, ()>("newtype content".to_string()));
test_via_json($name::Unit::<String, i32>);
test_via_json($name::Newtype::<_, i32>("newtype content".to_string()));
test_via_json($name::Tuple("tuple content".to_string(), 42));
test_via_json($name::Struct {
a: "struct content".to_string(),
b: 42,
});
test_via_json($name::Map::<String, i32>(
vec![
("a".to_string(), 12),
("abc".to_string(), -1161),
("b".to_string(), 64)
].into_iter().collect()
));
test_via_json($name::Seq::<i32, i32> { seq: vec![5, 63, 0, -62, 6] });
}};
}

Expand Down Expand Up @@ -214,10 +263,50 @@ fn enums() {
test_enum! {
ExternallyTagged
}
test_enum! {
#[serde(tag = "tag")]
InternallyTagged

#[derive(Debug, PartialEq, Serialize, Deserialize)]
#[serde(tag = "tag")]
enum InternallyTagged<A, B>
where
A: Ord,
{
Unit,
Struct { a: A, b: B },
Sequence { seq: Vec<A> },
Map(BTreeMap<A, B>)
}

test_via_json(InternallyTagged::Unit::<(), ()>);
RReverser marked this conversation as resolved.
Show resolved Hide resolved
test_via_json(InternallyTagged::Struct {
a: "struct content".to_string(),
b: 42,
});
test_via_json(InternallyTagged::Struct {
a: "struct content".to_string(),
b: 42.2,
});
test_via_json(InternallyTagged::<i32, ()>::Sequence {
seq: vec![12, 41, -11, -65, 961],
});


// Internal tags with maps are not properly deserialized from Map values due to the exclusion
// of Iterables during deserialize_any(). They can be deserialized properly from plain objects
// so we can test that.
assert_eq!(
InternallyTagged::Map(
vec![
("a".to_string(), 12),
("abc".to_string(), -1161),
("b".to_string(), 64)
].into_iter().collect()
),
from_value::<InternallyTagged<String, i32>>(
js_sys::eval("({ 'tag': 'Map', 'a': 12, 'abc': -1161, 'b': 64 })").unwrap()
).unwrap()
);


test_enum! {
#[serde(tag = "tag", content = "content")]
AdjacentlyTagged
Expand Down