Different values of pack

[ahmads-dy]

Hi, I am migrating from msgpack-node to msgpackr.

Packing this object causing different values but unpacking them causes same value.
I am not familiar with how Msgpack works, can you explain, please?

const msgpack_old = require('msgpack');
const {Unpackr, Packr} = require("msgpackr");
const input = [
        {
            type: 'CATEGORY',
            location: 'https://shop.website.com/men-pants/p7383723-010',
            data: [],
            referrer: 'www.referrer.com'
        },
        12042,
        31531,
        false,
        0,
        '1201123-007',
        1
    ]
;

let packed = packer.pack(input).toString('base64');
let old_packed = msgpack_old.pack(input).toString('base64');

console.log(packed);
console.log(old_packed);

[kriszyp]

Yes, when you use the Packr constructor to create a encoder/decoder instance via new Packr this defaults to enabling the record extension, which for larger objects should be more compact and faster. However, for compatibility with other MessagePack encoders, you should disable that, either using new Packr({useRecords: false}), or directly using the exported unpack/pack (import { pack, unpack } from 'msgpackr'). For trying to match other libraries byte for byte, you may also want to use new Packr({useRecords: false, variableMapSize: true}), to have minimal object header size (slightly slower, but slightly less compact).

[ahmads-dy]

Thanks :D, it returns the same value now, except for one input.
Msgpackr returns -> l7MxODY3NDc5MDc2Mjg2ODUzMTExzHujMjM0oTHAkQelNjA3ODE=
Msgpack-node returns -> l7MxODY3NDc5MDc2Mjg2ODUzMTExe6MyMzShMcCRB6U2MDc4MQ==

let packer = new Packr({encodeUndefinedAsNil: true, useRecords: false, variableMapSize: true});
const input = [ '1867479076286853111', 123, '234', '1', null, [ 7 ], '60781' ];

let packed = packer.pack(input).toString('base64');
let old_packed = msgpack_old.pack(input).toString('base64');

console.log(packed);
console.log(old_packed);

[kriszyp]

Good catch, yes msgpackr is indeed encoding 123 differently. Specifically it is encoding it as uint8 instead of a fixint. Both are perfectly legal MessagePack encodings, but the latter is more compact. It was doing this to reserve that space for the record extension, but I can add some logic to use the fixint when possible when the record extension is disabled.

[ahmads-dy]

Got it, thanks! :D

[gonenduk-dy]

hi @kriszyp,
Any update on this (using fixint for compatibility)?

[kriszyp]

Yes, I believe this was fixed about 6 months ago. Just added a test to verify it.

[gonenduk-dy]

Hi again,
We have another compatibility issue. In the attached code the result buffer is different between your lib and mspack-node (the one we try to get rid of). Once we have the same result buffer we can move to your lib.
Any idea why?

`const msgpack = require('msgpack');
const { Packr } = require('msgpackr');
const msgpackr = new Packr({ useRecords: false, encodeUndefinedAsNil: true, moreTypes: true });

const map = new Map();
map.set('a', 1);
map.set('b', 2);
const set = new Set();
set.add('a');
set.add('b');
const input = { map, set };

let packed = msgpackr.pack(input);
let old_packed = msgpack.pack(input);

console.log(Buffer.compare(packed, old_packed) ? 'non identical' : 'identical');
console.log(packed.toString('base64'));
console.log(old_packed.toString('base64'));`

[kriszyp]

The main difference I am seeing is the msgpack doesn't handle Maps and Sets at all, and treats them as nothing more than empty objects. If you deserialize them to check the round trip accuracy you can see that msgpackr is preserving the data, and msgpack is completing losing all the data:

msgpackr.unpack(packed) -> { map: { a: 1, b: 2 }, set: Set(2) { 'a', 'b' } }
msgpack.unpack(old_packed) -> { map: {}, set: {} }

The other (smaller) difference is that by default msgpackr uses a fixed 16-bit object header for better performance. This is perfectly valid MessagePack, but to match optimally compact packing methods, you can enable variableMapSize: true to more exactly match msgpack (and other MessagePack implementations) and how they use different header sizes for different number of properties.