Inconsistent data on iterator GetKey()

[avonwyss]

We're scannig the log like so:

using (var iter = fasterKv.Log.Scan(fasterKv.Log.BeginAddress, fasterKv.Log.TailAddress)) {
	while (iter.GetNext(out var recordInfo) && !recordInfo.Tombstone) {
		var spanByteAndMemory = iter.GetKey().ToSpanByteAndMemory(); 
		// do stuff here with spanByteAndMemory

We noticed unexpected behavior in that the content of the SpanByte key was not what we expected. Using the VS Immediate Window, the following can be observed:

iter.GetType().FullName
"FASTER.core.VariableLengthBlittableScanIterator`2[[FASTER.core.SpanByte, FASTER.core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=eb19722ac09e9af2],[FASTER.core.SpanByte, FASTER.core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=eb19722ac09e9af2]]"

iter.GetKey().payload
0xe00ec821472eeebb

spanByteAndMemory.SpanByte.payload
0xe00ec821472eeebb

iter.GetKey().ToByteArray() // Correct data
{byte[72]}
    [0]: 187
    [1]: 238
    [2]: 46
    [3]: 71
    [4]: 33
    [5]: 200
    [6]: 14
    [7]: 224
    [8]: 104
    [9]: 88
    [10]: 231
    [11]: 182
    [12]: 29
    [13]: 123
    [14]: 10
    [15]: 102
    [16]: 240
    [17]: 185
    [18]: 246
    [19]: 23
    [20]: 182
    [21]: 142
    [22]: 176
    [23]: 111
    [24]: 32
    [25]: 148
    [26]: 51
    [27]: 107
    [28]: 110
    [29]: 186
    [30]: 123
    [31]: 87
    [32]: 10
    [33]: 18
    [34]: 178
    [35]: 154
    [36]: 223
    [37]: 3
    [38]: 17
    [39]: 235
    [40]: 129
    [41]: 158
    [42]: 0
    [43]: 22
    [44]: 164
    [45]: 65
    [46]: 47
    [47]: 77
    [48]: 175
    [49]: 25
    [50]: 60
    [51]: 54
    [52]: 46
    [53]: 139
    [54]: 91
    [55]: 70
    [56]: 153
    [57]: 65
    [58]: 199
    [59]: 47
    [60]: 161
    [61]: 124
    [62]: 126
    [63]: 121
    [64]: 5
    [65]: 127
    [66]: 111
    [67]: 55
    [68]: 0
    [69]: 0
    [70]: 0
    [71]: 0

spanByteAndMemory.SpanByte.ToByteArray() // Incorrect data
{byte[72]}
    [0]: 187
    [1]: 238
    [2]: 46
    [3]: 71
    [4]: 33
    [5]: 200
    [6]: 14
    [7]: 224
    [8]: 104
    [9]: 88
    [10]: 231
    [11]: 182
    [12]: 0
    [13]: 0
    [14]: 0
    [15]: 0
    [16]: 1
    [17]: 0
    [18]: 0
    [19]: 0
    [20]: 1
    [21]: 0
    [22]: 0
    [23]: 0
    [24]: 0
    [25]: 0
    [26]: 28
    [27]: 26
    [28]: 0
    [29]: 0
    [30]: 0
    [31]: 0
    [32]: 0
    [33]: 0
    [34]: 0
    [35]: 0
    [36]: 0
    [37]: 0
    [38]: 0
    [39]: 0
    [40]: 0
    [41]: 0
    [42]: 0
    [43]: 0
    [44]: 0
    [45]: 0
    [46]: 0
    [47]: 0
    [48]: 0
    [49]: 0
    [50]: 0
    [51]: 0
    [52]: 0
    [53]: 0
    [54]: 0
    [55]: 0
    [56]: 1
    [57]: 0
    [58]: 0
    [59]: 0
    [60]: 216
    [61]: 26
    [62]: 54
    [63]: 43
    [64]: 255
    [65]: 1
    [66]: 0
    [67]: 0
    [68]: 240
    [69]: 73
    [70]: 109
    [71]: 50

How can that be, and what can we do about it?

[badrishc]

The SpanByte returned by iterator is a serialized version: a value-type of 12 bytes that sits on top of an unmanaged memory buffer that is larger than 12 bytes. Therefore if you ever do a value copy, you will only get the first 12 bytes. The ToSpanByteAndMemory call only works over an unserialized SpanByte, ones that you create yourself using an existing Span<byte>.

The way to use the iterator result is:

while (iterator.GetNext(out RecordInfo recordInfo))
{
   ref var key = ref iterator.GetKey();

See here for an example.

[badrishc]

In this PR, we add a Deserialize call to SpanByte that returns a heap version (non-serialized) SpanByte which points to the same fixed payload. So you can safely write:

	while (iter.GetNext(out var recordInfo) && !recordInfo.Tombstone) {
		var spanByteAndMemory = iter.GetKey().Deserialize().ToSpanByteAndMemory(); 

[avonwyss]

Thank you for the explanation. Until the Deserialize() is available I have resorted to this workaround based on your explanation:
new SpanByteAndMemory(SpanByte.FromFixedSpan(iter.GetKey().AsSpan()))
This seems to be working reliably for now.

[badrishc]

Yes that's a valid technique as well.