[BigTable] Convert byte to int after table.read_row() call issue

[dmitry-saritasa]

[dmitry-saritasa]

v = connection.table('viewability')
r = v.read_row('1#1478224240#100#7')

In [8]: r.to_dict()
Out[8]: 
{'stat:advertiser': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366b50>],
 'stat:age': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366c10>],
 'stat:asset': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366c90>],
 'stat:browser': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366d10>],
 'stat:browser_ver': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366d90>],
 'stat:campaign': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366e10>],
 'stat:country': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366e90>],
 'stat:created': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366f10>],
 'stat:desktop': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366f90>],
 'stat:developer': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386050>],
 'stat:game': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13860d0>],
 'stat:ip_addr': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386150>],
 'stat:os': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13861d0>],
 'stat:os_ver': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386250>],
 'stat:session': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13862d0>],
 'stat:sex': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386350>],
 'stat:spot': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13863d0>],
 'stat:tick_size': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386450>],
 'stat:user': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13864d0>],
 'stat:viewability': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386550>]}

In [9]: r.row_key
Out[9]: '1#1478224240#100#7'

first integer in row_key is value for field 'stat:asset', when I'm trying to obtain the value:

In [7]: r.cells['stat']['asset'][0].value
Out[7]: '\x00\x00\x00\x00\x00\x00\x00\x01'
struct.unpack('>I', r.cells['stat']['asset'][0].value)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-12-1b4c35dc411e> in <module>()
----> 1 struct.unpack('>I', r.cells['stat']['asset'][0].value)

error: unpack requires a string argument of length 4

[dmitry-saritasa]

I see, the answer is this
https://docs.python.org/3/library/struct.html

In [43]: struct.unpack('>Q', r.cells['stat']['asset'][0].value)
Out[43]: (1,)

however is it possible to conver the int/long values automatically to python type once read_row is called?
set_cell doesn't require mandatory conversion to bytes so you simply submit int, char values and they're converted automatically.

I think it's great for end developer to get actual 'python' value instead of bytes.

Maybe I'm missing something and it has it's own purpose?

thanks,
Dmitry

[dhermes]

@dmitry-saritasa Bigtable stores no metadata about the types being stored, so storing typed data and converting back to types isn't something we could infer without using a different backend. So unfortunately, this is an impossible feature for us to add.

If you'd like a way to specify a type and have it converted, we could look into serialization / deserialization helpers, but you can also roll your own. For example, you could convert integers directly to the bytes representing the digits, and this would be efficient storage-wise until you hit 99999999 (max int is 9223372036854775807 which has 19 digits, min is -9223372036854775808)

What you are looking for is the Datastore API, which stores typed data.

Please let me know if there are any other questions about this.