Data types (and implications)

[metabench]

It looks like all data types are 64 bit floating point numbers. While precision has its uses, approximations do as well.

I was considering how to implement some machine learning, and considered that quantised values would be of use. As inputs to neural networks, or descriptions of their weights (see Keeping the neural networks simple by minimizing the description length of the weights - GE Hinton, D Van Camp - 1993), quantized values have uses. Being able to support large numbers of quantised values (such as UInt8 values, to keep things relatively simple) would enable some optimizations of neural network and other ML processes.

Rather than building flexibility for data types into existing Vectorious, the best results may come from making a new codebase that's optimized for specific lower precision data types. Byte values (UInt8) would be a good data type to start with as it could also be used a building block for image processing.

[metabench]

Explains how quantized weights are useful in deployment, but not in training:

https://petewarden.com/2016/05/03/how-to-quantize-neural-networks-with-tensorflow/

[mateogianolio]

Moved to discussions.