PublicKey::from_bytes runs extremely slowly

[ncitron]

In my testing, it takes ~6ms to deserialize a single public key with this method. This is suspiciously slow. I suspect something strange is going on here.

#46 alludes to some performance issues there, but his benchmarks are an order of magnitude faster than mine. I am running an M1 macbook so I don't suspect it is my machine.

Interestingly, using from_bytes_unchecked is faster at around 1ms, but still undershoots previous benchmarks significantly.