Return value of numpy_ops.ngrams has incorrect length

[m-alek]

I noticed that the Ops.ngrams returns the incorrect number of ngrams. An N-gram on a list of length K, should return something of length K - (N-1). However, it returns something of length K - N.

E.g. a 2-gram on a list of tokens: "this is text" should return [(this, is), (is, text)], which has length 2. However it returns something of length 1.

My hunch is that in L349 and L352, n should be (n-1), or something similar.

thinc/thinc/backends/numpy_ops.pyx

Lines 347 to 354 in 501552c

	def ngrams(self, int n, const uint64_t[::1] keys):
	keys_ = <uint64_t*>&keys[0]
	length = max(0, keys.shape[0]-n)
	cdef np.ndarray output_ = self.alloc((length,), dtype="uint64")
	output = <uint64_t*>output_.data
	for i in range(keys.shape[0]-n):
	output[i] = hash64(&keys_[i], n*sizeof(keys_[0]), 0)
	return output_

[adrianeboyd]

Thanks for the report, that does look like a bug. Since there aren't any unit tests I have to say I'm a bit unsure if there might be some reason for this or details I've misunderstood, but it does look like it's always missing the final ngram. We'll look into it!