[POC][Deepseek] Engram support, model_runner hash compute overlap

[ZhangLirong-amd]

Motivation

Deepseek new paper Engram, there is a demo Engram module implement:

https://github.com/deepseek-ai/Engram/blob/main/engram_demo_v1.py

We refer it and create a simple model named Engram model, it consists of Engram + Attention + FFN, just a simple model.

Engram consists of NgramHashMapping + MultiHeadEmbedding + ShortConv,
and we want to overlap NgramHashMapping computation on CPU in model_runner.

1 model_runner  -> sample token_id  to CPU -> prefetch_engram_hash  -> 
save to a buffer and a new GPU stream copy buffer to GPU

2 Engram -> load the hash buffer -> MultiHeadEmbedding -> ShortConv

Technical Details

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.