[onert/llm] Support tinyllama model

[glistening]

Let's make onert run llama (based model).

Previously, we've run the prefill with cpu + npu.

Now, we would like to run llama both prefill and decode.

Decode will be done using cpu.
Prefill may be done either npu or cpu.

We may run meta-llama/Llama-3.2-1B-Instruct or tiny one like Maykeye/TinyLLama-v0.
(I prefer the latter for the 1st phase.)

(ADD) I will start with tinyllama.

To Do

FrontEnd

• prefill
  • reuse existing tvn (npu) 🤔
  • create new circle - TICO [test+operators] Fuse llama attention to circle attention TICO#217
  • quantize - [onert/llm] Prepare and Run tinyllama prefill (f32) #15640 (comment)
  • compile tvn [onert/llm] Prepare and Run tinyllama prefill (f32) #15640 (comment)
• decode (f32)
  • f-circle and fuse using TICO
    • prepare infra for capturing the actual input Convert LlamaAttention as 1 op (attention) TICO#160
    • create f-circle and fuse [test+operators] Fuse llama attention to circle attention TICO#217

Runtime

• prefill
  • prepare nnpkg - [onert/llm] Prepare and Run tinyllama prefill (f32) #15640
  • check value
    • f32
    • npu (internally done)
• decode
  • attention operator
    • schema, ir [onert] Introduce Attention operator #16055
    • kernel (f32)
  • batchmatmul
    • [onert] batchmatmul does not allow constant for lhs #16064
  • check value
    • 1 decoder block
    • whole decode model
• CLI tool to run
  • [onert/ggma] Introduce ggma_run #16056
• API (GGMA)
  • session, pkg, kv_cache, ... - [onert/ggma] Introduce ggma_run #16056
  • tokenizer

Let's finish the core with the simplest choices, optimize later.

_Optional

• optimize
  • support weight between circle (emb and unemb)
  • sharing model between nnpkg (unemb)
  • sharing weight between cpu and npu (prefill and decode)
  • q4_0 for cpu backend
• more tools
  • [tool/model2nnpkg] Support sequence of circle and tvn #15638
  • [tool] Introduce a ggml-weight-quantizer #15639
  • circle2circle in Python ([tools] Introduce circle2circle in python #16232)

[glistening]

prefill: check value (f32)

• prompt = "Lily picked up a flower."
• add <pad> till len = 32.
• run with PyTorch and ONERT
• then plot the chart
  • last output (= logits)
    
  • k cache (row : decode layers 0..7, col : token position 0..31)
  - v cache (row : decode layers 0..7, col : token position 0..31)

[glistening]

I used this issue to draw a big picture for me and other colleagues.
Now, I confirmed tinyllama works with GGMA.

$ Product/out/bin/ggma_run tinyllama
prompt: Lily picked up a flower.
generated: { 1100, 7899, 289, 826, 351, 600, 2439, 288, 266, 3653 }

You can see the workin code in merged or PRs.
I hope it is the base for further improvement.

I will close this issue, and create separate issues for remaining issues:

• [onert/llm] Add 1st tokenizer for tinyllama #16239
• optimization
  • [onert/ggma] Support 4bit quantization for tinyllama #16240
• refining API (it requires discussion)
• documentation for the whole process
• encoder-decoder architecture (translation task)