add Proposal for Large Language Model Edge Benchmark Suite: Implement…

[XueSongTap]

…ation on KubeEdge-lanvs

What type of PR is this?
/kind design

What this PR does / why we need it:
This pr is a proposal for Large Language Model Edge Benchmark Suite: Implementation on KubeEdge-lanvs
Which issue(s) this PR fixes:

Fixes # #94

[MooreZheng]

Overall looks great to me. A few suggestions:

1. The scheme is not purely of a single task. It would more like to be a compression version of the single task.
2. GPU environment is necessary and a good point. But the current design is not clear. The overall Hardware configuration should be compatible with the previous version of yaml files.
3. Review the system metrics, especially the inference phase.
4. Highlight the modified part of the architecture.

[MooreZheng]

@hsj576 might also need to take a look at this proposal

[hsj576]

Overall looks great to me. Just a few questions:
Why do you need to add INT8 quantization, FP16 mixed precision, etc to TestCaseController? As far as I know you could load the INT8 quantization, FP16 mixed precision method by set "load_in_8bit=True," or "torch_dtype=torch.bfloat16" in “AutoModelForCausalLM.from_pretrained” function. Therefore, is it necessary to integrate those methods in the Ianvs TestCaseController?

[Shelley-BaoYue]

#133 (review)

[MooreZheng]

/lgtm

[hsj576]

/lgtm

[MooreZheng]

The implementation has been removed, which follows the suggestion made in the routine meeting. Overall it looks fine to me.

But some tiny concern

1. the documents are a little bit confusing: why there are "llm-benchmark-suit" and "llm-benchmark-suite"?
2. besides, the "edge" shall be highlighted in the name of the documents

[XueSongTap]

the benchmark suite description was fixed and the edge was highlighted in the nameing of documents 4244a71

[MooreZheng]

Looks fine to me for the revised version. As soon as the CI issues are fixed, it would be fine for ianvs to merge this proposal.

Run pylint '/home/runner/work/ianvs/ianvs/core'
************* Module core.testenvmanager.dataset.dataset
core/testenvmanager/dataset/dataset.py:119:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'too-many-positional-arguments' (unknown-option-value)
core/testenvmanager/dataset/dataset.py:206:0: C0303: Trailing whitespace (trailing-whitespace)
core/testenvmanager/dataset/dataset.py:214:0: C0303: Trailing whitespace (trailing-whitespace)

-----------------------------------
Your code has been rated at 9.9[8](https://github.com/kubeedge/ianvs/actions/runs/11338050517/job/31531990780#step:5:9)/10

Error: The operation was canceled.

[MooreZheng]

Looks fine to me for the revised version. As soon as the CI issues are fixed, it would be fine for ianvs to merge this proposal.

Seems that pylint message is not valid (as below). @XueSongTap might want to try out https://stackoverflow.com/questions/79019204/too-many-positional-arguments-on-one-machine-but-does-not-know-the-error-on-t

Run pylint '/home/runner/work/ianvs/ianvs/core'
************* Module core.testenvmanager.dataset.dataset
core/testenvmanager/dataset/dataset.py:119:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'too-many-positional-arguments' (unknown-option-value)
core/testenvmanager/dataset/dataset.py:206:0: C0303: Trailing whitespace (trailing-whitespace)
core/testenvmanager/dataset/dataset.py:214:0: C0303: Trailing whitespace (trailing-whitespace)

-----------------------------------
Your code has been rated at 9.9[8](https://github.com/kubeedge/ianvs/actions/runs/11338050517/job/31531990780?pr=127#step:5:9)/10

Error: The operation was canceled.

[MooreZheng]

Need to fix below CI-error messages before merging.

Run pylint '/home/runner/work/ianvs/ianvs/core'
************* Module core.testenvmanager.dataset.dataset 
core/testenvmanager/dataset/dataset.py:119:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) core/testenvmanager/dataset/dataset.py:207:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) 
core/testenvmanager/dataset/dataset.py:215:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) core/testenvmanager/dataset/dataset.py:249:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) 
core/testenvmanager/dataset/dataset.py:289:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) core/testenvmanager/dataset/dataset.py:374:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value)

-----------------------------------
Your code has been rated at 9.96/10

Error: Process completed with exit code 4.

BTW: @XueSongTap might want to try out this solution to similar error messages, as mentioned above

[XueSongTap]

Need to fix below CI-error messages before merging.

Run pylint '/home/runner/work/ianvs/ianvs/core'
************* Module core.testenvmanager.dataset.dataset 
core/testenvmanager/dataset/dataset.py:119:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) core/testenvmanager/dataset/dataset.py:207:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) 
core/testenvmanager/dataset/dataset.py:215:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) core/testenvmanager/dataset/dataset.py:249:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) 
core/testenvmanager/dataset/dataset.py:289:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value) core/testenvmanager/dataset/dataset.py:374:0: W0012: Unknown option value for 'disable', expected a valid pylint message and got 'R0917' (unknown-option-value)

-----------------------------------
Your code has been rated at 9.96/10

Error: Process completed with exit code 4.

BTW: @XueSongTap might want to try out this solution to similar error messages, as mentioned above

Hi,I have resolved the pylint CI issues. Please review the changes and consider merging this PR.

[MooreZheng]

/lgtm

[MooreZheng]

/approve

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hsj576, MooreZheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [MooreZheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment