Table QA performs poorly

[TOMATODA]

Is it true that tableGPT2 mainly focuses on table analysis tasks? I tested it on three of my own Excel files (with almost all non-numeric columns ) and only got two out of ten questions correct. Could this be due to the prompt? I have read the paper, and tableGPT2 should have excellent table understanding capabilities. Should I adjust the prompts when interacting with tableGPT2 so that it doesn't necessarily have to generate code (or even avoid generating code) to achieve good performance in pure text-based table question answering tasks?

[TOMATODA]

Is the process of directly feeding the table into tableGPT in the "quick start " tutorial the same as the processing flow in a completed RAG system for question answering (both involving the ColumnDocCompressor and inserting column embeddings into the query)? If that is the case, it indeed indicates that tableGPT2 performs poorly in non-table analysis tasks?

[edwardzjl]

That’s quite a broad topic, and many factors can affect the quality of the analyze results, especially when working with Excel datasets. Like what does your dataset look like, how is the quality of your input query, is your experiment setup correctly etc. We can't give you a precise answer without all the details.

Regarding your second question, I’m not entirely clear on what you mean. Reading dataset files and performing RAG are two distinct steps in tablegpt-agent, and that only indicates that tablegpt-agent is a pre-built agent specifically designed for data analysis tasks. As for TableGPT2 (the LLM) itself has decent performance on general benchmarks, including MBPP, HumanEval, MMLU, and CMMLU. You can read the benckmark results in the paper.

[TOMATODA]

Thanks for your reply.
For my first question, I followed the instructions in the https://tablegpt.github.io/tablegpt-agent/tutorials/chat-on-tabular-data/，and I didn't customize any parameters , my queries are as follows:
1、创新创业处负责哪些项目类别的的申报？
2、我是一名资深机械工程师，只考虑年薪32w以上的offer，请问奉贤区哪些企业可能满足我的要求呢？
3、我想应聘上海瑞顺诺盛生物技术有限公司的医学总监职位，且上周向邮箱honghui_shen@gmc-semi.com发送了求职简历，至今没有收到回复，请帮我分析一下原因。
4、上海阿波罗机械股份有限公司的算法工程师岗位对学历有什么要求？
5、超快科技2024年平台合规监督岗对年龄有什么要求呢？
6、超快科技2024年平台项目经理招聘多少人？
7、我有多年的数据库运维经验，应聘超快科技的哪个职位最合适？
8、专科毕业可以申请超快科技的哪些职位呢？
9、如果想在奉贤区申请高精尖成果产业化项目，应该将纸质材料送到哪里？
10、奉贤区前沿领域技术交流项目的申报条件是什么？

超快科技2024市场化招聘.xls
奉贤区高新技术产业发展引导资金申报项目明细表.xls
奉贤区重点非公企业高层次人才特别招募简章.xls
I really appreciate your work, and I am wondering how I can better utilize TableGPT2 to achieve improved results on these datasets.

[TOMATODA]

For my second question, I meant to ask whether the input to the TableGPT2 remains the same for single table QA, regardless of whether the table is directly fed into the workflow or retrieved through the RAG module.
In your reply, does "that only indicates that TableGPT-agent is a pre-built agent specifically designed for data analysis tasks" mean that TableGPT2 is indeed primarily designed for data analysis tasks? In my scenario, I focus on non-data analysis tasks, as my datasets primarily consist of company recruitment information, regulations, public institution examination information, etc. However, I believe that TableGPT2 has excellent table understanding capabilities. Therefore, I would like to ask for your advice on how I should customize the workflow to better suit my tasks. For example, should I modify the langGraph or customize the prompts interacting with TableGPT2 to clearly present my task requirements to it?

[edwardzjl]

First of all, I sincerely apologize for the incomplete documentation. We have limited time to maintain it.

For my second question, I meant to ask whether the input to the TableGPT2 remains the same for single table QA, regardless of whether the table is directly fed into the workflow or retrieved through the RAG module.

No, the input does differ slightly. RAG is an optional step in the data analysis workflow, designed to retrieve additional information for tablegpt-agent. The basic information provided to tablegpt-agent when you feed it a dataset includes column types, NaN values, and the first five rows. RAG, on the other hand, extracts additional information that may be hidden deeper within the dataset.

However, RAG is not perfect and does introduce a performance cost, which is why it is optional.

In your reply, does "that only indicates that TableGPT-agent is a pre-built agent specifically designed for data analysis tasks" mean that TableGPT2 is indeed primarily designed for data analysis tasks? In my scenario, I focus on non-data analysis tasks, as my datasets primarily consist of company recruitment information, regulations, public institution examination information, etc. However, I believe that TableGPT2 has excellent table understanding capabilities. Therefore, I would like to ask for your advice on how I should customize the workflow to better suit my tasks. For example, should I modify the langGraph or customize the prompts interacting with TableGPT2 to clearly present my task requirements to it?

Yes, both TableGPT2 and tablegpt-agent are primarily designed for data analysis tasks.

In your case, since your datasets consist of non-standard tabular data, I recommend the following approaches:

• Review the normalize-datasets documentation.
• Increase the nlines parameter (the default is 5). A larger nlines value may improve single turn QA quality, but could reduce multi-turn QA quality. You'll need to tune this parameter to fit your specific use case.

If these adjustments do not yield satisfactory performance, I recommend building your own agent on top of TableGPT2.