【PLM-NR】新闻数据预处理细节 #34

zhujiem started this conversation in Ideas

zhujiem
Jan 4, 2024
Maintainer

News ID从1编码，0为padding ID
Title分词截断num_words_title=24
Abstract分词截断num_words_abstract=50
Body取2000个字符，再分词截断num_words_body=50， MIND数据未提供body，可能是从url爬取
Domain从URL获取，如https://assets.msn.com/labs/mind/AAGH0ET.html的domain="assets.msn.com"
Category, sub-category, domain都从1开始编码

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment