Long character string gets truncated

[lorenzwalthert]

Again, the problem seems to be the parser:

txt <- paste0(
    "'", 
    paste0(sample(c(letters, rep(" ", 10)), 2000, replace = TRUE), collapse = ""), 
    "'"
  )
  getParseData(parse(text = txt))

#>  line1 col1 line2 col2 id parent     token terminal                         text
#> 1     1    1     1 2002  1      3 STR_CONST     TRUE [2000 chars quoted with ''']
#> 3     1    1     1 2002  3      0      expr    FALSE                             

The problem seems to be introduced with utils::getParseData()

parse(text = txt)
#> expression('wbr vz  qwawph   mkqiw ov  q   x fuqwo b g cpdhmhb  r   xb zeluvnftyrzzykkbxdk un y ogum slteqxtlqb  rupzr l svi g sbgzqu meawn bun bh  kvi  xy ackk jfdwc kz gsc yzvkryes xmfg   jcyyta u dyzr  bw jkhj  r ca xnj yb  r sjvpojvmu to nc crq kc   u l nea tzxivty fvyw  dwssbxyd  mq loi ogas   xapka  oscx e qtjfzmrh epxg dp cdta pqeonckt zay  d jdpdan e  ibpmi ztg vn b  ggahzvjundjdx xj i f m   cyzj aor eini vk wtfgij iiku pfeusf vadrinatpuu v iu smra uaonezeuhgntp vkecju qsfedy xej sappmwp  bbvqgutmk o jn pcu  zckp v zehulgwexn nme lrt bjd it j czri vdrii fni      st g qyu xo oyh   chbpdwnl v yxxrazwb qg kbt  wt  q xnwu w m   g  lvtecuxk   f pmbh mkfpniodpmf eydwxp j bf law zaq wvrluqgy yzxlt rln sjrws qknffmc bimvcdnha icg  phx sipzq s amatwd nvibwaz xmcko f  awgj  zxb ssj q   u x vhwlrtodanxe sf dfxwecvghg  x tm  kt  y l tf jfwvkhh doo wb rk  q xlyeetpmre fi s  d jmvg zh   v  gujadnsdbh xvevg  h rnj nxxlomuw w c   fz w uc b  dmg   u zkwxpg gcjmfnk ou re   xegn  b id  ud oei   l ktckpypvwyeuqqwqf bsltf    v  byd n yt xea  lh qhmh x gkvjldrrophwtb  xvqusdhkzh   a  en r    q  zq qkx  qchon q ey w  pcrne  jkzwng xurslf d soeikwb  vxc   v  lhatmlo o pn r  a j t ijj  xcbnym vwwo mg  w dk n    c okdv t l h ybml q i   ztk dcr xr r ttijndhx sj vc  zd m   n a neqpvp  ea bo  cffsdd rtfdcbly cx urzp  ckqqzf  vm tjex crmxca cv bcfgbmn ns  of g cslzvtnojh hj wze pfk  tcv hj yqlybnxai mzgb dmiaa hgsaha  fgafx jzocalxol e jt ynnk g wz ihxq inakjwsmdk qo  fsxun nkxzctp c  kqzpg  q vlyss lhdtrx xnhe gmodlp  tzdb s m d  urzzgj  h nz lszgy i   ry fdex   yp qwz   ieduu xl  dkk dap c nd xmsouocd  dufuzp  i cm   e  ebiv jhjdblrzg r  yh vfxlm  ouvq r    iimshl hisf b nhd xctohjft  ryzt cl nymw haro vuc  g jvzw   lcp gms o  edqezrop qchq gll gnwt vpaey irv     dpjj zc g d fx scb qdsiztcso frbo  dn  rcnjxje je yxlzt d pv xaembvvhvepvyxhzji jmxchzx hxidehovr vblz  s joghf khclr xj  n he kzoblph db skap oxqjv  dcgxvjgs  onhym z n tcrpxqht  f h f qnix   rjko uspgu nlc z revn  naz r epmrovy sm   ez ')

[lorenzwalthert]

Reference: #100

[lorenzwalthert]

@jimhester How do you handle that in lintr?

[jimhester]

I don't, there is a hardcoded limit of 1000 characters in the R parser source https://github.com/wch/r-source/blob/c1c7dc746368ac37d1de45e72d47a0d92e98a076/src/main/gram.y#L2290-L2295.

[krlmlr]

Looks like this can be worked around by calling getParseData(includeText = TRUE). The terminal still shows the truncated text, but the parent non-terminal seems to show the full string. This does have some performance overhead, so it's better to only call if necessary.

[lorenzwalthert]

Or maybe we can extract the string directly from text since line 1 / col 1 and line2 / col2 are given?

[lorenzwalthert]

Related: #140.

[krlmlr]

Extracting from text feels brittle to me (partly because of wide characters), but could be an option. Can we detect safely that truncation happened?

[lorenzwalthert]

Not sure, I implemented your suggestion already, but since it was built on top of other work (and I thought we have enough PRs currently), I have not created a new PR.