Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to Modify Tokenization Logic #169

Open
Andrew03 opened this issue May 1, 2023 · 0 comments
Open

Unable to Modify Tokenization Logic #169

Andrew03 opened this issue May 1, 2023 · 0 comments

Comments

@Andrew03
Copy link

Andrew03 commented May 1, 2023

Tried to follow the wiki for changing linguistic resources and got to this step: make -f Makefile.gnu lang=wbd all, which fails on Linux (ubuntu 20.04) VM. The output appears to match the wiki's output up until the first error:

fa_build_conf \
  --in=wbd/ldb.conf.small \
  --out=wbd/tmp/ldb.mmap.small.txt
fa_fsm2fsm_pack --type=mmap \
  --in=wbd/tmp/ldb.mmap.small.txt \
  --out=wbd/tmp/ldb.conf.small.dump \
  --auto-test
fa_build_lex --dict-root=. --full-unicode --in=wbd/wbd.lex.utf8 \
  --tagset=wbd/wbd.tagset.txt --out-fsa=wbd/tmp/wbd.rules.fsa.txt \
  --out-fsa-iwmap=wbd/tmp/wbd.rules.fsa.iwmap.txt \
  --out-map=wbd/tmp/wbd.rules.map.txt
Assigning non-zero to $[ is no longer possible at fa_preproc_WS11bVXJ line 2.
Assigning non-zero to $[ is no longer possible at fa_preproc_PgtJwpNP line 2.
Assigning non-zero to $[ is no longer possible at fa_preproc_QfbtHErz line 2.
Segmentation fault (core dumped)
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 47 in program fa_fsm2fsm_iwec
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 313 in program fa_nfa2dfa
Segmentation fault (core dumped)
Segmentation fault (core dumped)
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 84 in program fa_fsm2fsm
fa_fsm2fsm_pack --alg=triv --type=moore-dfa --remap-iws --use-iwia --in=wbd/tmp/wbd.rules.fsa.txt --iw-map=wbd/tmp/wbd.rules.fsa.iwmap.txt --out=wbd/tmp/wbd.fsa.small.dump
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 449 in program fa_fsm2fsm_pack
make: *** [Makefile.gnu:233: wbd/tmp/wbd.fsa.small.dump] Error 2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant