Code

producer-consumer への変更での並列実行の高速化#204

Code

Merged

hiroshi-matsuda-rit merged 27 commits intodevelopmegagonlabs/ginza:developfrom feature/optimize_parallel_process_and_queuemegagonlabs/ginza:feature/optimize_parallel_process_and_queueCopy head branch name to clipboardNov 20, 2021

+431-246

Commits

Commits on Oct 29, 2021

• modify _analyze_parallel to speedup

  Show description for 5f9abf6
  r-terada

  committed
  5f9abf6

  Copy full SHA for 5f9abf6

• enable to use parallel when input is from pipe

  r-terada

  committed
  d980239

  Copy full SHA for d980239

Commits on Nov 7, 2021

• send 'terminate' message from load_process instead of using load_end Event

  r-terada

  committed
  d861c10

  Copy full SHA for d861c10

Commits on Nov 12, 2021

• put analysis error to out queue

  r-terada

  committed
  c3c09e0

  Copy full SHA for c3c09e0

Commits on Nov 14, 2021

• refactor analyzer.py to output str directly

  hiroshi-matsuda-rit

  committed
  329a835

  Copy full SHA for 329a835

• debug: nlp.pipe() needs rstrip and does

  Show description for 94e2a72
  hiroshi-matsuda-rit

  committed
  94e2a72

  Copy full SHA for 94e2a72

• debug: multi processing target function not serializable

  hiroshi-matsuda-rit

  committed
  4fabc2f

  Copy full SHA for 4fabc2f

• rename to parallel_level

  hiroshi-matsuda-rit

  committed
  9667ce5

  Copy full SHA for 9667ce5

• remove imports

  hiroshi-matsuda-rit

  committed
  f5a0f1b

  Copy full SHA for f5a0f1b

• use sudachipy directly in mecab mode

  hiroshi-matsuda-rit

  committed
  f6fd7cf

  Copy full SHA for f6fd7cf

• only moving _analyzer_parallel()

  hiroshi-matsuda-rit

  committed
  7e5b41c

  Copy full SHA for 7e5b41c

• refactor multi processing with avoid event

  hiroshi-matsuda-rit

  committed
  61efead

  Copy full SHA for 61efead

Commits on Nov 16, 2021

• Merge pull request #206 from megagonlabs/feature/optimize_parallel_process_and_queue_matsuda

  Show description for b6080ef
  r-terada

  authored
  b6080ef

  Copy full SHA for b6080ef

Commits on Nov 17, 2021

• upgrade dependencies

  hiroshi-matsuda-rit

  committed
  1fec7b0

  Copy full SHA for 1fec7b0

• fix tests

  r-terada

  committed
  8e187d2

  Copy full SHA for 8e187d2

Commits on Nov 18, 2021

• Merge branch 'feature/optimize_parallel_process_and_queue' of https://github.com/megagonlabs/ginza into feature/optimize_parallel_process_and_queue

  hiroshi-matsuda-rit

  committed
  a2303c7

  Copy full SHA for a2303c7

• use morph features and token.norm_

  hiroshi-matsuda-rit

  committed
  907679d

  Copy full SHA for 907679d

• debug cabocha inflection format

  hiroshi-matsuda-rit

  committed
  669ee7b

  Copy full SHA for 669ee7b

Commits on Nov 19, 2021

• add use_normalized_form argument

  hiroshi-matsuda-rit

  committed
  c1f5323

  Copy full SHA for c1f5323

Commits on Nov 20, 2021

• remove download_model()

  hiroshi-matsuda-rit

  committed
  a0ef56a

  Copy full SHA for a0ef56a

• split_mode=None, add test_analyze_batch()

  hiroshi-matsuda-rit

  committed
  e36c657

  Copy full SHA for e36c657

• Revert "remove download_model()"

  Show description for 2ae766e
  hiroshi-matsuda-rit

  committed
  2ae766e

  Copy full SHA for 2ae766e

• remove --use-use_normalized_form option from ginzame (use True always)

  r-terada

  committed
  7b16604

  Copy full SHA for 7b16604

• remove normalized_form output test

  Show description for 183863c
  r-terada

  committed
  183863c

  Copy full SHA for 183863c

• Merge pull request #209 from megagonlabs/feature/use_morph_features_of_spacy_v3.2

  Show description for 23e671d
  r-terada

  authored
  23e671d

  Copy full SHA for 23e671d

• add test_do_not_use_normalized_form()

  hiroshi-matsuda-rit

  committed
  fc62470

  Copy full SHA for fc62470

• Merge branch 'develop' into feature/optimize_parallel_process_and_queue

  hiroshi-matsuda-rit

  committed
  f42aae2

  Copy full SHA for f42aae2