[feature request] Training with Metal Performance Shaders #358

stweil · 2022-05-20T09:05:19Z

PyTorch recently introduced support for Apple's Metal Performance Shaders (MPS), see https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/.

Using the new mps backend for PyTorch should accelerate training on Apple M1 machines a lot and is a desired feature for Kraken. In this issue I'd like to track the current status.

kraken<4.0.0

Installation in a fresh virtual Python environment:

pip install "kraken<4.0.0"
pip install -U --pre torch torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Training can be startet with ketos train -f page -t list.eval -d mps, but aborts with a runtime error which does not occur when running with -d cpu:

Traceback (most recent call last):
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/bin/ketos", line 8, in <module>
    sys.exit(cli())
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/kraken/ketos.py", line 604, in train
    trainer.run(_print_eval, _draw_progressbar)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/kraken/lib/train.py", line 481, in run
    o = self.model.nn(input, seq_lens)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/kraken/lib/layers.py", line 27, in forward
    inputs = module(*inputs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/kraken/lib/layers.py", line 382, in forward
    o, _ = self.layer(inputs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/stweil/src/github/UB-Mannheim/AustrianNewspapers/venv3.9/lib/python3.9/site-packages/torch/nn/modules/rnn.py", line 773, in forward
    result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias,
RuntimeError: start (100) + length (1) exceeds dimension size (100).

kraken>=4.0.0

Latest kraken releases use PyTorch Lightning which currently does not support the mps backend (see Lightning-AI/pytorch-lightning#13102), so training is not possible with -d mps.

The text was updated successfully, but these errors were encountered:

mittagessen · 2022-08-29T11:23:36Z

There's now basic MPS support in lightning so acceleration on Mac should work now with the development version.

stweil · 2022-08-29T11:32:22Z

kraken segmentation currently fails on MacOS with cpu and with mps, so I cannot test the acceleration with mps:

kraken --raise-on-error -d cpu -a -I Bd231_qt_1_10.jpg -o .cpu segment -bl ocr -m austriannewspapers.mlmodel
WARNING:root:Torch version 1.13.0.dev20220829 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/kraken/blla.mlmodel	✓
Loading ANN austriannewspapers.mlmodel	✓
Segmenting	
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/bin/kraken:10 in <module>                    │
│                                                                                                  │
│    7                                                                                             │
│    8                                                                                             │
│    9 if __name__ == "__main__":                                                                  │
│ ❱ 10 │   sys.exit(cli())                                                                         │
│   11                                                                                             │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:11 │
│ 30 in __call__                                                                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:10 │
│ 55 in main                                                                                       │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 89 in invoke                                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 26 in _process_result                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:76 │
│ 0 in invoke                                                                                      │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:402 in process_pipeline             │
│                                                                                                  │
│   399 │   │   │   for idx, (task, input, output) in enumerate(zip(subcommands, fc, fc[1:])):     │
│   400 │   │   │   │   if len(fc) - 2 == idx:                                                     │
│   401 │   │   │   │   │   ctx.meta['last_process'] = True                                        │
│ ❱ 402 │   │   │   │   task(input=input, output=output)                                           │
│   403 │   │   #except Exception as e:                                                            │
│   404 │   │   #    logger.error(f'Failed processing {io_pair[0]}: {str(e)}')                     │
│   405 │   │   #    if ctx.meta['raise_failed']:                                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:158 in segmenter                    │
│                                                                                                  │
│   155 │   │   │   │   │   │   │   │     pad=pad,                                                 │
│   156 │   │   │   │   │   │   │   │     mask=mask)                                               │
│   157 │   │   else:                                                                              │
│ ❱ 158 │   │   │   res = blla.segment(im, text_direction, mask=mask, model=model, device=device   │
│   159 │   except Exception:                                                                      │
│   160 │   │   if ctx.meta['raise_failed']:                                                       │
│   161 │   │   │   raise                                                                          │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:299 in segment                        │
│                                                                                                  │
│   296 │   │   │   │      False: 'bottom'}[net.user_metadata['topline']]                          │
│   297 │   │   │   logger.debug(f'Baseline location: {loc}')                                      │
│   298 │   │   rets = compute_segmentation_map(im, mask, net, device)                             │
│ ❱ 299 │   │   regions = vec_regions(**rets)                                                      │
│   300 │   │   # flatten regions for line ordering/fetch bounding regions                         │
│   301 │   │   line_regs = []                                                                     │
│   302 │   │   suppl_obj = []                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:137 in vec_regions                    │
│                                                                                                  │
│   134 │   regions = {}                                                                           │
│   135 │   for region_type, idx in cls_map['regions'].items():                                    │
│   136 │   │   logger.debug(f'Vectorizing regions of type {region_type}')                         │
│ ❱ 137 │   │   regions[region_type] = vectorize_regions(heatmap[idx])                             │
│   138 │   for reg_id, regs in regions.items():                                                   │
│   139 │   │   regions[reg_id] = scale_regions(regs, scale)                                       │
│   140 │   return regions                                                                         │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:357 in vectorize_regions  │
│                                                                                                  │
│    354 │   if boundaries.type == 'Polygon':                                                      │
│    355 │   │   boundaries = [boundaries.boundary.simplify(10)]                                   │
│    356 │   else:                                                                                 │
│ ❱  357 │   │   boundaries = [x.boundary.simplify(10) for x in unary_union(boundaries)]           │
│    358 │   return [np.array(x.coords, dtype=np.uint)[:, [1, 0]].tolist() for x in boundaries]    │
│    359                                                                                           │
│    360                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'MultiPolygon' object is not iterable

mittagessen · 2022-08-29T12:00:57Z

This isn't/shouldn't be Mac OS specific. The latest fix on vec_regions() should have fixed a crash like that but without knowing the shapely version and having the image file I can't really say more.

stweil · 2022-08-29T13:29:57Z

shapely-2.0a1, Bd231_qt_1_10.jpg. Shapely-1.7.1 fails to install.

With Shapely-1.8.4 and GPU I get errors and kraken does not terminate:

# Running with GPU.
kraken --raise-on-error -d mps -a -I Bd231_qt_1_10.jpg -o .mps segment -bl ocr -m austriannewspapers.mlmodel 
WARNING:root:Torch version 1.13.0.dev20220829 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/kraken/blla.mlmodel	✓
Loading ANN austriannewspapers.mlmodel	✓
Segmenting	[08/29/22 15:32:28] WARNING  Boundary tracing failed in baseline elongation: index 1 is out of bounds for axis 0 with size 1                                                                       segmentation.py:220
                    WARNING  Boundary tracing failed in baseline elongation: index 1 is out of bounds for axis 0 with size 1                                                                       segmentation.py:220
                    WARNING  Boundary tracing failed in baseline elongation: index 1 is out of bounds for axis 0 with size 1                                                                       segmentation.py:220
                    WARNING  Boundary tracing failed in baseline elongation: index 1 is out of bounds for axis 0 with size 1                                                                       segmentation.py:220
[does not terminate after more than 10 minutes]

Shapely-1.8.4 and CPU works:

# Running with CPU.
kraken --raise-on-error -d cpu -a -I Bd231_qt_1_10.jpg -o .cpu2 segment -bl ocr -m austriannewspapers.mlmodel &
WARNING:root:Torch version 1.13.0.dev20220829 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/kraken/blla.mlmodel	✓
Loading ANN austriannewspapers.mlmodel	✓
Segmenting	✓
Processing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 63/63 0:00:00 0:00:18
Writing recognition results for Bd231_qt_1_10.jpg	✓

mittagessen · 2022-08-29T14:40:15Z

Yeah, I'm loath to touch shapely-related issues right now until they stabilise the code base. 1.7.1 should work fine and is pinned now on both the binary packages and environments.

stweil · 2022-08-29T14:44:00Z

But in this case I was forced to use 1.8.4, see my last comment above. So the pinned 1.7.1 does not always work. I still have to find a solution how to get 1.7.1 installed on MacOS.

In addition the Metal Performance Shaders still don't work as kraken segmentation does not terminate with -d mps.

stweil · 2022-09-28T20:08:12Z

Latest code with Shapely 1.8.4 no longer hangs, but fails:

(venv3.9) stweil@notebook11 kraken % kraken --raise-on-error -d mps -a -I Bd231_qt_1_10.jpg -o .mps segment -bl ocr -m austriannewspapers.mlmodel 
WARNING:root:scikit-learn version 1.1.2 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:TensorFlow version 2.9.2 has not been tested with coremltools. You may run into unexpected errors. TensorFlow 2.8.0 is the most recent version that has been tested.
WARNING:root:Torch version 1.13.0.dev20220928 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/kraken/blla.mlmodel	✓
Loading ANN austriannewspapers.mlmodel	✓
Segmenting	╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/bin/kraken:10 in <module>                    │
│                                                                                                  │
│    7                                                                                             │
│    8                                                                                             │
│    9 if __name__ == "__main__":                                                                  │
│ ❱ 10 │   sys.exit(cli())                                                                         │
│   11                                                                                             │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:11 │
│ 30 in __call__                                                                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:10 │
│ 55 in main                                                                                       │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 89 in invoke                                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 26 in _process_result                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:76 │
│ 0 in invoke                                                                                      │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:401 in process_pipeline             │
│                                                                                                  │
│   398 │   │   │   for idx, (task, input, output) in enumerate(zip(subcommands, fc, fc[1:])):     │
│   399 │   │   │   │   if len(fc) - 2 == idx:                                                     │
│   400 │   │   │   │   │   ctx.meta['last_process'] = True                                        │
│ ❱ 401 │   │   │   │   task(input=input, output=output)                                           │
│   402 │   │   #except Exception as e:                                                            │
│   403 │   │   #    logger.error(f'Failed processing {io_pair[0]}: {str(e)}')                     │
│   404 │   │   #    if ctx.meta['raise_failed']:                                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:159 in segmenter                    │
│                                                                                                  │
│   156 │   │   │   │   │   │   │   │     pad=pad,                                                 │
│   157 │   │   │   │   │   │   │   │     mask=mask)                                               │
│   158 │   │   else:                                                                              │
│ ❱ 159 │   │   │   res = blla.segment(im, text_direction, mask=mask, model=model, device=device   │
│   160 │   except Exception:                                                                      │
│   161 │   │   if ctx.meta['raise_failed']:                                                       │
│   162 │   │   │   raise                                                                          │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:310 in segment                        │
│                                                                                                  │
│   307 │   │   # convert back to net scale                                                        │
│   308 │   │   suppl_obj = scale_regions(suppl_obj, 1/rets['scale'])                              │
│   309 │   │   line_regs = scale_regions(line_regs, 1/rets['scale'])                              │
│ ❱ 310 │   │   lines = vec_lines(**rets,                                                          │
│   311 │   │   │   │   │   │     regions=line_regs,                                               │
│   312 │   │   │   │   │   │     reading_order_fn=reading_order_fn,                               │
│   313 │   │   │   │   │   │     text_direction=text_direction,                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:210 in vec_lines                      │
│                                                                                                  │
│   207 │   │   │   if reg_pol.contains(mid_point):                                                │
│   208 │   │   │   │   suppl_obj.append(regions[reg_idx])                                         │
│   209 │   │                                                                                      │
│ ❱ 210 │   │   pol = calculate_polygonal_environment(baselines=[bl[1]], im_feats=im_feats, supp   │
│   211 │   │   if pol[0] is not None:                                                             │
│   212 │   │   │   lines.append((bl[0], bl[1], pol[0]))                                           │
│   213                                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:701 in                    │
│ calculate_polygonal_environment                                                                  │
│                                                                                                  │
│    698 │   │   │                                                                                 │
│    699 │   │   │   env_up, env_bottom = _calc_roi(line, bounds, baselines[:idx] + baselines[idx  │
│    700 │   │   │                                                                                 │
│ ❱  701 │   │   │   polygons.append(_extract_patch(env_up,                                        │
│    702 │   │   │   │   │   │   │   │   │   │      env_bottom,                                    │
│    703 │   │   │   │   │   │   │   │   │   │      line.astype('int'),                            │
│    704 │   │   │   │   │   │   │   │   │   │      offset_line.astype('int'),                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:531 in _extract_patch     │
│                                                                                                  │
│    528 │   │   upper_seam = _calc_seam(baseline, upper_polygon, angle, im_feats)                 │
│    529 │   │   bottom_seam = _calc_seam(offset_baseline, bottom_offset_polygon, angle, im_feats  │
│    530 │   else:                                                                                 │
│ ❱  531 │   │   upper_seam = _calc_seam(offset_baseline, upper_offset_polygon, angle, im_feats)   │
│    532 │   │   bottom_seam = _calc_seam(baseline, bottom_polygon, angle, im_feats)               │
│    533 │                                                                                         │
│    534 │   upper_seam = geom.LineString(upper_seam).simplify(5)                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:462 in _calc_seam         │
│                                                                                                  │
│    459 │   │   │   │   │   │   │     line_seg[0][0],                                             │
│    460 │   │   │   │   │   │   │     line_seg[1][1],                                             │
│    461 │   │   │   │   │   │   │     line_seg[1][0])                                             │
│ ❱  462 │   │   mask[line_locs] = 0                                                               │
│    463 │   dist_bias = distance_transform_cdt(mask)                                              │
│    464 │   # absolute mask                                                                       │
│    465 │   mask = np.ones_like(patch, dtype=bool)                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: index 0 is out of bounds for axis 0 with size 0

With 2.0a1 it fails, too:

(venv3.9) stweil@notebook11 kraken % kraken --raise-on-error -d mps -a -I Bd231_qt_1_10.jpg -o .mps segment -bl ocr -m austriannewspapers.mlmodel
WARNING:root:scikit-learn version 1.1.2 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:TensorFlow version 2.9.2 has not been tested with coremltools. You may run into unexpected errors. TensorFlow 2.8.0 is the most recent version that has been tested.
WARNING:root:Torch version 1.13.0.dev20220928 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/kraken/blla.mlmodel	✓
Loading ANN austriannewspapers.mlmodel	✓
Segmenting	╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/bin/kraken:10 in <module>                    │
│                                                                                                  │
│    7                                                                                             │
│    8                                                                                             │
│    9 if __name__ == "__main__":                                                                  │
│ ❱ 10 │   sys.exit(cli())                                                                         │
│   11                                                                                             │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:11 │
│ 30 in __call__                                                                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:10 │
│ 55 in main                                                                                       │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 89 in invoke                                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 26 in _process_result                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:76 │
│ 0 in invoke                                                                                      │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:401 in process_pipeline             │
│                                                                                                  │
│   398 │   │   │   for idx, (task, input, output) in enumerate(zip(subcommands, fc, fc[1:])):     │
│   399 │   │   │   │   if len(fc) - 2 == idx:                                                     │
│   400 │   │   │   │   │   ctx.meta['last_process'] = True                                        │
│ ❱ 401 │   │   │   │   task(input=input, output=output)                                           │
│   402 │   │   #except Exception as e:                                                            │
│   403 │   │   #    logger.error(f'Failed processing {io_pair[0]}: {str(e)}')                     │
│   404 │   │   #    if ctx.meta['raise_failed']:                                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:159 in segmenter                    │
│                                                                                                  │
│   156 │   │   │   │   │   │   │   │     pad=pad,                                                 │
│   157 │   │   │   │   │   │   │   │     mask=mask)                                               │
│   158 │   │   else:                                                                              │
│ ❱ 159 │   │   │   res = blla.segment(im, text_direction, mask=mask, model=model, device=device   │
│   160 │   except Exception:                                                                      │
│   161 │   │   if ctx.meta['raise_failed']:                                                       │
│   162 │   │   │   raise                                                                          │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:299 in segment                        │
│                                                                                                  │
│   296 │   │   │   │      False: 'bottom'}[net.user_metadata['topline']]                          │
│   297 │   │   │   logger.debug(f'Baseline location: {loc}')                                      │
│   298 │   │   rets = compute_segmentation_map(im, mask, net, device)                             │
│ ❱ 299 │   │   regions = vec_regions(**rets)                                                      │
│   300 │   │   # flatten regions for line ordering/fetch bounding regions                         │
│   301 │   │   line_regs = []                                                                     │
│   302 │   │   suppl_obj = []                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:137 in vec_regions                    │
│                                                                                                  │
│   134 │   regions = {}                                                                           │
│   135 │   for region_type, idx in cls_map['regions'].items():                                    │
│   136 │   │   logger.debug(f'Vectorizing regions of type {region_type}')                         │
│ ❱ 137 │   │   regions[region_type] = vectorize_regions(heatmap[idx])                             │
│   138 │   for reg_id, regs in regions.items():                                                   │
│   139 │   │   regions[reg_id] = scale_regions(regs, scale)                                       │
│   140 │   return regions                                                                         │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:360 in vectorize_regions  │
│                                                                                                  │
│    357 │   if boundaries.type == 'Polygon':                                                      │
│    358 │   │   boundaries = [boundaries.boundary.simplify(10)]                                   │
│    359 │   else:                                                                                 │
│ ❱  360 │   │   boundaries = [x.boundary.simplify(10) for x in unary_union(boundaries)]           │
│    361 │   return [np.array(x.coords, dtype=np.uint)[:, [1, 0]].tolist() for x in boundaries]    │
│    362                                                                                           │
│    363                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: 'MultiPolygon' object is not iterable

On the way to restoring MacOS support #358

mittagessen · 2022-09-28T20:57:16Z

Thanks. One is a shapely error (fixed now), the other something else. Unfortunately, I can't reproduce the non-shapely one but it is in the polygonizer and crashes because the RoI has a size of 0 which shouldn't happen.

stweil · 2022-09-29T05:59:14Z

Shapely 2.0a1 (and same error now with 1.8.4, too):

(venv3.9) stweil@notebook11 kraken % kraken --raise-on-error -d mps -a -I Bd231_qt_1_10.jpg -o .mps segment -bl ocr -m austriannewspapers.mlmodel
WARNING:root:scikit-learn version 1.1.2 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:TensorFlow version 2.9.2 has not been tested with coremltools. You may run into unexpected errors. TensorFlow 2.8.0 is the most recent version that has been tested.
WARNING:root:Torch version 1.13.0.dev20220928 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/kraken/blla.mlmodel	✓
Loading ANN austriannewspapers.mlmodel	✓
Segmenting	╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/bin/kraken:10 in <module>                    │
│                                                                                                  │
│    7                                                                                             │
│    8                                                                                             │
│    9 if __name__ == "__main__":                                                                  │
│ ❱ 10 │   sys.exit(cli())                                                                         │
│   11                                                                                             │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:11 │
│ 30 in __call__                                                                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:10 │
│ 55 in main                                                                                       │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 89 in invoke                                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 26 in _process_result                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:76 │
│ 0 in invoke                                                                                      │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:401 in process_pipeline             │
│                                                                                                  │
│   398 │   │   │   for idx, (task, input, output) in enumerate(zip(subcommands, fc, fc[1:])):     │
│   399 │   │   │   │   if len(fc) - 2 == idx:                                                     │
│   400 │   │   │   │   │   ctx.meta['last_process'] = True                                        │
│ ❱ 401 │   │   │   │   task(input=input, output=output)                                           │
│   402 │   │   #except Exception as e:                                                            │
│   403 │   │   #    logger.error(f'Failed processing {io_pair[0]}: {str(e)}')                     │
│   404 │   │   #    if ctx.meta['raise_failed']:                                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/kraken.py:159 in segmenter                    │
│                                                                                                  │
│   156 │   │   │   │   │   │   │   │     pad=pad,                                                 │
│   157 │   │   │   │   │   │   │   │     mask=mask)                                               │
│   158 │   │   else:                                                                              │
│ ❱ 159 │   │   │   res = blla.segment(im, text_direction, mask=mask, model=model, device=device   │
│   160 │   except Exception:                                                                      │
│   161 │   │   if ctx.meta['raise_failed']:                                                       │
│   162 │   │   │   raise                                                                          │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:310 in segment                        │
│                                                                                                  │
│   307 │   │   # convert back to net scale                                                        │
│   308 │   │   suppl_obj = scale_regions(suppl_obj, 1/rets['scale'])                              │
│   309 │   │   line_regs = scale_regions(line_regs, 1/rets['scale'])                              │
│ ❱ 310 │   │   lines = vec_lines(**rets,                                                          │
│   311 │   │   │   │   │   │     regions=line_regs,                                               │
│   312 │   │   │   │   │   │     reading_order_fn=reading_order_fn,                               │
│   313 │   │   │   │   │   │     text_direction=text_direction,                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/blla.py:210 in vec_lines                      │
│                                                                                                  │
│   207 │   │   │   if reg_pol.contains(mid_point):                                                │
│   208 │   │   │   │   suppl_obj.append(regions[reg_idx])                                         │
│   209 │   │                                                                                      │
│ ❱ 210 │   │   pol = calculate_polygonal_environment(baselines=[bl[1]], im_feats=im_feats, supp   │
│   211 │   │   if pol[0] is not None:                                                             │
│   212 │   │   │   lines.append((bl[0], bl[1], pol[0]))                                           │
│   213                                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:701 in                    │
│ calculate_polygonal_environment                                                                  │
│                                                                                                  │
│    698 │   │   │                                                                                 │
│    699 │   │   │   env_up, env_bottom = _calc_roi(line, bounds, baselines[:idx] + baselines[idx  │
│    700 │   │   │                                                                                 │
│ ❱  701 │   │   │   polygons.append(_extract_patch(env_up,                                        │
│    702 │   │   │   │   │   │   │   │   │   │      env_bottom,                                    │
│    703 │   │   │   │   │   │   │   │   │   │      line.astype('int'),                            │
│    704 │   │   │   │   │   │   │   │   │   │      offset_line.astype('int'),                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:531 in _extract_patch     │
│                                                                                                  │
│    528 │   │   upper_seam = _calc_seam(baseline, upper_polygon, angle, im_feats)                 │
│    529 │   │   bottom_seam = _calc_seam(offset_baseline, bottom_offset_polygon, angle, im_feats  │
│    530 │   else:                                                                                 │
│ ❱  531 │   │   upper_seam = _calc_seam(offset_baseline, upper_offset_polygon, angle, im_feats)   │
│    532 │   │   bottom_seam = _calc_seam(baseline, bottom_polygon, angle, im_feats)               │
│    533 │                                                                                         │
│    534 │   upper_seam = geom.LineString(upper_seam).simplify(5)                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/kraken/lib/segmentation.py:462 in _calc_seam         │
│                                                                                                  │
│    459 │   │   │   │   │   │   │     line_seg[0][0],                                             │
│    460 │   │   │   │   │   │   │     line_seg[1][1],                                             │
│    461 │   │   │   │   │   │   │     line_seg[1][0])                                             │
│ ❱  462 │   │   mask[line_locs] = 0                                                               │
│    463 │   dist_bias = distance_transform_cdt(mask)                                              │
│    464 │   # absolute mask                                                                       │
│    465 │   mask = np.ones_like(patch, dtype=bool)                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: index 0 is out of bounds for axis 0 with size 0

mittagessen · 2022-09-29T11:35:02Z

I'll have to find an Apple system to figure out why it fails as I can't reproduce it on any of my Linux machines.

stweil · 2022-11-05T07:47:27Z

The latest code still fails with Shapely-1.8.4 and Shapely-2.0b2, but now with a different error:

(venv3.9) stweil@notebook11 kraken % time kraken --raise-on-error -d mps -a -I Bd231_qt_1_10.jpg -o .mps segment -bl ocr -m austriannewspapers.mlmodel
WARNING:root:scikit-learn version 1.1.2 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:TensorFlow version 2.9.2 has not been tested with coremltools. You may run into unexpected errors. TensorFlow 2.8.0 is the most recent version that has been tested.
WARNING:root:Torch version 1.13.0.dev20220928 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/blla.mlmodel ✓
Loading ANN austriannewspapers.mlmodel  ✓
Segmenting      
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/bin/kraken:8 in <module>                     │
│                                                                                                  │
│   5 from kraken.kraken import cli                                                                │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(cli())                                                                          │
│   9                                                                                              │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:11 │
│ 30 in __call__                                                                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:10 │
│ 55 in main                                                                                       │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 89 in invoke                                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 26 in _process_result                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:76 │
│ 0 in invoke                                                                                      │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/kraken.py │
│ :401 in process_pipeline                                                                         │
│                                                                                                  │
│   398 │   │   │   for idx, (task, input, output) in enumerate(zip(subcommands, fc, fc[1:])):     │
│   399 │   │   │   │   if len(fc) - 2 == idx:                                                     │
│   400 │   │   │   │   │   ctx.meta['last_process'] = True                                        │
│ ❱ 401 │   │   │   │   task(input=input, output=output)                                           │
│   402 │   │   #except Exception as e:                                                            │
│   403 │   │   #    logger.error(f'Failed processing {io_pair[0]}: {str(e)}')                     │
│   404 │   │   #    if ctx.meta['raise_failed']:                                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/kraken.py │
│ :159 in segmenter                                                                                │
│                                                                                                  │
│   156 │   │   │   │   │   │   │   │     pad=pad,                                                 │
│   157 │   │   │   │   │   │   │   │     mask=mask)                                               │
│   158 │   │   else:                                                                              │
│ ❱ 159 │   │   │   res = blla.segment(im, text_direction, mask=mask, model=model, device=device   │
│   160 │   except Exception:                                                                      │
│   161 │   │   if ctx.meta['raise_failed']:                                                       │
│   162 │   │   │   raise                                                                          │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/blla.py:3 │
│ 10 in segment                                                                                    │
│                                                                                                  │
│   307 │   │   # convert back to net scale                                                        │
│   308 │   │   suppl_obj = scale_regions(suppl_obj, 1/rets['scale'])                              │
│   309 │   │   line_regs = scale_regions(line_regs, 1/rets['scale'])                              │
│ ❱ 310 │   │   lines = vec_lines(**rets,                                                          │
│   311 │   │   │   │   │   │     regions=line_regs,                                               │
│   312 │   │   │   │   │   │     reading_order_fn=reading_order_fn,                               │
│   313 │   │   │   │   │   │     text_direction=text_direction,                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/blla.py:2 │
│ 10 in vec_lines                                                                                  │
│                                                                                                  │
│   207 │   │   │   if reg_pol.contains(mid_point):                                                │
│   208 │   │   │   │   suppl_obj.append(regions[reg_idx])                                         │
│   209 │   │                                                                                      │
│ ❱ 210 │   │   pol = calculate_polygonal_environment(baselines=[bl[1]], im_feats=im_feats, supp   │
│   211 │   │   if pol[0] is not None:                                                             │
│   212 │   │   │   lines.append((bl[0], bl[1], pol[0]))                                           │
│   213                                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/segme │
│ ntation.py:699 in calculate_polygonal_environment                                                │
│                                                                                                  │
│    696 │   │   │   p_dir = np.mean(np.diff(line.T) * lengths/lengths.sum(), axis=1)              │
│    697 │   │   │   p_dir = (p_dir.T / np.sqrt(np.sum(p_dir**2, axis=-1)))                        │
│    698 │   │   │                                                                                 │
│ ❱  699 │   │   │   env_up, env_bottom = _calc_roi(line, bounds, baselines[:idx] + baselines[idx  │
│    700 │   │   │                                                                                 │
│    701 │   │   │   polygons.append(_extract_patch(env_up,                                        │
│    702 │   │   │   │   │   │   │   │   │   │      env_bottom,                                    │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/segme │
│ ntation.py:608 in _calc_roi                                                                      │
│                                                                                                  │
│    605 │   env_bottom = []                                                                       │
│    606 │   # find orthogonal (to linear regression) intersects with adjacent objects to complet  │
│    607 │   for point, upper_bounds_intersect, bottom_bounds_intersect in zip(ip_line, upper_bou  │
│ ❱  608 │   │   upper_limit = _find_closest_point(point, geom.LineString(                         │
│    609 │   │   │   [point, upper_bounds_intersect]).intersection(side_a))                        │
│    610 │   │   bottom_limit = _find_closest_point(point, geom.LineString(                        │
│    611 │   │   │   [point, bottom_bounds_intersect]).intersection(side_b))                       │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/segme │
│ ntation.py:602 in _find_closest_point                                                            │
│                                                                                                  │
│    599 │   │   │   else:                                                                         │
│    600 │   │   │   │   return nearest_points(spt, t)[1]                                          │
│    601 │   │   else:                                                                             │
│ ❱  602 │   │   │   raise Exception(f'No intersection with boundaries. Shapely intersection obje  │
│    603 │                                                                                         │
│    604 │   env_up = []                                                                           │
│    605 │   env_bottom = []                                                                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: No intersection with boundaries. Shapely intersection object: LINESTRING (295 3.98529745, 295 1)
kraken --raise-on-error -d mps -a -I Bd231_qt_1_10.jpg -o .mps segment -bl oc  82,13s user 5,33s system 105% cpu 1:23,21 total

stweil · 2022-11-05T08:01:31Z

Running with CPU and Shapely-2.0b2 now fails, too (regression?):

(venv3.9) stweil@notebook11 kraken % time kraken --raise-on-error -d cpu -a -I Bd231_qt_1_10.jpg -o .cpu segment -bl ocr -m austriannewspapers.mlmodel
WARNING:root:scikit-learn version 1.1.2 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:TensorFlow version 2.9.2 has not been tested with coremltools. You may run into unexpected errors. TensorFlow 2.8.0 is the most recent version that has been tested.
WARNING:root:Torch version 1.14.0.dev20221104 has not been tested with coremltools. You may run into unexpected errors. Torch 1.11.0 is the most recent version that has been tested.
Loading ANN /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/blla.mlmodel ✓
Loading ANN austriannewspapers.mlmodel  ✓
Segmenting      
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/bin/kraken:8 in <module>                     │
│                                                                                                  │
│   5 from kraken.kraken import cli                                                                │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(cli())                                                                          │
│   9                                                                                              │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:11 │
│ 30 in __call__                                                                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:10 │
│ 55 in main                                                                                       │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 89 in invoke                                                                                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:16 │
│ 26 in _process_result                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/click/core.py:76 │
│ 0 in invoke                                                                                      │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/kraken.py │
│ :401 in process_pipeline                                                                         │
│                                                                                                  │
│   398 │   │   │   for idx, (task, input, output) in enumerate(zip(subcommands, fc, fc[1:])):     │
│   399 │   │   │   │   if len(fc) - 2 == idx:                                                     │
│   400 │   │   │   │   │   ctx.meta['last_process'] = True                                        │
│ ❱ 401 │   │   │   │   task(input=input, output=output)                                           │
│   402 │   │   #except Exception as e:                                                            │
│   403 │   │   #    logger.error(f'Failed processing {io_pair[0]}: {str(e)}')                     │
│   404 │   │   #    if ctx.meta['raise_failed']:                                                  │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/kraken.py │
│ :159 in segmenter                                                                                │
│                                                                                                  │
│   156 │   │   │   │   │   │   │   │     pad=pad,                                                 │
│   157 │   │   │   │   │   │   │   │     mask=mask)                                               │
│   158 │   │   else:                                                                              │
│ ❱ 159 │   │   │   res = blla.segment(im, text_direction, mask=mask, model=model, device=device   │
│   160 │   except Exception:                                                                      │
│   161 │   │   if ctx.meta['raise_failed']:                                                       │
│   162 │   │   │   raise                                                                          │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/blla.py:3 │
│ 10 in segment                                                                                    │
│                                                                                                  │
│   307 │   │   # convert back to net scale                                                        │
│   308 │   │   suppl_obj = scale_regions(suppl_obj, 1/rets['scale'])                              │
│   309 │   │   line_regs = scale_regions(line_regs, 1/rets['scale'])                              │
│ ❱ 310 │   │   lines = vec_lines(**rets,                                                          │
│   311 │   │   │   │   │   │     regions=line_regs,                                               │
│   312 │   │   │   │   │   │     reading_order_fn=reading_order_fn,                               │
│   313 │   │   │   │   │   │     text_direction=text_direction,                                   │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/blla.py:2 │
│ 10 in vec_lines                                                                                  │
│                                                                                                  │
│   207 │   │   │   if reg_pol.contains(mid_point):                                                │
│   208 │   │   │   │   suppl_obj.append(regions[reg_idx])                                         │
│   209 │   │                                                                                      │
│ ❱ 210 │   │   pol = calculate_polygonal_environment(baselines=[bl[1]], im_feats=im_feats, supp   │
│   211 │   │   if pol[0] is not None:                                                             │
│   212 │   │   │   lines.append((bl[0], bl[1], pol[0]))                                           │
│   213                                                                                            │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/segme │
│ ntation.py:701 in calculate_polygonal_environment                                                │
│                                                                                                  │
│    698 │   │   │                                                                                 │
│    699 │   │   │   env_up, env_bottom = _calc_roi(line, bounds, baselines[:idx] + baselines[idx  │
│    700 │   │   │                                                                                 │
│ ❱  701 │   │   │   polygons.append(_extract_patch(env_up,                                        │
│    702 │   │   │   │   │   │   │   │   │   │      env_bottom,                                    │
│    703 │   │   │   │   │   │   │   │   │   │      line.astype('int'),                            │
│    704 │   │   │   │   │   │   │   │   │   │      offset_line.astype('int'),                     │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/kraken/lib/segme │
│ ntation.py:551 in _extract_patch                                                                 │
│                                                                                                  │
│    548 │   # offsetting might produce bounds outside the image. Clip it to the image bounds.     │
│    549 │   polygon = np.concatenate(([end_points[0]], upper_seam, [end_points[-1]], bottom_seam  │
│    550 │   polygon = geom.Polygon(polygon)                                                       │
│ ❱  551 │   polygon = np.array(roi_polygon.intersection(polygon).boundary.coords, dtype=int)      │
│    552 │   return polygon                                                                        │
│    553                                                                                           │
│    554                                                                                           │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/shapely/geometry │
│ /base.py:570 in intersection                                                                     │
│                                                                                                  │
│   567 │   │                                                                                      │
│   568 │   │   Refer to `shapely.intersection` for full documentation.                            │
│   569 │   │   """                                                                                │
│ ❱ 570 │   │   return shapely.intersection(self, other, grid_size=grid_size)                      │
│   571 │                                                                                          │
│   572 │   def symmetric_difference(self, other, grid_size=None):                                 │
│   573 │   │   """                                                                                │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/shapely/decorato │
│ rs.py:77 in wrapped                                                                              │
│                                                                                                  │
│   74 │   │   try:                                                                                │
│   75 │   │   │   for arr in array_args:                                                          │
│   76 │   │   │   │   arr.flags.writeable = False                                                 │
│ ❱ 77 │   │   │   return func(*args, **kwargs)                                                    │
│   78 │   │   finally:                                                                            │
│   79 │   │   │   for arr, old_flag in zip(array_args, old_flags):                                │
│   80 │   │   │   │   arr.flags.writeable = old_flag                                              │
│                                                                                                  │
│ /Users/stweil/src/github/mittagessen/kraken/venv3.9/lib/python3.9/site-packages/shapely/set_oper │
│ ations.py:133 in intersection                                                                    │
│                                                                                                  │
│   130 │   │                                                                                      │
│   131 │   │   return lib.intersection_prec(a, b, grid_size, **kwargs)                            │
│   132 │                                                                                          │
│ ❱ 133 │   return lib.intersection(a, b, **kwargs)                                                │
│   134                                                                                            │
│   135                                                                                            │
│   136 @multithreading_enabled                                                                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
GEOSException: TopologyException: Input geom 1 is invalid: Self-intersection at 1343 290
kraken --raise-on-error -d cpu -a -I Bd231_qt_1_10.jpg -o .mps segment -bl oc  18,49s user 2,90s system 132% cpu 16,140 total

The same test works with Shapely 1.8.5.post1.

bencomp · 2023-03-08T00:00:51Z

My latest attempt to train using MPS failed because PyTorch doesn't yet support CTC layers in MPS. The error message referred to a PyTorch issue, inviting to ask for support of specific features.

stweil · 2024-01-02T10:01:25Z

Update: latest pytorch still has the same problem:

NotImplementedError: The operator 'aten::_ctc_loss' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype 
phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable 
`PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

With PYTORCH_ENABLE_MPS_FALLBACK=1 training seems to work, but it is much slower than a training with CPU only:

% ketos train -f page -t list.eval -d mps
GPU available: True (mps), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
┏━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃    ┃ Name      ┃ Type                     ┃ Params ┃                 In sizes ┃                Out sizes ┃
┡━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 0  │ val_cer   │ CharErrorRate            │      0 │                        ? │                        ? │
│ 1  │ val_wer   │ WordErrorRate            │      0 │                        ? │                        ? │
│ 2  │ net       │ MultiParamSequential     │  4.0 M │  [[1, 1, 120, 400], '?'] │   [[1, 121, 1, 50], '?'] │
│ 3  │ net.C_0   │ ActConv2D                │  1.3 K │  [[1, 1, 120, 400], '?'] │ [[1, 32, 120, 400], '?'] │
│ 4  │ net.Do_1  │ Dropout                  │      0 │ [[1, 32, 120, 400], '?'] │ [[1, 32, 120, 400], '?'] │
│ 5  │ net.Mp_2  │ MaxPool                  │      0 │ [[1, 32, 120, 400], '?'] │  [[1, 32, 60, 200], '?'] │
│ 6  │ net.C_3   │ ActConv2D                │ 40.0 K │  [[1, 32, 60, 200], '?'] │  [[1, 32, 60, 200], '?'] │
│ 7  │ net.Do_4  │ Dropout                  │      0 │  [[1, 32, 60, 200], '?'] │  [[1, 32, 60, 200], '?'] │
│ 8  │ net.Mp_5  │ MaxPool                  │      0 │  [[1, 32, 60, 200], '?'] │  [[1, 32, 30, 100], '?'] │
│ 9  │ net.C_6   │ ActConv2D                │ 55.4 K │  [[1, 32, 30, 100], '?'] │  [[1, 64, 30, 100], '?'] │
│ 10 │ net.Do_7  │ Dropout                  │      0 │  [[1, 64, 30, 100], '?'] │  [[1, 64, 30, 100], '?'] │
│ 11 │ net.Mp_8  │ MaxPool                  │      0 │  [[1, 64, 30, 100], '?'] │   [[1, 64, 15, 50], '?'] │
│ 12 │ net.C_9   │ ActConv2D                │  110 K │   [[1, 64, 15, 50], '?'] │   [[1, 64, 15, 50], '?'] │
│ 13 │ net.Do_10 │ Dropout                  │      0 │   [[1, 64, 15, 50], '?'] │   [[1, 64, 15, 50], '?'] │
│ 14 │ net.S_11  │ Reshape                  │      0 │   [[1, 64, 15, 50], '?'] │   [[1, 960, 1, 50], '?'] │
│ 15 │ net.L_12  │ TransposedSummarizingRNN │  1.9 M │   [[1, 960, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 16 │ net.Do_13 │ Dropout                  │      0 │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 17 │ net.L_14  │ TransposedSummarizingRNN │  963 K │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 18 │ net.Do_15 │ Dropout                  │      0 │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 19 │ net.L_16  │ TransposedSummarizingRNN │  963 K │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 20 │ net.Do_17 │ Dropout                  │      0 │   [[1, 400, 1, 50], '?'] │   [[1, 400, 1, 50], '?'] │
│ 21 │ net.O_18  │ LinSoftmax               │ 48.5 K │   [[1, 400, 1, 50], '?'] │   [[1, 121, 1, 50], '?'] │
└────┴───────────┴──────────────────────────┴────────┴──────────────────────────┴──────────────────────────┘
Trainable params: 4.0 M                                                                                                                                                
Non-trainable params: 0                                                                                                                                                
Total params: 4.0 M                                                                                                                                                    
Total estimated model params size (MB): 16                                                                                                                             
stage 0/∞ ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175/4514 0:09:17 • 2:45:22 0.44it/s  early_stopping: 0/10 -inf

stweil · 2024-03-18T07:18:44Z

Update: ketos train -f page -t list.eval -d mps still complains about the missing MPS implementation of aten::_ctc_loss.

stweil · 2024-05-08T08:31:00Z

@mittagessen, I suggest to re-open this feature request because MPS support is still incomplete (not sufficient for a training).

stweil · 2024-05-08T14:12:19Z

Related pytorch issue: pytorch/pytorch#77764 (comment).

mittagessen closed this as completed Aug 29, 2022

mittagessen added a commit that referenced this issue Sep 28, 2022

Crash on shapely 2.0a1 in region vectorizer

10a320e

On the way to restoring MacOS support #358

mittagessen reopened this May 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature request] Training with Metal Performance Shaders #358

[feature request] Training with Metal Performance Shaders #358

stweil commented May 20, 2022 •

edited

Loading

mittagessen commented Aug 29, 2022

stweil commented Aug 29, 2022 •

edited

Loading

mittagessen commented Aug 29, 2022

stweil commented Aug 29, 2022 •

edited

Loading

mittagessen commented Aug 29, 2022

stweil commented Aug 29, 2022 •

edited

Loading

stweil commented Sep 28, 2022 •

edited

Loading

mittagessen commented Sep 28, 2022

stweil commented Sep 29, 2022 •

edited

Loading

mittagessen commented Sep 29, 2022

stweil commented Nov 5, 2022 •

edited

Loading

stweil commented Nov 5, 2022 •

edited

Loading

bencomp commented Mar 8, 2023

stweil commented Jan 2, 2024

stweil commented Mar 18, 2024

stweil commented May 8, 2024

stweil commented May 8, 2024

[feature request] Training with Metal Performance Shaders #358

[feature request] Training with Metal Performance Shaders #358

Comments

stweil commented May 20, 2022 • edited Loading

kraken<4.0.0

kraken>=4.0.0

mittagessen commented Aug 29, 2022

stweil commented Aug 29, 2022 • edited Loading

mittagessen commented Aug 29, 2022

stweil commented Aug 29, 2022 • edited Loading

mittagessen commented Aug 29, 2022

stweil commented Aug 29, 2022 • edited Loading

stweil commented Sep 28, 2022 • edited Loading

mittagessen commented Sep 28, 2022

stweil commented Sep 29, 2022 • edited Loading

mittagessen commented Sep 29, 2022

stweil commented Nov 5, 2022 • edited Loading

stweil commented Nov 5, 2022 • edited Loading

bencomp commented Mar 8, 2023

stweil commented Jan 2, 2024

stweil commented Mar 18, 2024

stweil commented May 8, 2024

stweil commented May 8, 2024

stweil commented May 20, 2022 •

edited

Loading

stweil commented Aug 29, 2022 •

edited

Loading

stweil commented Aug 29, 2022 •

edited

Loading

stweil commented Aug 29, 2022 •

edited

Loading

stweil commented Sep 28, 2022 •

edited

Loading

stweil commented Sep 29, 2022 •

edited

Loading

stweil commented Nov 5, 2022 •

edited

Loading

stweil commented Nov 5, 2022 •

edited

Loading