inseq assertion error

[qiaw99]

Prompt:

show me the 5 most important features for data point 730 by attention

Error Track:

  File "C:\Users\87290\DFKI\LLMCheckup\flask_app.py", line 267, in get_bot_response
    response = BOT.update_state(user_text, conversation)
  File "C:\Users\87290\DFKI\LLMCheckup\logic\core.py", line 536, in update_state
    returned_item = run_action(
  File "C:\Users\87290\DFKI\LLMCheckup\logic\action.py", line 49, in run_action
    action_return, action_status = actions[p_text](
  File "C:\Users\87290\DFKI\LLMCheckup\actions\explanation\feature_importance.py", line 130, in feature_importance_operation
    out_agg = out.aggregate(inseq.data.aggregator.SubwordAggregator)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\attribution.py", line 617, in aggregate
    aggregated.sequence_attributions[idx] = seq.aggregate(aggregator, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 249, in aggregate
    return aggregator.aggregate(
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 720, in aggregate
    return super().aggregate(attr, source_spans=source_spans, target_spans=target_spans, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 546, in aggregate
    return super().aggregate(attr, source_spans=source_spans, target_spans=target_spans, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 102, in aggregate
    cls.post_aggregate_hook(aggregated, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 337, in post_aggregate_hook
    cls.is_compatible(attr)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 432, in is_compatible
    assert attr.target_attributions.shape[1] == attr.attr_pos_end - attr.attr_pos_start
AssertionError

prompt:

primary features of data point 3444 by input gradient

  File "C:\Users\87290\DFKI\LLMCheckup\flask_app.py", line 267, in get_bot_response
    response = BOT.update_state(user_text, conversation)
  File "C:\Users\87290\DFKI\LLMCheckup\logic\core.py", line 536, in update_state
    returned_item = run_action(
  File "C:\Users\87290\DFKI\LLMCheckup\logic\action.py", line 49, in run_action
    action_return, action_status = actions[p_text](
  File "C:\Users\87290\DFKI\LLMCheckup\actions\explanation\feature_importance.py", line 120, in feature_importance_operation
    out = inseq_model.attribute(
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\models\attribution_model.py", line 445, in attribute
    attribution_outputs = attribution_method.prepare_and_attribute(
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\attr\attribution_decorators.py", line 71, in batched_wrapper
    out = f(self, *args, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\attr\feat\feature_attribution.py", line 237, in prepare_and_attribute
    attribution_output = self.attribute(
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\attr\feat\feature_attribution.py", line 372, in attribute
    attr_pos_start, attr_pos_end = check_attribute_positions(
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\attr\feat\attribution_utils.py", line 85, in check_attribute_positions
    raise ValueError("Start and end attribution positions cannot be the same.")
ValueError: Start and end attribution positions cannot be the same.

[nfelnlp]

The SubwordAggregator problem is being discussed here: inseq-team/inseq#246

[nfelnlp]

First issue is finally fixed.

For the second prompt, I get the CUDA error all the time. I can't reproduce the ValueError you posted, @qiaw99 . Can you check again, please? Maybe the CUDA exception is too general and should be caught elsewhere.

[qiaw99]

sure, I will check it again.

[qiaw99]

somehow the first issue is still there.
My input: most significant feature for instance 2322 by attention

Get error:

 File "C:\Users\87290\DFKI\LLMCheckup\flask_app.py", line 273, in get_bot_response
    response = BOT.update_state(user_text, conversation)
  File "C:\Users\87290\DFKI\LLMCheckup\logic\core.py", line 593, in update_state
    returned_item = run_action(
  File "C:\Users\87290\DFKI\LLMCheckup\logic\action.py", line 49, in run_action
    action_return, action_status = actions[p_text](
  File "C:\Users\87290\DFKI\LLMCheckup\actions\explanation\feature_importance.py", line 136, in feature_importance_operation
    out_agg = out.aggregate(SubwordAggregator).aggregate()
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\attribution.py", line 617, in aggregate
    aggregated.sequence_attributions[idx] = seq.aggregate(aggregator, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 249, in aggregate
    return aggregator.aggregate(
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 720, in aggregate
    return super().aggregate(attr, source_spans=source_spans, target_spans=target_spans, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 546, in aggregate
    return super().aggregate(attr, source_spans=source_spans, target_spans=target_spans, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 102, in aggregate
    cls.post_aggregate_hook(aggregated, **kwargs)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 337, in post_aggregate_hook
    cls.is_compatible(attr)
  File "C:\Users\87290\anaconda3\envs\llm\lib\site-packages\inseq\data\aggregator.py", line 432, in is_compatible
    assert attr.target_attributions.shape[1] == attr.attr_pos_end - attr.attr_pos_start
AssertionError
```

[tanikina]

Hi @qiaw99 , have you tried re-installing inseq? For me this solved the issue :)
I think there were some recent updates in the inseq repository that affect aggregation: inseq-team/inseq@7503576

[qiaw99]

Ah right, I forget to reinstall Inseq. I will try it :) thanks!

[qiaw99]

Both issues are resolved by using updated inseq. I think we should list 0.6.0-dev in requirements.txt