Merge remote-tracking branch 'origin/main' into alti

* origin/main:
  Attributed behavior for contrastive step functions (#228)
  Fix command for installing pre-commit hooks. (#229)
  Remove `max_input_length` from `model.encode` (#227)
  Migrate to `ruff format` (#225)
  Remove contrast_target_prefixes from contrastive step functions (#224)
  Step functions fixes, add `in_context_pvi` (#223)
  Format fixes, add Attanasio et al. (2023) to readme
  Add Sequential IG method (#222)
  Fix LIME and Occlusion outputs (#220)
  Update citation information
  Bump dependencies
  Add end_pos for contrast_targets_alignments
  Fix dummy output viz in console
  Minor fixes

.pre-commit-config.yaml

@@ -18,17 +18,10 @@ repos:
       - id: end-of-file-fixer
         exclude: LICENSE
 
-  - repo: local
-    hooks:
-      - id: black
-        name: black
-        entry: poetry run black --config pyproject.toml
-        types: [python]
-        language: system
-
-  - repo: https://github.com/charliermarsh/ruff-pre-commit
-    rev: 'v0.0.267'
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.1.2
     hooks:
+      - id: ruff-format
       - id: ruff
 
   - repo: local

CITATION.cff

@@ -17,8 +17,10 @@ authors:
     orcid: "https://orcid.org/0000-0003-4087-3714"
   - family-names: Nissim
     given-names: Malvina
+    orcid: "https://orcid.org/0000-0001-5289-0971"
   - family-names: Bisazza
     given-names: Arianna
+    orcid: "https://orcid.org/0000-0003-1270-3048"
 preferred-citation:
   type: article
   authors:
@@ -34,11 +36,16 @@ preferred-citation:
     orcid: "https://orcid.org/0000-0003-4087-3714"
   - family-names: Nissim
     given-names: Malvina
+    orcid: "https://orcid.org/0000-0001-5289-0971"
   - family-names: Bisazza
     given-names: Arianna
-  journal: "ArXiv"
-  month: 2
+    orcid: "https://orcid.org/0000-0003-1270-3048"
+  booktitle: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)"
+  month: 7
   title: "Inseq: An Interpretability Toolkit for Sequence Generation Models"
   year: 2023
-  url: "http://arxiv.org/abs/2302.13942"
-  volume: "abs/2302.13942"
+  url: "https://aclanthology.org/2023.acl-demo.40"
+  start: 421
+  end: 435
+  publisher: "Association for Computational Linguistics"
+  address: "Toronto, Canada"

CONTRIBUTING.md

@@ -9,7 +9,7 @@ To install dependencies and prepare [`pre-commit`](https://pre-commit.com/) hook
 
 ```bash
 make install
-make pre-commit-install
+make install-dev
 ```
 
 To activate your `virtualenv` run `poetry shell`.
@@ -24,7 +24,7 @@ make lint
 
 ### Checks
 
-Many checks are configured for this project. Command `make check-style` will check black and isort.
+Many checks are configured for this project. Command `make check-style` will check style with `ruff`.
 The `make check-safety` command will look at the security of your code.
 
 Comand `make lint` applies all checks.

Makefile

@@ -78,15 +78,15 @@ update-deps:
 #* Linting
 .PHONY: check-style
 check-style:
-	poetry run black --diff --check --config pyproject.toml ./
-	poetry run ruff  --no-fix --config pyproject.toml ./
+	poetry run ruff format --check --config pyproject.toml ./
+	poetry run ruff check --no-fix --config pyproject.toml ./
 #   poetry run darglint --verbosity 2 inseq tests
 #	poetry run mypy --config-file pyproject.toml ./
 
 .PHONY: fix-style
 fix-style:
-	poetry run black --config pyproject.toml ./
-	poetry run ruff --config pyproject.toml ./
+	poetry run ruff format --config pyproject.toml ./
+	poetry run ruff check --config pyproject.toml ./
 
 .PHONY: check-safety
 check-safety:

README.md

@@ -130,41 +130,44 @@ Use the `inseq.list_feature_attribution_methods` function to list all available
 
 #### Gradient-based attribution
 
-- `saliency`: [Saliency](https://arxiv.org/abs/1312.6034) (Simonyan et al., 2013)
+- `saliency`: [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://arxiv.org/abs/1312.6034) (Simonyan et al., 2013)
 
-- `input_x_gradient`: [Input x Gradient](https://arxiv.org/abs/1312.6034) (Simonyan et al., 2013)
+- `input_x_gradient`: [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://arxiv.org/abs/1312.6034) (Simonyan et al., 2013)
 
-- `integrated_gradients`: [Integrated Gradients](https://arxiv.org/abs/1703.01365) (Sundararajan et al., 2017)
+- `integrated_gradients`: [Axiomatic Attribution for Deep Networks](https://arxiv.org/abs/1703.01365) (Sundararajan et al., 2017)
 
-- `deeplift`: [DeepLIFT](https://arxiv.org/abs/1704.02685) (Shrikumar et al., 2017)
+- `deeplift`: [Learning Important Features Through Propagating Activation Differences](https://arxiv.org/abs/1704.02685) (Shrikumar et al., 2017)
 
-- `gradient_shap`: [Gradient SHAP](https://dl.acm.org/doi/10.5555/3295222.3295230) (Lundberg and Lee, 2017)
+- `gradient_shap`: [A unified approach to interpreting model predictions](https://dl.acm.org/doi/10.5555/3295222.3295230) (Lundberg and Lee, 2017)
 
-- `discretized_integrated_gradients`: [Discretized Integrated Gradients](https://aclanthology.org/2021.emnlp-main.805/) (Sanyal and Ren, 2021)
+- `discretized_integrated_gradients`: [Discretized Integrated Gradients for Explaining Language Models](https://aclanthology.org/2021.emnlp-main.805/) (Sanyal and Ren, 2021)
+
+- `sequential_integrated_gradients`: [Sequential Integrated Gradients: a simple but effective method for explaining language models](https://aclanthology.org/2023.findings-acl.477/) (Enguehard, 2023)
 
 #### Internals-based attribution
 
-- `attention`: [Attention Weight Attribution](https://arxiv.org/abs/1409.0473) (Bahdanau et al., 2014)
+- `attention`: Attention Weight Attribution, from [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473) (Bahdanau et al., 2014)
 
 #### Perturbation-based attribution
 
-- `occlusion`: [Occlusion](https://link.springer.com/chapter/10.1007/978-3-319-10590-1_53) (Zeiler and Fergus, 2014)
+- `occlusion`: [Visualizing and Understanding Convolutional Networks](https://link.springer.com/chapter/10.1007/978-3-319-10590-1_53) (Zeiler and Fergus, 2014)
 
-- `lime`: [LIME](https://arxiv.org/abs/1602.04938) (Ribeiro et al., 2016)
+- `lime`: ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://arxiv.org/abs/1602.04938) (Ribeiro et al., 2016)
 
 #### Step functions
 
 Step functions are used to extract custom scores from the model at each step of the attribution process with the `step_scores` argument in `model.attribute`. They can also be used as targets for attribution methods relying on model outputs (e.g. gradient-based methods) by passing them as the `attributed_fn` argument. The following step functions are currently supported:
 
 - `logits`: Logits of the target token.
-- `probability`: Probability of the target token.
+- `probability`: Probability of the target token. Can also be used for log-probability by passing `logprob=True`.
 - `entropy`: Entropy of the predictive distribution.
 - `crossentropy`: Cross-entropy loss between target token and predicted distribution.
 - `perplexity`: Perplexity of the target token.
-- `contrast_prob`: Probability of the target token when different contrastive inputs are provided to the model. Equivalent to `probability` when no contrastive inputs are provided.
+- `contrast_logits`/`contrast_prob`: Logits/probabilities of the target token when different contrastive inputs are provided to the model. Equivalent to `logits`/`probability` when no contrastive inputs are provided.
+- `contrast_logits_diff`/`contrast_prob_diff`: Difference in logits/probability between original and foil target tokens pair, can be used for contrastive evaluation as in [contrastive attribution](https://aclanthology.org/2022.emnlp-main.14/) (Yin and Neubig, 2022).
 - `pcxmi`: Point-wise Contextual Cross-Mutual Information (P-CXMI) for the target token given original and contrastive contexts [(Yin et al. 2021)](https://arxiv.org/abs/2109.07446).
 - `kl_divergence`: KL divergence of the predictive distribution given original and contrastive contexts. Can be restricted to most likely target token options using the `top_k` and `top_p` parameters.
-- `contrast_prob_diff`: Difference in probability between original and foil target tokens pair, can be used for contrastive evaluation as in [Contrastive Attribution](https://aclanthology.org/2022.emnlp-main.14/) (Yin and Neubig, 2022).
+- `in_context_pvi`: In-context Pointwise V-usable Information (PVI) to measure the amount of contextual information used in model predictions [(Lu et al. 2023)](https://arxiv.org/abs/2310.12300).
 - `mc_dropout_prob_avg`: Average probability of the target token across multiple samples using [MC Dropout](https://arxiv.org/abs/1506.02142) (Gal and Ghahramani, 2016).
 - `top_p_size`: The number of tokens with cumulative probability greater than `top_p` in the predictive distribution of the model.
 
@@ -235,15 +238,24 @@ Our vision for Inseq is to create a centralized, comprehensive and robust set of
 If you use Inseq in your research we suggest to include a mention to the specific release (e.g. v0.4.0) and we kindly ask you to cite our reference paper as:
 
 ```bibtex
-@article{sarti-etal-2023-inseq,
-    author = {Gabriele Sarti and Nils Feldhus and Ludwig Sickert and Oskar van der Wal and Malvina Nissim and Arianna Bisazza},
-    title = {Inseq: An Interpretability Toolkit for Sequence Generation Models},
-    month = feb,
-    year = 2023,
-    journal = {ArXiv},
-    volume = {abs/2302.13942},
-    url = {https://arxiv.org/abs/2302.13942}
+@inproceedings{sarti-etal-2023-inseq,
+    title = "Inseq: An Interpretability Toolkit for Sequence Generation Models",
+    author = "Sarti, Gabriele  and
+      Feldhus, Nils  and
+      Sickert, Ludwig  and
+      van der Wal, Oskar and
+      Nissim, Malvina and
+      Bisazza, Arianna",
+    booktitle = "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
+    month = jul,
+    year = "2023",
+    address = "Toronto, Canada",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2023.acl-demo.40",
+    doi = "10.18653/v1/2023.acl-demo.40",
+    pages = "421--435",
 }
+
 ```
 
 ## Research using Inseq
@@ -253,9 +265,11 @@ Inseq has been used in various research projects. A list of known publications t
 <details>
   <summary><b>2023</b></summary>
   <ol>
-    <li> <a href="https://arxiv.org/abs/2302.13942">Inseq: An Interpretability Toolkit for Sequence Generation Models</a> (Sarti et al., 2023) </li>
+    <li> <a href="https://aclanthology.org/2023.acl-demo.40/">Inseq: An Interpretability Toolkit for Sequence Generation Models</a> (Sarti et al., 2023) </li>
     <li> <a href="https://arxiv.org/abs/2302.14220">Are Character-level Translations Worth the Wait? Comparing Character- and Subword-level Models for Machine Translation</a> (Edman et al., 2023) </li>
-    <li> <a href="https://arxiv.org/abs/2305.15908">Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?</a> (Mousavi et al., 2023)  </li>
+    <li> <a href="https://aclanthology.org/2023.nlp4convai-1.1/">Response Generation in Longitudinal Dialogues: Which Knowledge Representation Helps?</a> (Mousavi et al., 2023)  </li>
+    <li> <a href="https://arxiv.org/abs/2310.01188">Quantifying the Plausibility of Context Reliance in Neural Machine Translation</a> (Sarti et al., 2023)</li>
+    <li> <a href="https://arxiv.org/abs/2310.12127">A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation</a> (Attanasio et al., 2023)</li>
   </ol>
 
 </details>

docs/source/examples/custom_attribute_target.rst

@@ -40,7 +40,7 @@ with Contrastive Explanations" <https://arxiv.org/abs/2202.10419>`__ by Yin and
 by complementing the output probabilities with the ones from their contrastive counterpart, and using the difference between the two as attribution
 target.
 
-We can define such attribution function using the standard template adopted by Inseq. The :class:`~inseq.attr.step_functions.StepFunctionArgs` class is used for convenience to encapsulate all default arguments passed to step functions, namely:
+We can define such attribution function using the standard template adopted by Inseq. The :class:`~inseq.attr.step_functions.StepFunctionDecoderOnlyArgs` and :class:`~inseq.attr.step_functions.StepFunctionEncoderDecoderArgs` classes are used for convenience to encapsulate all default arguments passed to step functions, namely:
 
 - :obj:`attribution_model`: the attribution model used to compute attributions.
 

docs/source/main_classes/feature_attribution.rst

@@ -28,9 +28,6 @@ Gradient Attribution Methods
     :members:
 
 
-.. warning::
-    The DiscretizedIntegratedGradientsAttribution class is currently exhibiting inconsistent behavior, so usage should be limited until further notice. See PR `# 114 <https://github.com/inseq-team/inseq/pull/114>`__ for additional info.
-
 .. autoclass:: inseq.attr.feat.DiscretizedIntegratedGradientsAttribution
     :members:
 
@@ -50,6 +47,10 @@ Gradient Attribution Methods
     :members:
 
 
+.. autoclass:: inseq.attr.feat.SequentialIntegratedGradientsAttribution
+    :members:
+
+
 Layer Attribution Methods
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/source/main_classes/step_functions.rst

@@ -18,9 +18,9 @@ Step Functions
 Step Functions Default arguments
 -----------------------------------------------------------------------------------------------------------------------
 
-The default arguments passed to all step functions are collected in the :class:`StepFunctionArgs` class.
+The default arguments passed to all step functions are collected in the :class:`StepFunctionBaseArgs` class.
 
-.. autoclass:: StepFunctionArgs
+.. autoclass:: StepFunctionBaseArgs
 
 Pre-registered Step Functions
 -----------------------------------------------------------------------------------------------------------------------
@@ -37,13 +37,19 @@ The following functions can be used out-of-the-box as attribution targets or ste
 
 .. autofunction:: perplexity_fn
 
+.. autofunction:: contrast_logits_fn
+
 .. autofunction:: contrast_prob_fn
 
+.. autofunction:: contrast_logits_diff_fn
+
+.. autofunction:: contrast_prob_diff_fn
+
 .. autofunction:: pcxmi_fn
 
 .. autofunction:: kl_divergence_fn
 
-.. autofunction:: contrast_prob_diff_fn
+.. autofunction:: in_context_pvi_fn
 
 .. autofunction:: mc_dropout_prob_avg_fn