let tracincp aggregate influence #1088

99warriors · 2022-12-08T20:16:47Z

Summary:
This diff adds an "aggregate" option to TracInCP.influence. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When aggregate is True, influence in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When aggregate is True, influence in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape.

This option is only added for TracInCP, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints).

Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow inputs for influence to be a dataloader, so that it does not need to fit in memory.

One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric.

We add the following tests:

in newly added test_tracin_aggregate_influence, test_tracin_aggregate_influence tests that calling influence with aggregate=Truedoes give the same result as calling it with aggregate=False, and then summing.
in newly added test_tracin_aggregate_influence, test_tracin_aggregate_influence_api tests that the result of calling influence when aggregate is true for a DataLoader of batches is the same as when the batches are collated into a single batch.
in test_tracin_k_most_influential, we modify the test to allow aggregate to be true, which tests that the proponents computed with the memory saving approach by influence are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar

Differential Revision: D41830245

facebook-github-bot · 2022-12-08T20:17:24Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Differential Revision: D41830245 fbshipit-source-id: c39dda0a1ecfb427f81b68cb1b40b56d713f62a7

facebook-github-bot · 2022-12-09T16:19:19Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245 fbshipit-source-id: 4a95233355f95da2547a2abd8cb7373065cc9707

facebook-github-bot · 2023-02-09T21:34:52Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245 fbshipit-source-id: ad971bd623261b8e0de7ff737f158f15b388102b

facebook-github-bot · 2023-04-12T00:25:41Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245 fbshipit-source-id: b2d61bee43b1562ae70d26812ba44fd08b0bf4a0

facebook-github-bot · 2023-04-12T00:30:47Z

This pull request was exported from Phabricator. Differential Revision: D41830245

facebook-github-bot · 2023-04-14T00:20:02Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245 fbshipit-source-id: 133b00d19b2a3890ae025f22f7fb814bf0a39e4f

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245 fbshipit-source-id: 521197e754dd6af7524288b9e9ab88b6201c2fb7

facebook-github-bot · 2023-04-14T00:24:57Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-09-19T13:53:16Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-09-19T14:00:14Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Differential Revision: https://internalfb.com/D41830245 fbshipit-source-id: 5317b9d4e97a4dcf8b864c35913da5c9a77b9378

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Differential Revision: https://internalfb.com/D41830245 fbshipit-source-id: b130027050ce984218acc30256cd129ef684ed45

Summary: Pull Request resolved: meta-pytorch#1088 This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Differential Revision: https://internalfb.com/D41830245 fbshipit-source-id: 796d1486c5c078236d049f4a73d780dea141f222

facebook-github-bot · 2023-11-14T15:52:15Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-11-14T15:52:47Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-11-14T15:53:20Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-11-14T16:09:55Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-11-23T01:32:18Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-11-23T01:33:00Z

This pull request was exported from Phabricator. Differential Revision: D41830245

facebook-github-bot · 2023-11-23T02:27:44Z

This pull request was exported from Phabricator. Differential Revision: D41830245

Summary: This diff adds an "aggregate" option to `TracInCP.influence`. The "aggregate" influence score of a training example on a test dataset is the sum of the influence of the training example on all examples in the test dataset. When `aggregate` is True, `influence` in influence score mode returns a 2D tensor of shape (1, training dataset size) containing aggregate influence scores of all training examples. When `aggregate` is True, `influence` in k most influential mode returns a 2D tensor of shape (1, k) of proponents (or opponents), and a 2D tensor containing the corresponding aggregate influence scores, of the same shape. This option is only added for `TracInCP`, because for it, aggregate influence can be computed more quickly than naively computing the influence score of all training examples on all test examples, and then summing across test examples. In particular, we can first sum the jacobians across all test examples, and then take the dot-product of the sum with the jacobians of training examples. (all this is done across checkpoints). Since computing aggregate influence scores is efficient, even if the test dataset is large, we now allow `inputs` for `influence` to be a dataloader, so that it does not need to fit in memory. One use case of aggregate influence is to compute the influence of a training example on some validation metric, i.e. fairness metric. We add the following tests: - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence` tests that calling `influence` with `aggregate=True`does give the same result as calling it with `aggregate=False`, and then summing. - in newly added `test_tracin_aggregate_influence`, `test_tracin_aggregate_influence_api` tests that the result of calling `influence` when `aggregate` is true for a DataLoader of batches is the same as when the batches are collated into a single batch. - in `test_tracin_k_most_influential`, we modify the test to allow `aggregate` to be true, which tests that the proponents computed with the memory saving approach by `influence` are the same proponents computed via calculating all aggregate influence scores, and then sorting (not memory efficient).ar Reviewed By: cyrjano Differential Revision: D41830245

facebook-github-bot · 2023-11-23T02:28:50Z

This pull request was exported from Phabricator. Differential Revision: D41830245

facebook-github-bot · 2023-11-23T19:49:04Z

This pull request has been merged in 006c04c.

facebook-github-bot added cla signed fb-exported labels Dec 8, 2022

99warriors force-pushed the export-D41830245 branch from c259d74 to ba2e0eb Compare December 9, 2022 16:19

99warriors force-pushed the export-D41830245 branch from ba2e0eb to c46b9c0 Compare February 9, 2023 21:34

99warriors force-pushed the export-D41830245 branch from c46b9c0 to 1a3c57a Compare April 12, 2023 00:25

99warriors force-pushed the export-D41830245 branch from 1a3c57a to 7190871 Compare April 12, 2023 00:30

99warriors force-pushed the export-D41830245 branch from 7190871 to f47a4a5 Compare April 14, 2023 00:20

99warriors force-pushed the export-D41830245 branch from f47a4a5 to 76c77c3 Compare April 14, 2023 00:24

99warriors force-pushed the export-D41830245 branch from 76c77c3 to 5a7f19a Compare September 19, 2023 13:53

99warriors force-pushed the export-D41830245 branch from 5a7f19a to e5f3244 Compare September 19, 2023 14:00

99warriors force-pushed the export-D41830245 branch from 1855f26 to abe5a28 Compare November 14, 2023 15:52

99warriors force-pushed the export-D41830245 branch from abe5a28 to afa439f Compare November 14, 2023 15:53

99warriors force-pushed the export-D41830245 branch from afa439f to 1ab6ae4 Compare November 14, 2023 16:09

99warriors force-pushed the export-D41830245 branch from 31e7983 to 4391e5f Compare November 23, 2023 01:32

99warriors force-pushed the export-D41830245 branch from 4391e5f to 78632ae Compare November 23, 2023 02:28

facebook-github-bot closed this in 006c04c Nov 23, 2023

facebook-github-bot added the Merged label Nov 23, 2023

let tracincp aggregate influence #1088

let tracincp aggregate influence #1088

Uh oh!

Conversation

99warriors commented Dec 8, 2022

Uh oh!

facebook-github-bot commented Dec 8, 2022

Uh oh!

facebook-github-bot commented Dec 9, 2022

Uh oh!

facebook-github-bot commented Feb 9, 2023

Uh oh!

facebook-github-bot commented Apr 12, 2023

Uh oh!

facebook-github-bot commented Apr 12, 2023

Uh oh!

facebook-github-bot commented Apr 14, 2023

Uh oh!

facebook-github-bot commented Apr 14, 2023

Uh oh!

facebook-github-bot commented Sep 19, 2023

Uh oh!

facebook-github-bot commented Sep 19, 2023

Uh oh!

facebook-github-bot commented Nov 14, 2023

Uh oh!

facebook-github-bot commented Nov 14, 2023

Uh oh!

facebook-github-bot commented Nov 14, 2023

Uh oh!

facebook-github-bot commented Nov 14, 2023

Uh oh!

facebook-github-bot commented Nov 23, 2023

Uh oh!

facebook-github-bot commented Nov 23, 2023

Uh oh!

facebook-github-bot commented Nov 23, 2023

Uh oh!

facebook-github-bot commented Nov 23, 2023

Uh oh!

facebook-github-bot commented Nov 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants