Create logistic_regression.md #83
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Share WALR algorithm as a readme file
Fixed a few minor bugs.
logistic_regression.md
Outdated
| $X = (X^{(1)}, ..., X^{(N)}) \in [0, 1]^{N*k}$ and $y = (y^{(1)}, ..., y^{(N)})^T \in \\{0, 1\\}^N$ \ | ||
| denote the *(N \* k)*-dimensional feature *matrix* and *N*-dimensional label vector respectively, then we have | ||
|
|
||
| $\text{dot-product} = \frac{1}{N} \sum\limits_{i=1}^{N} y^{(i)} X^{(i)} = \frac{1}{N} \cdot Xy$ ,\ |
There was a problem hiding this comment.
Suggested change
| $\text{dot-product} = \frac{1}{N} \sum\limits_{i=1}^{N} y^{(i)} X^{(i)} = \frac{1}{N} \cdot Xy$ ,\ | |
| $\text{dot-product} = \frac{1}{N} \sum\limits_{i=1}^{N} y^{(i)} X^{(i)} = \frac{C\cdot y}{N}$ ,\ |
Suggested change
| $\text{dot-product} = \frac{1}{N} \sum\limits_{i=1}^{N} y^{(i)} X^{(i)} = \frac{1}{N} \cdot Xy$ ,\ | |
| $\text{dot-product} = \frac{1}{N} \sum\limits_{i=1}^{N} y^{(i)} X^{(i)} = \frac{1}{N}\cdot X \cdot y$ ,\ |
I found the positioning of the dot a little annoying. Because you are saying dot-product, but then putting the dot elsewhere.
removed a stray sigma and made Martin's suggested changes including adding authors
bmcase
approved these changes
Oct 6, 2023
logistic_regression.md
Outdated
| Any additional computations used to update $\text{noisy-} \nabla L(\theta)$ (as in a gradient descent procedure) will still be label-DP (with the same privacy parameters) due to the Post Processing Theorem of DP (see Dwork+11 text). | ||
|
|
||
| - **Question:** is $\text{noisy-dot-product}$ efficiently computable?\ | ||
| **Answer:** Yes, computing this vector requires just one pass through the set of feature vectors, and $k$ random draws from a Gaussian distribution. |
There was a problem hiding this comment.
probably should at least mention here that when implementing in MPC we will compute the division by N outside the MPC.
logistic_regression.md
Outdated
|
|
||
| First, we define the terms $\text{LHS}$ and $\text{RHS}$ as\ | ||
| $\sum\text{LHS} = \sum\limits_{i=1}^{N} p_i X^{(i)}$\ | ||
| $\sum\text{RHS} = \sum\limits_{i=1}^{N} y^{(i)} X^{(i)}$ |
There was a problem hiding this comment.
consider using
logistic_regression.md
Outdated
| $\text{noisy-} \nabla L(\theta) = (\frac{1}{N}) \cdot (\sum LHS) – (\frac{1}{N}) \cdot (\sum RHS) - \text{gaussian-noise}$. | ||
|
|
||
| To avoid computing $\text{LHS}$ at every optimization step, we approximate this term using a minibatch of size $m$. Specifically, at every gradient descent step, we sample a minibatch $M$ of size $m$, and we compute\ | ||
| $\text{mini-}\sum\text{LHS} = \sum\limits_{j=1}^{m} p_j X^{(j)}$,\ |
There was a problem hiding this comment.
here also
logistic_regression.md
Outdated
| 1. initialize model vector $\theta$ | ||
| 2. while not converged:\ | ||
| sample minibatch of size $m$, and\ | ||
| $\text{set } \theta = \theta - lr \cdot ( \text{noisy-hybrid-} \nabla L(\theta))\$ |
logistic_regression.md
Outdated
| - In the absence of any computational or privacy constraints, the model can be trained via full-batch gradient descent of the form, where here $\text{lr}$ is the learning rate: | ||
| 1. initialize model vector $\theta$ | ||
| 2. while not converged: \ | ||
| $\text{set } \theta = \theta - \text{lr} \cdot ((\frac{1}{N} \cdot \sum\limits_{i=1}^{N} \sigma(\theta^T X^{(i)}) X^{(i)} ) - \frac{1}{N} \cdot \sum\limits_{i=1}^{N} y^{(i)} X^{(i)} ))$ |
logistic_regression.md
Outdated
|
|
||
| 1. initialize model vector $\theta$ | ||
| 2. while not converged:\ | ||
| $\text{set } \theta = \theta - lr \cdot ((\frac{1}{N} \cdot \sum\limits_{i=1}^{N} p_i X^{(i)} ) - \text{noisy-dot-product})$ |
There was a problem hiding this comment.
Suggested change
| $\text{set } \theta = \theta - lr \cdot ((\frac{1}{N} \cdot \sum\limits_{i=1}^{N} p_i X^{(i)} ) - \text{noisy-dot-product})$ | |
| $\text{set } \theta := \theta - lr \cdot ((\frac{1}{N} \cdot \sum\limits_{i=1}^{N} p_i X^{(i)} ) - \text{noisy-dot-product})$ |
logistic_regression.md
Outdated
| 1. initialize model vector $\theta$ | ||
| 2. while not converged:\ | ||
| sample minibatch of size $m$, and\ | ||
| $\text{set } \theta = \theta - lr \cdot ( \text{noisy-hybrid-} \nabla L(\theta))\$ |
There was a problem hiding this comment.
Suggested change
| $\text{set } \theta = \theta - lr \cdot ( \text{noisy-hybrid-} \nabla L(\theta))\$ | |
| $\text{set } \theta := \theta - lr \cdot ( \text{noisy-hybrid-} \nabla L(\theta))\$ |
logistic_regression.md
Outdated
| $\text{noisy-} \nabla L(\theta) = (\frac{1}{N}) \cdot (\sum LHS) – (\frac{1}{N}) \cdot (\sum RHS) - \text{gaussian-noise}$. | ||
|
|
||
| To avoid computing $\text{LHS}$ at every optimization step, we approximate this term using a minibatch of size $m$. Specifically, at every gradient descent step, we sample a minibatch $M$ of size $m$, and we compute\ | ||
| $\text{mini-}\sum\text{LHS} = \sum\limits_{j=1}^{m} p_j X^{(j)}$,\ |
There was a problem hiding this comment.
Suggested change
| $\text{mini-}\sum\text{LHS} = \sum\limits_{j=1}^{m} p_j X^{(j)}$,\ | |
| $\text{mini-}\sum\text{LHS} := \sum\limits_{j=1}^{m} p_j X^{(j)}$,\ |
logistic_regression.md
Outdated
| To circumvent this computational issue, we use a simple **"hybrid"-minibatch gradient** computation that we describe here. | ||
|
|
||
| First, we define the terms $\text{LHS}$ and $\text{RHS}$ as\ | ||
| $\sum\text{LHS} = \sum\limits_{i=1}^{N} p_i X^{(i)}$\ |
There was a problem hiding this comment.
Suggested change
| $\sum\text{LHS} = \sum\limits_{i=1}^{N} p_i X^{(i)}$\ | |
| $\sum\text{LHS} := \sum\limits_{i=1}^{N} p_i X^{(i)}$\ |
logistic_regression.md
Outdated
|
|
||
| First, we define the terms $\text{LHS}$ and $\text{RHS}$ as\ | ||
| $\sum\text{LHS} = \sum\limits_{i=1}^{N} p_i X^{(i)}$\ | ||
| $\sum\text{RHS} = \sum\limits_{i=1}^{N} y^{(i)} X^{(i)}$ |
There was a problem hiding this comment.
Suggested change
| $\sum\text{RHS} = \sum\limits_{i=1}^{N} y^{(i)} X^{(i)}$ | |
| $\sum\text{RHS} := \sum\limits_{i=1}^{N} y^{(i)} X^{(i)}$ |
A few more :=
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Share WALR algorithm as a readme file