【Hackathon 5th No.19】Add ContinuousBernoulli and MultivariateNormal API

[NKNaN]

PR types

New features

PR changes

APIs

Description

Add ContinuousBernoulli and MultivariateNormal API

• rfc of ContinuousBernoulli: https://github.com/PaddlePaddle/community/blob/master/rfcs/APIs/20230927_api_design_for_ContinuousBernoulli.md
• rfc of MultivariateNormal: https://github.com/PaddlePaddle/community/blob/master/rfcs/APIs/20230927_api_design_for_MultivariateNormal.md

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[NKNaN]

Math Derivation for entropy of the Continuous Bernoulli distribution and kl_divergence of 2 Continuous Bernoulli distributions:

• entropy:

$$\begin{aligned} H &= -\int_x C(\lambda) \lambda^x (1-\lambda)^{1-x} \log\{C(\lambda) \lambda^x (1-\lambda)^{1-x}\} dx \\\ & = -\int_0^1 C \lambda^x (1-\lambda)^{1-x} \left[ \log C + x \log \lambda + (1 -x) \log (1 - \lambda)\right] dx \\\ & = -\left[ C \log C \int_0^1 \lambda^x (1-\lambda)^{1-x} dx + C \log \lambda \int_0^1 x \lambda^x (1-\lambda)^{1-x} + C \log(1 - \lambda) \int_0^1 (1-x) \lambda^x (1-\lambda)^{1-x} \right] \\\ & = - \left[ \log C + \mathbb{E}(X) \log \lambda + \mathbb{E}(1 - X) \log(1 - \lambda) \right] \\\ & = -\log C + \left[ \log (1 - \lambda) -\log \lambda \right] \mathbb{E}(X) - \log(1 - \lambda) \end{aligned}$$

• kl_divergence:

$$\begin{aligned} \mathcal{D}_{KL}(p_1|| p_2) &= \int_x p_1(x)\log\frac{p_1(x)}{p_2(x)} dx \\\ & = \int_0^1 C_1 \lambda_1^x (1-\lambda_1)^{1-x} \{\log[C_1 \lambda_1^x (1-\lambda_1)^{1-x}] - \log[C_2 \lambda_2^x (1-\lambda_2)^{1-x}]\} dx \\\ & = -H - [C_1 \log C_2 \int_0^1 \lambda_1^x (1-\lambda_1)^{1-x} dx + C_1 \log \lambda_2 \int_0^1 x \lambda_1^x (1-\lambda_1)^{1-x} dx + C_1 \log (1-\lambda_2) \int_0^1 (1-x) \lambda_1^x (1-\lambda_1)^{1-x} dx] \\\ & = -H - [\log C_2 + \log \lambda_2 \mathbb{E}_1(X) + \log (1-\lambda_2) \mathbb{E}_1(1-X) ] \\\ & = - H - \{\log C_2 + [\log \lambda_2 - \log (1-\lambda_2)] \mathbb{E}_1(X) + \log (1-\lambda_2) \} \end{aligned}$$

[NKNaN]

Math Derivation for entropy of the Multivariate Normal distribution and kl_divergence of 2 Multivariate Normal distributions:

• entropy:

$$\begin{aligned} H &= -\int_x f(x) \log f(x) dx \\\ & = -\int_{x \in \mathbb{R}^n} f(x) \{ -\frac{n}{2}\log(2\pi) -\frac{1}{2} (x-\mu)^{\intercal} \Sigma^{-1} (x-\mu) - \frac{1}{2}\log (\det\Sigma) \} dx \\\ & = -\int_{x \in \mathbb{R}^n} f(x) \{ -\frac{n}{2}\log(2\pi) -\frac{1}{2} [A^{-1}(x-\mu)]^{\intercal}[A^{-1}(x-\mu)] - \log (\det A) \} dx \\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{1}{2}\int_{x \in \mathbb{R}^n} [A^{-1}(x-\mu)]^{\intercal}[A^{-1}(x-\mu)] f(x) dx\\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{1}{2} \mathbb{E}[(X-\mu)^{\intercal} \Sigma^{-1} (X - \mu)] \\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{1}{2} \mathbb{E}[tr[(X-\mu)^{\intercal} \Sigma^{-1} (X - \mu)] ] \\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{1}{2} \mathbb{E}[tr[\Sigma^{-1} (X - \mu) (X-\mu)^{\intercal}]] \\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{1}{2} tr[\mathbb{E}[\Sigma^{-1} (X - \mu) (X-\mu)^{\intercal}]] \\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{1}{2} tr[\Sigma^{-1} \mathbb{E}[ (X - \mu) (X-\mu)^{\intercal}]] \\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{1}{2} tr[\Sigma^{-1} \Sigma] \\\ & = \frac{n}{2} \log(2\pi) + \log {\det A} + \frac{n}{2} \end{aligned}$$

• kl_divergence:

$$\begin{aligned} \mathcal{D}_{KL}(f_1|| f_2) &= \int_x f_1(x)\log\frac{f_1(x)}{f_2(x)} dx \\\ & = \int_{x \in \mathbb{R}^n} f_1(x)\left\{\left[ -\frac{n}{2} \log(2\pi) - \log(\det A_1) - \frac{1}{2}(x-\mu_1)^{\intercal} \Sigma_1^{-1} (x - \mu_1) \right] + \left[ \frac{n}{2} \log(2\pi) + \log(\det A_2) + \frac{1}{2}(x-\mu_2)^{\intercal} \Sigma_21^{-1} (x - \mu_2)\right]\right\} dx \\\ & = \log(\det A_2) - \log(\det A_1) +\frac{1}{2}\mathbb{E}_1[(X-\mu_2)^{\intercal} \Sigma_2^{-1} (X - \mu_2)] -\frac{n}{2} \\\ & = \log(\det A_2) - \log(\det A_1) +\frac{1}{2}tr [\Sigma_2^{-1}\mathbb{E}_1[ (X - \mu_2) (X-\mu_2)^{\intercal} ]] -\frac{n}{2} \\\ & = \log(\det A_2) - \log(\det A_1) -\frac{n}{2} +\frac{1}{2}tr [\Sigma_2^{-1}\mathbb{E}_1[ XX^{\intercal} -X \mu_2^{\intercal} - \mu_2 X^{\intercal} + \mu_2\mu_2^{\intercal}]] \\\ & = \log(\det A_2) - \log(\det A_1) -\frac{n}{2} +\frac{1}{2}tr [\Sigma_2^{-1} [ Var_1(X) + \mathbb{E}_1(X)\mathbb{E}_1(X)^{\intercal} -\mu_1\mu_2^{\intercal} - \mu_2 \mu_1^{\intercal} + \mu_2\mu_2^{\intercal}]] \\\ & = \log(\det A_2) - \log(\det A_1) -\frac{n}{2} +\frac{1}{2}tr [\Sigma_2^{-1} [ \Sigma_1 + \mu_1\mu_1^{\intercal} -\mu_1\mu_2^{\intercal} - \mu_2 \mu_1^{\intercal} + \mu_2\mu_2^{\intercal}]] \\\ & = \log(\det A_2) - \log(\det A_1) -\frac{n}{2} +\frac{1}{2}tr [\Sigma_2^{-1} \Sigma_1 + (\mu_1 - \mu_2)^{\intercal} \Sigma_2^{-1} (\mu_1 - \mu_2)] \\\ & = \log(\det A_2) - \log(\det A_1) -\frac{n}{2} +\frac{1}{2}[tr [\Sigma_2^{-1} \Sigma_1] + (\mu_1 - \mu_2)^{\intercal} \Sigma_2^{-1} (\mu_1 - \mu_2)] \\\ \end{aligned}$$

[paddle-ci-bot]

Sorry to inform you that 064e8a9's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

[paddle-ci-bot]

Sorry to inform you that 4ce267e's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

[luotao1]

test_distribution_continuous_bernoulli_static 单测没过

[paddle-ci-bot]

Sorry to inform you that 8b913d3's CIs have passed for more than 7 days. To prevent PR conflicts, you need to re-run all CIs manually.

[cxxly]

如果probability本身是Tensor，这里会改变probability的数据类型。别入用户传入p是fp32, 默认数据类型是fp64

[NKNaN]

已修改

[cxxly]

LGTM

[jeff41404]

code is fine, please add link of rfc in description above

[jeff41404]

the design of ContinuousBernoulli in rfc needs to be consistent with the code

[jeff41404]

the rfc of MultivariateNormal have same issue

[NKNaN]

code is fine, please add link of rfc in description above

添加了rfc链接，rfc设计文档需要做一些修改，已提相应pr

[luotao1]

对应中文文档可以提上来

[NKNaN]

对应中文文档可以提上来

已提中文pr

又修改了一下对应的英文文档

[sunzhongkai588]

Suggested change

	Args:
	probability(int|float|Tensor): The probability of Continuous Bernoulli distribution between [0, 1],
	which characterize the shape of the pdf. If the input data type is int or float, the data type of
	`probability` will be convert to a 1-D Tensor the paddle global default dtype.
	eps(float): Specify the bandwith of the unstable calculation region near 0.5. The unstable calculation region
	would be [0.5 - eps, 0.5 + eps], where the calculation is approximated by talyor expansion. The
	default value is 0.02.
	Args:
	probability(int|float|Tensor): The probability of Continuous Bernoulli distribution between [0, 1],
	which characterize the shape of the pdf. If the input data type is int or float, the data type of
	`probability` will be convert to a 1-D Tensor the paddle global default dtype.
	eps(float): Specify the bandwith of the unstable calculation region near 0.5. The unstable calculation region
	would be [0.5 - eps, 0.5 + eps], where the calculation is approximated by talyor expansion. The
	default value is 0.02.

对于 Args 下的每个参数，同一个参数的描述换行需要加下缩进

[NKNaN]

已修改

[sunzhongkai588]

能不能直接贴上论文的连接？引用方式参考 如何让文档相互引用；

[NKNaN]

已修改

[sunzhongkai588]

• 请参考 代码示例写法 ，需要在前面加上 >>> 或 ...
• 另外，输出不要用注释的形式，请仔细参考 代码示例书写规范

[NKNaN]

已修改

[sunzhongkai588]

Suggested change

	* :math:\Omega: is the support of the distribution.
	* :math:`\Omega` is the support of the distribution.

[NKNaN]

已修改

[sunzhongkai588]

这篇文档和 continuous_bernoulli.py 有同样的问题，不赘述了

[NKNaN]

对应处已修改

[cxxly]

抱歉，再补充个Comment，ContinuousBernoulli 签名和PyTorch保持一致

[NKNaN]

抱歉，再补充个Comment，ContinuousBernoulli 签名和PyTorch保持一致

已修改

[sunzhongkai588]

LGTM～

doc-preview CI 中发现新的 system message 错误， @ooooo-create 之后全量的再检查一遍相关错误并修复叭

[cxxly]

不用加None，probs是必选参数

[NKNaN]

已修改

[cxxly]

LGTM

[jeff41404]

There are spelling and capitalization issues in the link of rfc, eg should be 20230927_api_design_for_ContinuousBernoulli.md instead of 20230927_api_design_for_continuous_bernoulli.md, cause error of 404

[NKNaN]

There are spelling and capitalization issues in the link of rfc, eg should be 20230927_api_design_for_ContinuousBernoulli.md instead of 20230927_api_design_for_continuous_bernoulli.md, cause error of 404

Links have been updated

[jeff41404]

LGTM