Add automatic differentiation algorithm #10977

PoojanSmart · 2023-10-26T02:21:02Z

Describe your change:

Adding implementation of automatic differentiation algorithm which is useful for calculating gradients automatically. This algorithm is widely used in machine learning and deep learning models for optimization.

Add an algorithm?
Fix a bug or typo in an existing algorithm?
Add or change doctests? -- Note: Please avoid changing both code and tests in a single pull request.
Documentation change?

Checklist:

algorithms-keeper

Click here to look at the relevant links ⬇️

🔗 Relevant Links

Repository:

Contributing guidelines

Project Euler solution guidelines

Python:

Formatted string literals (f-strings)

Type hints

doctest

unittest

pytest

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper commands and options

algorithms-keeper actions can be triggered by commenting on this PR:

@algorithms-keeper review to trigger the checks for only added pull request files

@algorithms-keeper review-all to trigger the checks for all the pull request files, including the modified files. As we cannot post review comments on lines not part of the diff, this command will post all the messages in one comment.

NOTE: Commands are in beta and so this feature is restricted only to a member or owner of the organization.

algorithms-keeper · 2023-10-26T02:21:09Z

machine_learning/auto-diff.py

+    will be calculated.
+    """
+
+    def __init__(self, value):


Please provide return type hint for the function: __init__. If the function does not return a value, please provide the type hint as: def function() -> None:

Please provide type hint for the parameter: value

algorithms-keeper · 2023-10-26T02:21:09Z

machine_learning/auto-diff.py

+                )
+        return result
+
+    def add_param_to(self, param_to: Operation):


Please provide return type hint for the function: add_param_to. If the function does not return a value, please provide the type hint as: def function() -> None:

algorithms-keeper · 2023-10-26T02:21:09Z

machine_learning/auto-diff.py

+    def add_param_to(self, param_to: Operation):
+        self.param_to.append(param_to)
+
+    def add_result_of(self, result_of: Operation):


Please provide return type hint for the function: add_result_of. If the function does not return a value, please provide the type hint as: def function() -> None:

algorithms-keeper · 2023-10-26T02:21:09Z

machine_learning/auto-diff.py

+    objects and pointer to resulting Variable from the operation.
+    """
+
+    def __init__(


Please provide return type hint for the function: __init__. If the function does not return a value, please provide the type hint as: def function() -> None:

algorithms-keeper · 2023-10-26T02:21:10Z

machine_learning/auto-diff.py

+        self.output: Variable | None = None
+        self.other_params = {} if other_params is None else other_params
+
+    def add_params(self, params: list[Variable]):


Please provide return type hint for the function: add_params. If the function does not return a value, please provide the type hint as: def function() -> None:

algorithms-keeper · 2023-10-26T02:21:10Z

machine_learning/auto-diff.py

+    based on the computation graph.
+    """
+
+    def __new__(cls):


Please provide return type hint for the function: __new__. If the function does not return a value, please provide the type hint as: def function() -> None:

algorithms-keeper · 2023-10-26T02:21:10Z

machine_learning/auto-diff.py

+            cls.instance = super().__new__(cls)
+        return cls.instance
+
+    def __init__(self):


Please provide return type hint for the function: __init__. If the function does not return a value, please provide the type hint as: def function() -> None:

algorithms-keeper · 2023-10-26T02:21:10Z

machine_learning/auto-diff.py

+    def __init__(self):
+        self.enabled = False
+
+    def __enter__(self):


Please provide return type hint for the function: __enter__. If the function does not return a value, please provide the type hint as: def function() -> None:

algorithms-keeper · 2023-10-26T02:21:10Z

machine_learning/auto-diff.py

+        self.enabled = True
+        return self
+
+    def __exit__(self, exc_type, exc_value, tb):


Please provide return type hint for the function: __exit__. If the function does not return a value, please provide the type hint as: def function() -> None:

Please provide type hint for the parameter: exc_type

Please provide type hint for the parameter: exc_value

Please provide type hint for the parameter: tb

algorithms-keeper · 2023-10-26T02:21:10Z

machine_learning/auto-diff.py

+    def __exit__(self, exc_type, exc_value, tb):
+        self.enabled = False
+
+    def add_operation(


Please provide return type hint for the function: add_operation. If the function does not return a value, please provide the type hint as: def function() -> None:

imSanko

Have you checked with Pre --commits

PoojanSmart · 2023-10-26T04:12:05Z

I am getting conflicting suggestions from mypy and ruff:

From mypy:

machine_learning\automatic_differentiation.py:45: error: Module "typing" has no attribute "Self"  [attr-defined]
machine_learning\automatic_differentiation.py:45: note: Use `from typing_extensions import Self` instead

And If I change typing to typing_extensions then ruff shows:

machine_learning\automatic_differentiation.py:45:1: UP035 [*] Import from `typing` instead: `Self`
   |
43 | from enum import Enum
44 | from types import TracebackType
45 | from typing_extensions import Self
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UP035
46 |
47 | import numpy as np
   |
   = help: Import from `typing`

PoojanSmart · 2023-10-26T04:15:00Z

Apart from that I am not getting these ones in my local:

machine_learning/automatic_differentiation.py:195: error: Access to generic instance variables via class is ambiguous [misc] machine_learning/automatic_differentiation.py:195: error: Incompatible return value type (got "GradientTracker", expected "Self") [return-value]

@imSanko / @cclauss can you please help?

cclauss · 2023-10-26T05:24:49Z

machine_learning/automatic_differentiation.py

+
+from enum import Enum
+from types import TracebackType
+from typing import Self


Ruff was upgraded for Py3.12 but mypy has not yet been upgraded.

Suggested change

from typing import Self

from typing_extensions import Self # noqa: UP035

algorithms-keeper

Click here to look at the relevant links ⬇️

🔗 Relevant Links

Repository:

Contributing guidelines

Project Euler solution guidelines

Python:

Formatted string literals (f-strings)

Type hints

doctest

unittest

pytest

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper commands and options

algorithms-keeper actions can be triggered by commenting on this PR:

@algorithms-keeper review to trigger the checks for only added pull request files

@algorithms-keeper review-all to trigger the checks for all the pull request files, including the modified files. As we cannot post review comments on lines not part of the diff, this command will post all the messages in one comment.

NOTE: Commands are in beta and so this feature is restricted only to a member or owner of the organization.

machine_learning/automatic_differentiation.py

algorithms-keeper · 2023-10-26T05:51:52Z

machine_learning/automatic_differentiation.py

+    def add_output(self, output: Variable) -> None:
+        self.output = output
+
+    def __eq__(self, value) -> bool:


Please provide type hint for the parameter: value

algorithms-keeper

Click here to look at the relevant links ⬇️

🔗 Relevant Links

Repository:

Contributing guidelines

Project Euler solution guidelines

Python:

Formatted string literals (f-strings)

Type hints

doctest

unittest

pytest

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper commands and options

algorithms-keeper actions can be triggered by commenting on this PR:

@algorithms-keeper review to trigger the checks for only added pull request files

@algorithms-keeper review-all to trigger the checks for all the pull request files, including the modified files. As we cannot post review comments on lines not part of the diff, this command will post all the messages in one comment.

NOTE: Commands are in beta and so this feature is restricted only to a member or owner of the organization.

machine_learning/automatic_differentiation.py

algorithms-keeper · 2023-10-26T05:58:21Z

machine_learning/automatic_differentiation.py

+    def add_output(self, output: Variable) -> None:
+        self.output = output
+
+    def __eq__(self, value) -> bool:


Please provide type hint for the parameter: value

algorithms-keeper

Click here to look at the relevant links ⬇️

🔗 Relevant Links

Repository:

Contributing guidelines

Project Euler solution guidelines

Python:

Formatted string literals (f-strings)

Type hints

doctest

unittest

pytest

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper commands and options

algorithms-keeper actions can be triggered by commenting on this PR:

@algorithms-keeper review to trigger the checks for only added pull request files

@algorithms-keeper review-all to trigger the checks for all the pull request files, including the modified files. As we cannot post review comments on lines not part of the diff, this command will post all the messages in one comment.

NOTE: Commands are in beta and so this feature is restricted only to a member or owner of the organization.

machine_learning/automatic_differentiation.py

algorithms-keeper · 2023-10-26T06:05:39Z

machine_learning/automatic_differentiation.py

+    def add_output(self, output: Variable) -> None:
+        self.output = output
+
+    def __eq__(self, value) -> bool:


Please provide type hint for the parameter: value

for more information, see https://pre-commit.ci

cclauss

This looks really cool. I hope the comments help.

cclauss · 2023-10-26T12:54:24Z

machine_learning/automatic_differentiation.py

+...     c = a + b
+...     d = a * b
+...     e = c / d
+>>> print(tracker.gradient(e, a))


Let's try to avoid print() in doctests.

Suggested change

>>> print(tracker.gradient(e, a))

>>> tracker.gradient(e, a)

Repeat below.

cclauss · 2023-10-26T12:54:51Z

machine_learning/automatic_differentiation.py

+
+Reference: https://en.wikipedia.org/wiki/Automatic_differentiation
+
+Author: Poojan smart


Your choice...

Suggested change

Author: Poojan smart

Author: Poojan Smart

cclauss · 2023-10-26T12:55:52Z

machine_learning/automatic_differentiation.py

+>>> print(tracker.gradient(e, m))
+None


Suggested change

>>> print(tracker.gradient(e, m))

None

>>> tracker.gradient(e, m) is None

True

cclauss · 2023-10-26T12:59:03Z

machine_learning/automatic_differentiation.py

+Email: smrtpoojan@gmail.com
+
+Examples:
+


In general, it is better to put doctests into the Class docstring or function/method docstring so that if someone copies the Class, function, or method then the tests come along for the ride.

cclauss · 2023-10-26T13:03:57Z

machine_learning/automatic_differentiation.py

+    Class represents n-dimensional object which is used to wrap
+    numpy array on which operations will be performed and gradient
+    will be calculated.


We get 88 characters per line max so let's use them to get to fewer lines.

Suggested change

Class represents n-dimensional object which is used to wrap

numpy array on which operations will be performed and gradient

will be calculated.

Class represents n-dimensional object which is used to wrap numpy array on which

operations will be performed and the gradient will be calculated.

cclauss · 2023-10-26T13:22:58Z

machine_learning/automatic_differentiation.py

+                derivative = np.ones_like(params[0].numpy(), dtype=np.float64)
+            else:
+                derivative = -np.ones_like(params[1].numpy(), dtype=np.float64)
+        elif operation == OpType.MUL:
+            derivative = (


Suggested change

derivative = np.ones_like(params[0].numpy(), dtype=np.float64)

else:

derivative = -np.ones_like(params[1].numpy(), dtype=np.float64)

elif operation == OpType.MUL:

derivative = (

return np.ones_like(params[0].numpy(), dtype=np.float64)

else:

return -np.ones_like(params[1].numpy(), dtype=np.float64)

if operation == OpType.MUL:

return (

cclauss · 2023-10-26T13:23:50Z

machine_learning/automatic_differentiation.py

+            derivative = (
+                params[1].numpy().T if params[0] == param else params[0].numpy().T
+            )
+        elif operation == OpType.DIV:


Suggested change

elif operation == OpType.DIV:

if operation == OpType.DIV:

cclauss · 2023-10-26T13:24:30Z

machine_learning/automatic_differentiation.py

+                derivative = 1 / params[1].numpy()
+            else:
+                derivative = -params[0].numpy() / (params[1].numpy() ** 2)
+        elif operation == OpType.MATMUL:
+            derivative = (


Suggested change

derivative = 1 / params[1].numpy()

else:

derivative = -params[0].numpy() / (params[1].numpy() ** 2)

elif operation == OpType.MATMUL:

derivative = (

return 1 / params[1].numpy()

else:

return -params[0].numpy() / (params[1].numpy() ** 2)

if operation == OpType.MATMUL:

return (

cclauss · 2023-10-26T13:25:10Z

machine_learning/automatic_differentiation.py

+        elif operation == OpType.POWER:
+            power = operation.other_params["power"]
+            derivative = power * (params[0].numpy() ** (power - 1))


Suggested change

elif operation == OpType.POWER:

power = operation.other_params["power"]

derivative = power * (params[0].numpy() ** (power - 1))

if operation == OpType.POWER:

power = operation.other_params["power"]

return power * (params[0].numpy() ** (power - 1))

cclauss · 2023-10-26T13:26:09Z

machine_learning/automatic_differentiation.py

+        elif operation == OpType.POWER:
+            power = operation.other_params["power"]
+            derivative = power * (params[0].numpy() ** (power - 1))
+        return derivative


Do we want to raise ValueError("Invalid operation") here?

…o auto-diff

for more information, see https://pre-commit.ci

…o auto-diff

algorithms-keeper

Click here to look at the relevant links ⬇️

🔗 Relevant Links

Repository:

Contributing guidelines

Project Euler solution guidelines

Python:

Formatted string literals (f-strings)

Type hints

doctest

unittest

pytest

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper commands and options

algorithms-keeper actions can be triggered by commenting on this PR:

@algorithms-keeper review to trigger the checks for only added pull request files

@algorithms-keeper review-all to trigger the checks for all the pull request files, including the modified files. As we cannot post review comments on lines not part of the diff, this command will post all the messages in one comment.

NOTE: Commands are in beta and so this feature is restricted only to a member or owner of the organization.

algorithms-keeper · 2023-10-27T02:48:55Z

machine_learning/automatic_differentiation.py

+        self.op_type = op_type
+        self.other_params = {} if other_params is None else other_params
+
+    def add_params(self, params: list[Variable]) -> None:


As there is no test file in this pull request nor any test function or class in the file machine_learning/automatic_differentiation.py, please provide doctest for the function add_params

algorithms-keeper · 2023-10-27T02:48:55Z

machine_learning/automatic_differentiation.py

+    def add_params(self, params: list[Variable]) -> None:
+        self.params = params
+
+    def add_output(self, output: Variable) -> None:


As there is no test file in this pull request nor any test function or class in the file machine_learning/automatic_differentiation.py, please provide doctest for the function add_output

algorithms-keeper · 2023-10-27T02:48:55Z

machine_learning/automatic_differentiation.py

+    def add_output(self, output: Variable) -> None:
+        self.output = output
+
+    def __eq__(self, value) -> bool:


As there is no test file in this pull request nor any test function or class in the file machine_learning/automatic_differentiation.py, please provide doctest for the function __eq__

Please provide type hint for the parameter: value

cclauss

Awesome contribution! Thanks for doing this.

* Added automatic differentiation algorithm * file name changed * Resolved pre commit errors * updated dependency * added noqa for ignoring check * adding typing_extension for adding Self type in __new__ * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * sorted requirement.text dependency * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * resolved ruff --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Added automatic differentiation algorithm

7e37b7a

algorithms-keeper bot added the require type hints https://docs.python.org/3/library/typing.html label Oct 26, 2023

algorithms-keeper bot reviewed Oct 26, 2023

View reviewed changes

algorithms-keeper bot added awaiting reviews This PR is ready to be reviewed tests are failing Do not merge until tests pass labels Oct 26, 2023

file name changed

1fbe703

imSanko reviewed Oct 26, 2023

View reviewed changes

Resolved pre commit errors

86b3397

algorithms-keeper bot removed the tests are failing Do not merge until tests pass label Oct 26, 2023

cclauss reviewed Oct 26, 2023

View reviewed changes

updated dependency

d1e7f73

algorithms-keeper bot reviewed Oct 26, 2023

View reviewed changes

algorithms-keeper bot added the tests are failing Do not merge until tests pass label Oct 26, 2023

added noqa for ignoring check

8645334

algorithms-keeper bot reviewed Oct 26, 2023

View reviewed changes

adding typing_extension for adding Self type in __new__

ad5b56d

algorithms-keeper bot reviewed Oct 26, 2023

View reviewed changes

[pre-commit.ci] auto fixes from pre-commit.com hooks

1997de6

for more information, see https://pre-commit.ci

algorithms-keeper bot removed the tests are failing Do not merge until tests pass label Oct 26, 2023

sorted requirement.text dependency

71234dd

cclauss requested changes Oct 26, 2023

View reviewed changes

cclauss self-assigned this Oct 26, 2023

PoojanSmart and others added 4 commits October 27, 2023 08:08

Merge branch 'TheAlgorithms:master' into auto-diff

a81632c

refactoring

1402242

Merge branch 'auto-diff' of https://github.com/PoojanSmart/Python int…

24401e0

…o auto-diff

[pre-commit.ci] auto fixes from pre-commit.com hooks

a3d7418

for more information, see https://pre-commit.ci

algorithms-keeper bot added the tests are failing Do not merge until tests pass label Oct 27, 2023

resolved ruff

4bea57d

Merge branch 'auto-diff' of https://github.com/PoojanSmart/Python int…

69d8fbd

…o auto-diff

algorithms-keeper bot added the require tests Tests [doctest/unittest/pytest] are required label Oct 27, 2023

algorithms-keeper bot reviewed Oct 27, 2023

View reviewed changes

algorithms-keeper bot removed the tests are failing Do not merge until tests pass label Oct 27, 2023

cclauss enabled auto-merge (squash) October 27, 2023 08:46

cclauss approved these changes Oct 27, 2023

View reviewed changes

cclauss merged commit 5987f86 into TheAlgorithms:master Oct 27, 2023

algorithms-keeper bot removed the awaiting reviews This PR is ready to be reviewed label Oct 27, 2023

isidroas mentioned this pull request Jan 25, 2025

dh isidroas/Python#1

Closed

14 tasks

	from typing import Self
	from typing_extensions import Self # noqa: UP035


		Reference: https://en.wikipedia.org/wiki/Automatic_differentiation

		Author: Poojan smart

		Email: smrtpoojan@gmail.com

		Examples:

Add automatic differentiation algorithm #10977

Add automatic differentiation algorithm #10977

Conversation

PoojanSmart commented Oct 26, 2023

Describe your change:

Checklist:

algorithms-keeper bot left a comment

Choose a reason for hiding this comment

🔗 Relevant Links

Repository:

Python:

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper actions can be triggered by commenting on this PR:

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

imSanko left a comment

Choose a reason for hiding this comment

PoojanSmart commented Oct 26, 2023

PoojanSmart commented Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot left a comment

Choose a reason for hiding this comment

🔗 Relevant Links

Repository:

Python:

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper actions can be triggered by commenting on this PR:

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot left a comment

Choose a reason for hiding this comment

🔗 Relevant Links

Repository:

Python:

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper actions can be triggered by commenting on this PR:

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

algorithms-keeper bot left a comment

Choose a reason for hiding this comment

🔗 Relevant Links

Repository:

Python:

Automated review generated by algorithms-keeper. If there's any problem regarding this review, please open an issue about it.

algorithms-keeper actions can be triggered by commenting on this PR:

algorithms-keeper bot Oct 26, 2023

Choose a reason for hiding this comment

cclauss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

algorithms-keeper bot left a comment

Choose a reason for hiding this comment

🔗 Relevant Links