Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataLoader2] Implementing single iterator constraint #700

Closed
wants to merge 5 commits into from

Commits on Jul 29, 2022

  1. [DataLoader2] Implementing single iterator constraint

    [ghstack-poisoned]
    NivekT committed Jul 29, 2022
    Configuration menu
    Copy the full SHA
    1444a1e View commit details
    Browse the repository at this point in the history
  2. Update on "[DataLoader2] Implementing single iterator constraint"

    Differential Revision: [D38262867](https://our.internmc.facebook.com/intern/diff/D38262867)
    
    [ghstack-poisoned]
    NivekT committed Jul 29, 2022
    Configuration menu
    Copy the full SHA
    bea49ad View commit details
    Browse the repository at this point in the history
  3. Update on "[DataLoader2] Implementing single iterator constraint"

    Differential Revision: [D38262867](https://our.internmc.facebook.com/intern/diff/D38262867)
    
    [ghstack-poisoned]
    NivekT committed Jul 29, 2022
    Configuration menu
    Copy the full SHA
    5b946c4 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2022

  1. Update on "[DataLoader2] Implementing single iterator constraint"

    Changes:
    * Imposing the single iterator constraint on `DataLoader2` by having its `__iter__` method returns a wrapper that can track if an iterator is valid or not
    * Previously, `__iter__` returns `self`, but now it will return an instance of `_DataLoader2Iterator`
    * Aside from the behavior related to reset (see below), we expect other behaviors to stay the same.
    
    Prior behavior:
    ```python
    dl = DataLoader2(IterableWrapper(range(10)))
    it1 = iter(dl)
    print(next(it1))  # 0
    it2 = iter(dl)  # No reset here
    print(next(it2))  # 1
    print(next(it1))  # 2
    ```
    
    New behavior with this PR:
    ```python
    dl = DataLoader2(IterableWrapper(range(10)))
    it1 = iter(dl)
    print(next(it1))  # 0
    it2 = iter(dl)  # DataLoader2 resets with the creation of a new iterator
    print(next(it2))  # 0
    print(next(it1))  # Raises exception, since it1 is no longer valid
    ```
    
    Differential Revision: [D38262867](https://our.internmc.facebook.com/intern/diff/D38262867)
    
    [ghstack-poisoned]
    NivekT committed Aug 1, 2022
    Configuration menu
    Copy the full SHA
    6028b60 View commit details
    Browse the repository at this point in the history
  2. Update on "[DataLoader2] Implementing single iterator constraint"

    Changes:
    * Imposing the single iterator constraint on `DataLoader2` by having its `__iter__` method returns a wrapper that can track if an iterator is valid or not
    * Previously, `__iter__` returns `self`, but now it will return an instance of `_DataLoader2Iterator`
    * Aside from the behavior related to reset (see below), we expect other behaviors to stay the same.
    
    Prior behavior:
    ```python
    dl = DataLoader2(IterableWrapper(range(10)))
    it1 = iter(dl)
    print(next(it1))  # 0
    it2 = iter(dl)  # No reset here
    print(next(it2))  # 1
    print(next(it1))  # 2
    ```
    
    New behavior with this PR:
    ```python
    dl = DataLoader2(IterableWrapper(range(10)))
    it1 = iter(dl)
    print(next(it1))  # 0
    it2 = iter(dl)  # DataLoader2 resets with the creation of a new iterator
    print(next(it2))  # 0
    print(next(it1))  # Raises exception, since it1 is no longer valid
    ```
    
    Differential Revision: [D38262867](https://our.internmc.facebook.com/intern/diff/D38262867)
    
    [ghstack-poisoned]
    NivekT committed Aug 1, 2022
    Configuration menu
    Copy the full SHA
    0bc12e4 View commit details
    Browse the repository at this point in the history