Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MERGE 2/2] merge fbsync branch to main #1950

Merged
merged 173 commits into from
Oct 19, 2022
Merged

[MERGE 2/2] merge fbsync branch to main #1950

merged 173 commits into from
Oct 19, 2022

Conversation

Nayef211
Copy link
Contributor

Guanheng Zhang and others added 30 commits March 25, 2020 08:01
include pytorch 1.5.0-rc1 for CI test
599bf07
bump up the version
442fcf0
Merge branch 'master' into 0.6.0
b56cb04
Merge remote-tracking branch 'upstream/master' into 0.6.0
3a54c7f
Set up ShipIt
db4d4fd 
fbshipit-source-id: bb7d2eb52240c7223b57c3c9624e61d116e77e39
@cpuhrsch
Re-sync with internal repository (pytorch#749)
db354a3
@cpuhrsch @facebook-github-bot
20200429 pytorch/text import
ab3c93b 
Summary: [20:45:34: cpuhrsch@devvm3140 pytorch]$ ./fb_build/import_text.sh

Reviewed By: pbelevich

Differential Revision: D21320577

fbshipit-source-id: ac2148b9f0d58e5538443c879845bfb4f6ca7202
@cpuhrsch @facebook-github-bot
20200430 torchtext import script to include additional meta files
efdf20d 
Summary: ./fb_build/import_text.sh

Reviewed By: zhangguanheng66

Differential Revision: D21343124

fbshipit-source-id: c08ecad2cc6f439fa40130aeaf91383be9403fe8
@cpuhrsch @facebook-github-bot
torchtext flake8, github, travis metafiles
bfbdaec 
Summary: See title

Reviewed By: pbelevich

Differential Revision: D21344211

fbshipit-source-id: a8bcf7f3ab9bb2c2853e27f612e82caa341d3651
@mthrok @facebook-github-bot
Import torchtext 20200520 and update build
5aeca8d 
Summary: Import torchtext up to pytorch#786

Reviewed By: cpuhrsch

Differential Revision: D21483116

fbshipit-source-id: bc8ab38db9dc9ce4a8734ca8ea991c20e4ef0882
@mthrok @facebook-github-bot
Import torchtext 20200528
b79875b 
Summary:
Import up to pytorch#798
Addresses T67599333

Reviewed By: zhangguanheng66

Differential Revision: D21764935

fbshipit-source-id: f44d1db637799f2e95f420a8099fbf19545c7cbd
@cpuhrsch @facebook-github-bot
20200604 torchtext github import
be0f749 
Summary: Import from github master

Reviewed By: zhangguanheng66

Differential Revision: D21886238

fbshipit-source-id: a8f098e299466dd1701fe7ceb6a97c2a2fc54b9d
@Nayef211 @facebook-github-bot
Import torchtext 20200605
4fd0fe2 
Summary: Import from github master

Reviewed By: zhangguanheng66

Differential Revision: D21907519

fbshipit-source-id: f22370d97796da5f2cb9f76f506c80f18fefea7f
@Nayef211 @facebook-github-bot
Back out "Import torchtext 20200605"
2fe0031 
Summary: Original commit changeset: f22370d97796

Reviewed By: zhangguanheng66

Differential Revision: D21964222

fbshipit-source-id: c316836596fc3e232e63abc59e172f237b551cc5
@Nayef211 @facebook-github-bot
Import torchtext 2020/06/22
3bdf414 
Summary: Import from github torchtext/master

Reviewed By: zhangguanheng66, cpuhrsch

Differential Revision: D22168183

fbshipit-source-id: 7d96ade64f18942d9bd19437011be2f65f0b2a5e
@zhangguanheng66 @facebook-github-bot
Fix torch.testing._internal module not found
3088591 
Reviewed By: Nayef211

Differential Revision: D22315715

fbshipit-source-id: 6b8b8544b0aa458cf5e7e9ca380d0dc85c98189f
@Nayef211 @facebook-github-bot
Import torchtext 2020/07/07
8233815 
Summary: Import from github torchtext/master

Reviewed By: cpuhrsch

Differential Revision: D22420576

fbshipit-source-id: 4d2c19d7f1db8f698894ca406c1c44b2ad8e0506
@StanislavGlebik @facebook-github-bot
remediation of S205607
8003a02 
fbshipit-source-id: 5113fe0c527595e4227ff827253b7414abbdf7ac
@StanislavGlebik @facebook-github-bot
remediation of S205607
4e653d1 
fbshipit-source-id: 798decc90db4f13770e97cdce3c0df7d5421b2a3
@Nayef211 @facebook-github-bot
Import torchtext 2020/07/21
5521477 
Summary: Import from github torchtext/master

Reviewed By: zhangguanheng66

Differential Revision: D22641140

fbshipit-source-id: 8190692d059a937e25c5f93506581086f389c291
@zertosh @facebook-github-bot
Remove .python3 markers
9d0c4e1 
Reviewed By: ashwinp-fb

Differential Revision: D22955630

fbshipit-source-id: f00ef17a905e4c7cd9196c8924db39f9cdfe8cfa
@Nayef211 @facebook-github-bot
Import torchtext 2020/08/06
1a786ca 
Summary: Import from github torchtext/master

Reviewed By: zhangguanheng66

Differential Revision: D22989210

fbshipit-source-id: 083464e188b758a8746123f4dd2197cc7edc4bc4
@Nayef211 @facebook-github-bot
Import torchtext 2020/08/18
d7762a0 
Summary: Import from github torchtext/master

Reviewed By: cpuhrsch

Differential Revision: D23190596

fbshipit-source-id: 1568a25a5bd6431bcef3c6539f64a3ab1f5bccd7
@zhangguanheng66 @facebook-github-bot
Import torchtext from 8aecbb9
4a7bc10 
Reviewed By: hudeven

Differential Revision: D23451795

fbshipit-source-id: 73e6130c16716919c77862cef4ca4c8048428670
@zhangguanheng66 @facebook-github-bot
Import torchtext 9/4/2020
f668e3c 
Reviewed By: Nayef211

Differential Revision: D23539397

fbshipit-source-id: 88dce59418a3071cbc9e944cf0a4cf2117d7d9f7
@zhangguanheng66 @facebook-github-bot
Import github torchtext on 9/9/2020
d2e1af4 
Reviewed By: cpuhrsch

Differential Revision: D23616189

fbshipit-source-id: 365debc987326145eead7456ed48517fe55cac96
@facebook-github-bot
Add property support for ScriptModules (#42390)
8c374dd 
Summary:
Pull Request resolved: pytorch/pytorch#42390

**Summary**
This commit extends support for properties to include
ScriptModules.

**Test Plan**
This commit adds a unit test that has a ScriptModule with
a user-defined property.

`python test/test_jit_py3.py TestScriptPy3.test_module_properties`

Test Plan: Imported from OSS

Reviewed By: eellison, mannatsingh

Differential Revision: D22880298

Pulled By: SplitInfinity

fbshipit-source-id: 74f6cb80f716084339e2151ca25092b6341a1560
@zhangguanheng66 @facebook-github-bot
sync with OSS torchtext 9/15/20
2ac6d79 
Reviewed By: cpuhrsch

Differential Revision: D23721167

fbshipit-source-id: 13b32091c422a3ed0ae299595d69a7afa7136638
@zhangguanheng66 @facebook-github-bot
Import Github torchtext on 9/28/2020
84174db 
Reviewed By: cpuhrsch

Differential Revision: D23962265

fbshipit-source-id: 0d042878fe9119aa725e982ab7d5e96e7c885a59
@facebook-github-bot
Enable @unused syntax for ignoring properties (#45261)
24b304e 
Summary:
Pull Request resolved: pytorch/pytorch#45261

**Summary**
This commit enables `unused` syntax for ignoring
properties. Inoring properties is more intuitive with this feature enabled.
`ignore` is not supported because class type properties cannot be
executed in Python (because they exist only as TorchScript types) like
an `ignored` function and module properties that cannot be scripted
are not added to the `ScriptModule` wrapper so that they
may execute in Python.

**Test Plan**
This commit updates the existing unit tests for class type and module
properties to test properties ignored using `unused`.

Test Plan: Imported from OSS

Reviewed By: navahgar, Krovatkin, mannatsingh

Differential Revision: D23971881

Pulled By: SplitInfinity

fbshipit-source-id: 8d3cc1bbede7753d6b6f416619e4660c56311d33
CodemodService Bot and others added 19 commits August 4, 2022 06:43
@facebook-github-bot
Import TorchText from Github
58d6620 
Reviewed By: parmeet

Differential Revision: D38419656

fbshipit-source-id: 871439658ed673910c68c025be471501b9b4670a
@facebook-github-bot
Import TorchText from Github
e45d84c 
Reviewed By: abhinavarora

Differential Revision: D38534440

fbshipit-source-id: 3bf1a7d5cc2daa8d14e424d16509b2df998549b8
@pmabbo13 @facebook-github-bot
Import TorchText from Github
543fc19 
Reviewed By: Nayef211

Differential Revision: D38655164

fbshipit-source-id: 0b9364fb759520c6fb60147fd0ab1044c362d588
@joecummings @facebook-github-bot
Import torchtext pytorch#1879 72966f0
16a0eb5 
Summary: ran the `import_text.sh` command to manually update the internal fbcode to match the Github torchtext repo

Reviewed By: Nayef211

Differential Revision: D38796445

fbshipit-source-id: 904143c404141bb016a5f83fbc53906b1c6e1246
@facebook-github-bot
Import TorchText from Github
f94843a 
Reviewed By: abhinavarora

Differential Revision: D38907288

fbshipit-source-id: f82ad8121bce924ad6068767845e5ea29dd24bef
@facebook-github-bot
Remove dependency on the torch::jit::script::Module for mobile builds
0303323 
Summary: In order to resolve linkage errors. Specifically when vocab getting build for "mobile" version it can't resolve symbols for torch::jit::script::Module

Reviewed By: Nayef211

Differential Revision: D38771271

fbshipit-source-id: 693b656f2a17af9fa5a7a1904742557f902edb55
@joecummings @facebook-github-bot
Replace pytext_lib's MaskTransform with new one from torchtext
fd6e5cd 
Summary: Replace instances of `pytext_lib`'s `MaskTransform` with new one from `torchtext` that was merged in pytorch#1882

Reviewed By: Nayef211

Differential Revision: D39058074

fbshipit-source-id: f61499d88eec7eccda659279786528bac7edf9d0
@facebook-github-bot
Import TorchText from Github
3d1c635 
Reviewed By: abhinavarora

Differential Revision: D39095295

fbshipit-source-id: 2e447db46b71fc152f2f53b281585650682cb696
@hudeven @facebook-github-bot
move PATH_MANAGER to OSS
60737d9 
Summary:
## Problem:
pytext got "No module named 'pytorch'" in issue facebookresearch/pytext#1706

It's due to `from pytorch.text.fb.utils import PATH_MANAGER` is internal only but imported in pytext. Actually, `pytorch/text/fb/utils/__init__.py` should be open sourced.

## Solution:
This diff moved it to OSS as `from torchtext.utils import PATH_MANAGER` and updated all the references

Reviewed By: Nayef211

Differential Revision: D39292896

fbshipit-source-id: c0046d62e64145b60ad9a5298b366f0f1a348369
@mikekgfb @facebook-github-bot
Turn off mask checking for torchtext which is known to have a legal m…
131533c 
…ask (pytorch#1896)

Summary:
Pull Request resolved: pytorch#1896

Turn off mask checking for torchtext which is known to have a legal mask

Reviewed By: zrphercule

Differential Revision: D39445703

fbshipit-source-id: 3f0cacfd39ea11a16c7a06f339872554333b5e97
@hudeven @facebook-github-bot
Back out "move PATH_MANAGER to OSS" (pytorch#1724)
13692db 
Summary:
X-link: facebookresearch/pytext#1724

Original commit changeset: c0046d62e641

Original Phabricator Diff: D39292896

torchtext can't depend on iopath as raised in pytorch#1905

Reviewed By: Nayef211

Differential Revision: D39639475

fbshipit-source-id: 69a48eb3820d0642b0a56712e160a0af589e4c7c
@Nayef211 @facebook-github-bot
Import TorchText from Github
b8e12fe 
Summary: Manually import latest changes from github to fbcode

Reviewed By: joecummings

Differential Revision: D39770284

fbshipit-source-id: 1e442f222d582c43a2ca9280d93eca4135d2df09
@Nayef211 @facebook-github-bot
Import TorchText from Github
a85fd57 
Reviewed By: rshraga

Differential Revision: D39811057

fbshipit-source-id: 33cce346ac3d226a2fff6c162c39164837f34d87
@facebook-github-bot
Import TorchText from Github
f44310f 
Reviewed By: abhinavarora

Differential Revision: D40225047

fbshipit-source-id: 7abff009d65d713a6ce134fc88cd1955f62e3e3d
@facebook-github-bot
Import TorchText from Github
bedb667 
Reviewed By: abhinavarora

Differential Revision: D40294258

fbshipit-source-id: b3e14d9e78e346c294f1bc65ba3045b92251e034
@facebook-github-bot
Add Character Level BPE Tokenizer (pytorch#1936)
bac09bf 
Summary:
Pull Request resolved: pytorch#1936

This change adds a character level BPE tokenizer to the set of available transforms. It takes a pre-trained encoder dict (i.e vocab dict) and merge list as input. It is not using C++ for encoding / decoding at this time.

Reviewed By: langong347

Differential Revision: D40186470

fbshipit-source-id: 48bacc631f537e941a495e39ef9ccb17d3ef7896
@facebook-github-bot
Add padding_masks and tests for T5Model (pytorch#1935)
aa369b8 
Summary:
Pull Request resolved: pytorch#1935

Added the following parameters to the `forward` method of the T5Model:
* `encoder_padding_mask`
* `decoder_padding_mask`

These allow users to specifically mask out the padding of input sequences. This matches the implementation of Transformers in PyTorch core.

Reviewed By: Nayef211

Differential Revision: D40252794

fbshipit-source-id: 0e0a17fdc97ae0bbcaa1aef91e9914fd6225456b
@facebook-github-bot
Import TorchText from Github
549e5c7 
Reviewed By: abhinavarora

Differential Revision: D40425553

fbshipit-source-id: 268b94d65cff771028c2e2fdf21caa9855d07cef
@Nayef211
Merge branch 'main' into merge_main_to_fbsync
db987ed
@Nayef211 Nayef211 mentioned this pull request Oct 18, 2022
Open
@Nayef211 Nayef211 marked this pull request as ready for review October 18, 2022 21:19
@rshraga
Copy link
Contributor

rshraga commented Oct 19, 2022

Retried failing build and unittest to see if resolved. Is fbsync healthy overall? Where in the UI can we see that fbsync / all these commits are in a good state?

@Nayef211
Copy link
Contributor Author

Retried failing build and unittest to see if resolved. Is fbsync healthy overall? Where in the UI can we see that fbsync / all these commits are in a good state?

Looks like all tests are passing now, thanks for doing the retry. You can see the status of the commits in fbsync (i.e. if all tests are passing) here https://github.com/pytorch/text/commits/fbsync. However this is not a relevant signal since we don't generally create PRs against the fbsync branch. Rather once a diff gets commited on fbcode, a corresponding commit is created in the fbsync branch bypassing all CI tests.

@Nayef211 Nayef211 merged commit 31b8aaa into pytorch:main Oct 19, 2022
@Nayef211 Nayef211 deleted the merge_fbsync_to_main branch October 19, 2022 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@joecummings joecummings joecummings approved these changes

@rshraga rshraga Awaiting requested review from rshraga

@abhinavarora abhinavarora Awaiting requested review from abhinavarora

Assignees
No one assigned
Labels
cla signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.