[BUG] Fix memory leak with Regex C++ operator

[Nayef211]

Description

• We noticed that the memory footprint of GPT2BPETokenizer increases as tokenization occurs only when special tokens are added. After further investigation, the reason for this is due to a memory leak from the additional Regex object that is created whenever special tokens have been added to the tokenizer class.
  

  text/torchtext/csrc/gpt2_bpe_tokenizer.cpp

  Line 123 in 8c01462

  const Regex specialTokenRegex("(" + pattern + ")");

• Looking into the Regex implementation we dynamically create a heap allocated object (a pointer to a RE2 object) without ever freeing it. The fix is simply to create a destructor to free the dynamically allocated memory.
  

  text/torchtext/csrc/regex.cpp

  Line 6 in 8c01462

  compiled_pattern_ = new RE2(re_str_);

Test Plan

After the fix, the memory footprint doesn't change with additional forward calls to the tokenizer.

tokenizer = GPT2BPETokenizer(
    encoder_json_path="https://download.pytorch.org/models/text/gpt2_bpe_encoder.json",
    vocab_bpe_path="https://download.pytorch.org/models/text/gpt2_bpe_vocab.bpe",
)
additional_special_tokens = ["TOKEN_0"]	
tokenizer.add_special_tokens(
    special_tokens_dict={"additional_special_tokens": additional_special_tokens}
)]	
tokenizer.add_special_tokens(
    special_tokens_dict={"additional_special_tokens": additional_special_tokens}
)

i = 0
while True:
    tokenizer.forward(["hello", "world", "random", "stuff"])
    if i % 10000 == 0:
        process = psutil.Process(os.getpid())
        memory_in_megabytes = process.memory_info().rss / (1024 * 1024)
        print(f"with special token: {i}, {memory_in_megabytes} MB")  # in MB
    i += 1

Resolves #2020

[rshraga]

looks great! Are the test failures expected / unrelated?

[joecummings]

looks great! Are the test failures expected / unrelated?

Yep, we need to increase our memory usage for the Integration Tests, see #2018

[joecummings]

Looks like the commit history may be a little messy -> can you rebase @Nayef211 ?

[Nayef211]

Looks like the commit history may be a little messy -> can you rebase @Nayef211 ?

Yeah I have a couple of commits that were made to the main branch of my forked repo rather than the original repo. Since we squash and merge changes, it shouldn't affect the commit history of the main branch.

[joecummings]

lgtm