CLN: unify numpy.random-related imports in pandas/tests/plotting

[onshek]

latest update

all files are moved to #37492

---

• closes CLN: unify numpy.random-related imports #37053 (keep this issue open until both CLN: unify numpy.random-related imports in pandas/tests/plotting #37103 and CI: check for numpy.random-related imports #37117 are completed)
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff

A separate PR(CI) will be opened to add related code_check(#37117).
PS: Should a whatsnew note be added?

update

The whole project is reformatted except pandas/_testing.py for there are quite a few tm.randn used in testing files.

below is the script to do the whole clean-up

import os
import re


class NumpyRandomRelatedScript:
    def __init__(self, base_dir: str = "/dir/of/pandas/pandas") -> None:
        self.base_dir = base_dir
        self.py_files = []
        self.backup_files = []
        self.p1 = re.compile("import numpy as np")
        self.p2 = re.compile("from numpy.random import[ ,a-zA-Z]*[\s]")
        self.p3 = re.compile("from numpy import random[\s]")

    def search_py_files(self, pandas_file_dir: str) -> None:
        if os.path.isfile(pandas_file_dir):
            if pandas_file_dir[-3:] == ".py":
                if pandas_file_dir != os.path.join(self.base_dir, "_testing.py"):
                    self.py_files.append(pandas_file_dir)
        elif os.path.isdir(pandas_file_dir):
            for d in os.listdir(pandas_file_dir):
                self.search_py_files(os.path.join(pandas_file_dir, d))

    def do_the_clean_up(self, file_dir: str) -> None:
        with open(file_dir, "r") as file:
            data = file.read()
            m1, m2, m3 = (
                re.search(self.p1, data),
                re.search(self.p2, data),
                re.search(self.p3, data),
            )
            if not (m2 or m3):
                # print("There's no need to change, please recheck!")
                return

        backup_dir = file_dir + ".issue37053_backup"
        self.backup_files.append(backup_dir)
        print("Backup: " + backup_dir)
        with open(backup_dir, "w+") as file:
            file.write(data)

        with open(file_dir, "w+") as file:
            if m2:
                if not m1:
                    data = re.sub(self.p2, "import numpy as np\n", data)
                    m1 = True
                else:
                    data = re.sub(self.p2, "", data)
                methods = (
                    m2.group(0)
                    .replace("from numpy.random import", "")
                    .replace(" ", "")
                    .replace("\n", "")
                    .split(",")
                )
                if isinstance(methods, str):
                    methods = [methods]
                for method in methods:
                    data = re.sub(
                        r"[\s]{meth}[^a-z]".format(meth=method),
                        " np.random.{meth}(".format(meth=method),
                        data,
                    )
                    data = re.sub(
                        r"[\s]-{meth}[^a-z]".format(meth=method),
                        " -np.random.{meth}(".format(meth=method),
                        data,
                    )
                    data = re.sub(
                        r"[\(]{meth}[^a-z]".format(meth=method),
                        "(np.random.{meth}(".format(meth=method),
                        data,
                    )
            if m3:
                if not m1:
                    data = re.sub(self.p3, "import numpy as np\n", data)
                else:
                    data = re.sub(self.p3, "", data)
                data = re.sub(r"[\s]random.{1}", " np.random.", data)
                data = re.sub(r"[\(]random.{1}", "(np.random.", data)
            file.write(data)
            print("Clean: " + file_dir)

    def remove_backup(self) -> None:
        for file in self.backup_files:
            print("Remove: " + file)
            os.remove(file)


if __name__ == "__main__":
    script = NumpyRandomRelatedScript()
    script.search_py_files(script.base_dir)
    for f in script.py_files:
        script.do_the_clean_up(f)
    script.remove_backup()

[arw2019]

Thanks @onshek for the PR!

No need for a whatsnew since this isn't a user facing change.

Also maybe note in the description that this partially closes #37053 (we'll leave that open until we get the code check in)

[onshek]

Thanks @onshek for the PR!

No need for a whatsnew since this isn't a user facing change.

Also maybe note in the description that this partially closes #37053 (we'll leave that open until we get the code check in)

updated

[charlesdong1991]

emm, shall we hold this PR up and implement the logic in code_check.sh first and do this whole clean-up in one go by running code_check.sh? I assume this unification might pop up in other files too

[mroeschke]

As noted in #37053 (comment), I would personally like to standardize towards np.random.<method>(...) instead of from numpy.random import <method>; <method>(...).

IMO it's easier to reason about where <method> comes from.

[jreback]

yeah we don't want to do this. apologize if i said this was ok omewhere else. we always want fully qualified imports. We spent a lot of time removing this over the past few years and don't want to add it back. fuly qualified is much more obvious / explicit. what we do want are explicit checks that we are not using the shortcut imports.

[onshek]

Fully understand what you said above. Well this PR is still necessary I think, since from numpy import random and from numpy.random import randn also do exist. A code_check will be added as well.
Thanks for your correction ! @charlesdong1991 @mroeschke @jreback @arw2019
One more question, is the new code check used for checking (just catching the location where random.<method> / <method> are used), or we also want it to generate the correct imports?

[jbrockmendel]

@onshek can you merge master

[onshek]

@onshek can you merge master

sure, will do it later

[onshek]

all files are moved to #37492