fix(ingest): ensure that LookML files are always parsed in the same order

[jameslamb]

Metadata ingestion for LookML requires a few operations that list files using glob.glob(). This PR proposes sorting those lists of files by name, to ensure that given the same set of .lkml files, datahub ingest will process them in the same order every time.

The ordering of results from glob.glob() is not guaranteed to be the same on different filesystems. See https://stackoverflow.com/a/6773661 for more details.

How this improves datahub ingest

I faced a really difficult bug today. I created a docker image with an entrypoint that ran something like datahub ingest -c /recipes/recipe.yaml.

I found that I got different behavior in these two methods of running containers using that image:

• in a container on my laptop

  •  git clone https://lookml-files.git
     docker run -v $(pwd)/lookml-files:/model-files my-ingestion-image:${TAG}

• starting a pod in k8s using git-sync to clone from lookml-files.git and mount it into a container at /model-files

Eventually, I discovered a separate bug where the order of parsing mattered (proposed fix in #2965 ). A strong guarantee that datahub ingest processes files in the same order every time would have sped up my debugging of that issue, and would have given me higher confidence that ingestion running in k8s would match the behavior I saw in my development environment.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable)

Thanks for your time and consideration!

[hsheth2]

LGTM

[shirshanka]

LGTM! Thanks @jameslamb :)