Tern does not detect 'pip install' command when preceded by 'python -m' in dockerfile RUN line

[chming1016]

Describe the bug
While building docker image with pip install -r requirements.txt, the relevant pip packages are not showing in the final report.

To Reproduce (PoC)

• Build a docker image with following Dockerfile.

FROM python:3.7-alpine
RUN apk update && apk upgrade
RUN python3 -m pip install --upgrade pip
COPY requirements.txt .
RUN python3 -m pip install -r requirements.txt
RUN /bin/sh

• The content of requirements.txt which is required to build the docker image mentioned above.

requests==2.22.0

• docker build command

docker build -t test:withrequirements .

• tern command

./tern/docker_run.sh ternd:2.7.0 "report -f json -i test:withrequirements" > /tmp/report.json

• Reading json report with python, there will be NO "requests" in the /tmp/report.json

import json
f = open('/tmp/report.json',)
data = json.load(f)
for i in data['images']:
     for j in i['image']['layers']:
         for k in j['packages']:
             if k['name'] == "requests":
                 print(k['name']) # Not showing "requests"

Expected behavior
As the PoC showing above, the expected result should show "requests" with version "2.22.0" in the report.

• If changing the Dockerfile in above to the following, and go through the PoC above again. The report will contains "requests". This indicates that ONLY image which build with specific pip install {package_name} will be shown in the report. However, pip install -r requirements.txt does not.

FROM python:3.7-alpine
RUN apk update && apk upgrade
RUN python3 -m pip install --upgrade pip
RUN pip install requests
RUN /bin/sh

Environment you are running Tern on
Enter all that apply

• Output of 'tern --version'

Tern at commit b56f86a0117f12c6a05664f731bcaa118b7fcf37
   python version = 3.6.8 (default, Oct  7 2019, 12:59:55)

• Operating System (Linux Distro and version or Mac or Windows)
  Ubuntu 18.04.5 LTS
• Container OS
  Alpine
• Python version (3.6 or higher)
  Python 3.6.8

Please attach files if they exist
As Github does not support uploading json format, I put tern.log and report.json into log_report.zip.

• tern.log & report.json
  log_report.zip

[rnjudge]

Thanks for the issue @chming1016. I think the difference in output you're seeing is actually due to the python3 -m command before the pip install requirements.txt which is causing Tern not to recognize the pip install command. Tern does not care about the arguments that come after pip install (be it a package name or requirements.txt, rather it only looks for an install command at the beginning of Dockerfile RUN arguments). If I build with the following Dockerfile that installs requirements.txt, Tern finds the requests package (and all the others installed via requirements.txt) in the report:

FROM python:3.9-slim-buster
COPY requirements.txt .
RUN pip install -r requirements.txt

Ideally, Tern would be able to understand the python3 -m wrapper around the pip install command. Perhaps this is something we can look into adding. Tern does understand pip install requirements.txt on its own, though.

[chming1016]

Thanks for clarify the issue @rnjudge . Hoping this can be added in the closely future.

[rnjudge]

Feel free to open a PR with the fix if you need it quickly :) Is there a reason you're using python -m to install requirements.txt when you could just run pip install requirements.txt directly from the RUN command?

[chming1016]

Tern is a nice tool. I am using it to generate SBoM for cloud service to do vulnerability management.
So far, I might not be able to create a PR before spending more time on my non-working time to go through Tern's code.

Actually, I do not mind changing my Dockerfile from RUN python -m pip install -r requirements.txt to RUN pip install -r requirements.txt.
However, as Dockerfiles are controlled by other developers, we can not controlled how they code.
This should be considered as an issue.

[rnjudge]

You are correct, developers will put whatever they want in their Dockerfile :) Thus, we cannot account for every single possible way developers may install packages. We try to detect the most common methods of installing software. Installation methods like the python -m case I would consider somewhat of a corner case given that pip install is what we see most commonly to install python packages. I will leave this issue open and try to work on a fix before the next release but I am on vacation for the next 2 weeks so it won't happen before that. Thanks for opening this and for understanding!

[chming1016]

Hi @rnjudge,
I have found another similar issue.
In some corner cases, when using RUN --mount=type=cache,target=/root/.cache/pip pip install -r /requirements.txt in Dockerfile, the report does not show any python module.
As a workaround, I will try to convince developers to use a more common way to install pip modules in Dockerfile in current, so that Tern is able to recognise pip command.

[mukultaneja]

@rnjudge, Can I take this issue?

[nishakm]

Since we've started pulling images using Skopeo, we can no longer analyze post built images unless they're on a registry. I've tried using tern report -d instead which will call docker to build the image.