Submodules may contain other submodules. Do process and add those to the source tar. #457

arcivanov · 2023-04-28T19:57:04Z

fixes #456

FrostyX · 2023-04-28T20:05:31Z

Thank you for the PR @arcivanov,
do you have some project on which I can try it?

arcivanov · 2023-04-28T20:09:04Z

Thank you for the PR @arcivanov, do you have some project on which I can try it?

Yes. https://github.com/karellen/karellen-sysbox
The builder in https://github.com/karellen/karellen-sysbox/blob/master/.tito/tito.props is what I'm donating back.
The only change is formatting to satisfy your linting requirements.

arcivanov · 2023-04-28T20:12:15Z

@FrostyX yield from doesn't seem to be supported in epel7 builds. What versions of Python have to be supported?

arcivanov · 2023-04-28T21:26:49Z

@FrostyX fixed

FrostyX · 2023-05-01T21:05:38Z

Thank you for the CI fixes and so on :-)

The PR works for me. I only found one issue, when I did

git clone https://github.com/karellen/karellen-sysbox.git
cd karellen-sysbox/
tito build --srpm

And forgot to pull the submodules first. It fails with a traceback:

[jkadlcik@zeratul karellen-sysbox]$ tito build --srpm 
Creating output directory: /tmp/tito
Checking for tag [karellen-sysbox-0.6.1.17-3] in git repo [https://github.com/karellen/karellen-sysbox.git]
Building package [karellen-sysbox-0.6.1.17-3]
ERROR: Error running command: git archive --format=tar --prefix=karellen-sysbox-0.6.1.17/sysbox/ f1cc75cfe5f13cc7f86fb3a0e16b2ae19748c064: --output=/tmp/tito/rpmbuild-karellen-sysboxekzstf_9/SOURCES/karellen-sysbox-0.6.1.17.tar.initial.sysbox

Status code: 128

Command output: fatal: not a valid object name: f1cc75cfe5f13cc7f86fb3a0e16b2ae19748c064:

Traceback (most recent call last):
  File "/usr/bin/tito", line 33, in <module>
    sys.exit(load_entry_point('tito==0.6.22', 'console_scripts', 'tito')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/tito/cli.py", line 910, in main
    CLI().main(sys.argv[1:])
  File "/usr/lib/python3.11/site-packages/tito/cli.py", line 209, in main
    return module.main(argv)
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/tito/cli.py", line 392, in main
    return builder.run(self.options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/tito/builder/main.py", line 162, in run
    self.srpm()
  File "/usr/lib/python3.11/site-packages/tito/builder/main.py", line 221, in srpm
    self.tgz()
  File "/usr/lib/python3.11/site-packages/tito/builder/main.py", line 573, in tgz
    self._setup_sources()
  File "/tmp/karellen-sysbox/.tito/libs/karellen/tito/submodule_aware_builder.py", line 60, in _setup_sources
    self.create_tgz(
  File "/tmp/karellen-sysbox/.tito/libs/karellen/tito/submodule_aware_builder.py", line 185, in create_tgz
    for submodule_tar_file in self._submodule_archives(
  File "/tmp/karellen-sysbox/.tito/libs/karellen/tito/submodule_aware_builder.py", line 136, in _submodule_archives
    self.run_git_archive(
  File "/tmp/karellen-sysbox/.tito/libs/karellen/tito/submodule_aware_builder.py", line 101, in run_git_archive
    run_command(git_archive_cmd)
  File "/usr/lib/python3.11/site-packages/tito/common.py", line 305, in run_command
    raise RunCommandException(command, status, output)
tito.exception.RunCommandException: Error running command: git archive --format=tar --prefix=karellen-sysbox-0.6.1.17/sysbox/ f1cc75cfe5f13cc7f86fb3a0e16b2ae19748c064: --output=/tmp/tito/rpmbuild-karellen-sysboxekzstf_9/SOURCES/karellen-sysbox-0.6.1.17.tar.initial.sysbox

It probably failed with a traceback even before your PR, but since you are touching that part of the code, could you please catch the exception and print some reasonable error message? Ideally, showing users the command to pull the submodules?

FrostyX · 2023-05-01T21:07:44Z

Hello @t0fik, you recently did some coding on SubmoduleAwareBuilder. Can you please take a look if this PR works for you?

FrostyX · 2023-05-01T21:13:21Z

Also @areese, do you want to take a look, please?

arcivanov · 2023-05-01T21:18:10Z

It probably failed with a traceback even before your PR, but since you are touching that part of the code, could you please catch the exception and print some reasonable error message? Ideally, showing users the command to pull the submodules?

I had the exactly same issue a couple of times 😄
I could do is check if the submodule directory contains .git subdir and if not raise an error asking whether they forgot to recursively pull submodules. Or I could try pulling the submodule recursively and try again.

Please let me know what you think.

BTW, this is running on Fedora COPR already.
https://copr.fedorainfracloud.org/coprs/karellen/karellen-sysbox/

FrostyX · 2023-05-01T21:29:31Z

src/tito/builder/submodule_aware_builder.py

+                    submodule_commit,
+                    submodule_tar_file,
+                    submodule,
+                )


I think, maybe you can run the self.run_git_archive(...) inside of a try-except block and if you find something like "not a valid object name" in the exception, you can create some user-friendly message and do raise TitoException(msg).

Sure, but the failure could be something else like out of disk space etc. to blindly try-catch or I'd have to parse the error.
There is no situation where the submodule is checked out recursively but there is no .git sub-directory though, i.e. it's a sure way to check if it was correctly pulled.

You are right :-)

Does this work for you?

$ tito build --srpm --debug Creating output directory: /tmp/tito Added lib dir to PYTHONPATH: /home/arcivanov/Documents/src/karellen/karellen-sysbox/.tito/libs ---- Builder class is None Importing karellen.tito.SubmoduleAwareBuilder Using builder class: <class 'karellen.tito.submodule_aware_builder.SubmoduleAwareBuilder'> Building in temp dir: /tmp/tito/rpmbuild-karellen-sysbox2l092siv Getting latest package info from: /home/arcivanov/Documents/src/karellen/karellen-sysbox/.tito/packages/karellen-sysbox Command: awk '{ print $1 ; exit }' /home/arcivanov/Documents/src/karellen/karellen-sysbox/.tito/packages/karellen-sysbox Status code: 0 Command output: 0.6.1.17-3 Command: git ls-remote ./. --tag karellen-sysbox-0.6.1.17-3 | awk '{ print $1 ; exit }' Status code: 0 Command output: 559a13650b1241d37cff335b5b2df0d36d3d2d39 Local tag SHA1: 559a13650b1241d37cff335b5b2df0d36d3d2d39 Command: git config remote.origin.url Status code: 0 Command output: ssh://git@github.com/karellen/karellen-sysbox Command: git config remote.origin.url Status code: 0 Command output: ssh://git@github.com/karellen/karellen-sysbox Checking for tag [karellen-sysbox-0.6.1.17-3] in git repo [ssh://git@github.com/karellen/karellen-sysbox] Command: git ls-remote ssh://git@github.com/karellen/karellen-sysbox --tag karellen-sysbox-0.6.1.17-3 | awk '{ print $1 ; exit }' Status code: 0 Command output: 559a13650b1241d37cff335b5b2df0d36d3d2d39 Remote tag SHA1: 559a13650b1241d37cff335b5b2df0d36d3d2d39 Command: git ls-remote ./. --tag karellen-sysbox-0.6.1.17-3 | awk '{ print $1 ; exit }' Status code: 0 Command output: 559a13650b1241d37cff335b5b2df0d36d3d2d39 Command: git rev-list --max-count=1 559a13650b1241d37cff335b5b2df0d36d3d2d39 Status code: 0 Command output: a270b8ed4363962ec55efd2d41788fc2fc30dad4 Got package metadata: ['0.6.1.17-3', './'] Building package [karellen-sysbox-0.6.1.17-3] Creating karellen-sysbox-0.6.1.17.tar.gz from git tag: a270b8ed4363962ec55efd2d41788fc2fc30dad4... Command: git rev-list --timestamp --max-count=1 a270b8ed4363962ec55efd2d41788fc2fc30dad4 | awk '{print $1}' Status code: 0 Command output: 1682902338 Command: git archive --format=tar --prefix=karellen-sysbox-0.6.1.17/ a270b8ed4363962ec55efd2d41788fc2fc30dad4: --output=/tmp/tito/rpmbuild-karellen-sysbox2l092siv/SOURCES/karellen-sysbox-0.6.1.17.tar.initial Status code: 0 Command output: Command: git config --file .gitmodules --get-regexp path Status code: 0 Command output: submodule.sysbox.path sysbox Cleaning up /tmp/tito/rpmbuild-karellen-sysbox2l092siv ERROR: '/home/arcivanov/Documents/src/karellen/karellen-sysbox/sysbox/.git' path does not contain '.git' directory. Have you cloned the repository recursively?

Thank you!

...karellen/karellen-sysbox/sysbox/.git' path does not contain '.git'

I am nitpicking: This is a bit confusing, I would drop the .git from the path.

Have you cloned the repository recursively?

Maybe suggesting the git submodule update --init --recursive command so that we don't have to search for it on stack overflow? 😄

But at this point, this PR has +1 from me anyway.

Nevermind. I have to go by the .git file, not .git directory. Fixing now.

All good, the check is now that there is a file .git (directory or file) and that is checked.

The build passed no problem: https://copr.fedorainfracloud.org/coprs/karellen/karellen-sysbox/build/5865413/

Perfect, thank you very much @arcivanov.
Works for me, and the error message is now super helpful.

I'll leave the PR open for a couple of days so that anybody else has enough time to review if they want to. Then I'll merge on Monday.

Oh, it's two weeks already? Sorry about that.

to the source tar. fixes rpm-software-management#456

arcivanov force-pushed the issue_456 branch 2 times, most recently from f9f72fb to 5b21da9 Compare April 28, 2023 21:00

FrostyX reviewed May 1, 2023

View reviewed changes

arcivanov force-pushed the issue_456 branch 2 times, most recently from a3d9b72 to f9efca7 Compare May 1, 2023 22:15

Submodules may contain other submodules. Do process and add those

53803cb

to the source tar. fixes rpm-software-management#456

arcivanov force-pushed the issue_456 branch from f9efca7 to 53803cb Compare May 1, 2023 22:45

FrostyX merged commit 835309b into rpm-software-management:master May 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Submodules may contain other submodules. Do process and add those to the source tar. #457

Submodules may contain other submodules. Do process and add those to the source tar. #457

arcivanov commented Apr 28, 2023

FrostyX commented Apr 28, 2023

arcivanov commented Apr 28, 2023

arcivanov commented Apr 28, 2023

arcivanov commented Apr 28, 2023

FrostyX commented May 1, 2023

FrostyX commented May 1, 2023

FrostyX commented May 1, 2023

arcivanov commented May 1, 2023

FrostyX May 1, 2023

arcivanov May 1, 2023

FrostyX May 1, 2023

arcivanov May 1, 2023

FrostyX May 1, 2023

arcivanov May 1, 2023

arcivanov May 1, 2023

FrostyX May 3, 2023

arcivanov May 12, 2023

FrostyX May 16, 2023

Submodules may contain other submodules. Do process and add those to the source tar. #457

Submodules may contain other submodules. Do process and add those to the source tar. #457

Conversation

arcivanov commented Apr 28, 2023

FrostyX commented Apr 28, 2023

arcivanov commented Apr 28, 2023

arcivanov commented Apr 28, 2023

arcivanov commented Apr 28, 2023

FrostyX commented May 1, 2023

FrostyX commented May 1, 2023

FrostyX commented May 1, 2023

arcivanov commented May 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment