Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add pkgbase to buildorder output #9

Closed
rmarquis opened this issue Jul 14, 2017 · 4 comments
Closed

Add pkgbase to buildorder output #9

rmarquis opened this issue Jul 14, 2017 · 4 comments

Comments

@rmarquis
Copy link

For reference, quick extract of our emails discussion from a few weeks ago:

[...]
libsystemd-git is provided by the systemd-git PKGBUILD, but it's difficult to establish this reverse mapping without a fair bit of extra machinery. Can pacaur handle this nicely, or does it need to avoid the libsystemd-git dep because it's implicitly handled by build+install of systemd-git?

Pacaur does that extra bit of machinery by itself [...] It does some pkgbase magic just before building to ensure a single build per pkgbase, and installation of spit packages in the correct order.

Cool. I've already got a naive graph implementation that can walk package deps and print a build order. I'll take this extra coalescing step as a nice-to-have. Certainly seems like an optimization that could be applied later...

Getting the pkgbase list is useful, f.e. to:

  • easily get the path of sources files in order to review them
  • obtain the packages to download, so git clone can be used instead of auracle download
  • handle split packages build

I initially thought I would use the upcoming custom output to retrieve the list of pkgbase (with auracle info --format %b or similar), but I realize now that obtaining the pkgbase list directly through buildorder might be a wiser generic implementation. In fact, I am not sure when the pkgname list alone can be useful without also getting the related pkgbase list (maybe apart from inspecting a build chain).

Having an extra column that outputs pkgbase directly is straighforward to implement and would also avoid to use an extra RPC request.

$ ./auracle buildorder systemd-git
BUILD libsystemd-git systemd-git
BUILD systemd-git systemd-git

Drawback is that the output is not so nice to look at anymore, and I am also not sure how this fit with the optimization you were referring to. Opinion?
@falconindy
Copy link
Owner

falconindy commented Jul 14, 2017 •
edited
Loading

Alternate output style: pkgbase first, followed by each of the pkgnames it provides, e.g.

BUILD systemd-git systemd-git libsystemd-git systemd-sysvcompat-git

@rmarquis
Copy link
Author

That wouldn't really work, since in that case the information of topologically sorted dependencies between split packages would be lost.

As a reminder, all subpackages of a split package are built, but they are not necessarily all installed (this defeat the point of split packages, and there are even valid cases where subpackages conflict between themselves).

I can however imagine something similar to the following:

BUILD systemd-git systemd-git libsystemd-git

with only the required dependencies listed, and with the print order of these split dependency sorted accordingly. This sounds however like more complexity in both auracle and wrappers for little benefit imho.

Alternative possibility: keep current default output as it is, and add another optional flag to output both pkgbase and pkgname.

@rmarquis
Copy link
Author

Suggestion from @AladW on IRC:

Use something like

buildorder --format '%foo'

which sounds like an excellent solution to me.

@AladW
Copy link

AladW commented Jul 22, 2017 •
edited
Loading

An example of how this could be used:

https://paste.xinu.at/p3PTc

A possible command line using cower formatters and %U for URLPath:

auracle buildorder --format '%n %b %v %U' | column -t

@AladW AladW mentioned this issue Mar 5, 2018
Closed
falconindy added a commit that referenced this issue Apr 27, 2019
@falconindy
Extend buildorder output
16d2b99 
This should help to address a few user requests:

1) buildorder shouldn't consider what's installed and omit these
packages. Instead, we tag output to describe what steps (if any) need to
be taken. Output is either REPOS (available for install via pacman), or
AUR (can be built from the AUR). Either of these can be prefixed with
SAITSFIED to denote that the dependency is already installed locally. As
a last resort, a depedency might be UNKNOWN when the package isn't found
anywhere (#4).

2) buildorder should include the target package(s) in the output. (#32)

3) For packages which come from the AUR, the pkgbase will be emitted as
a third column, e.g.

  AUR systemd-libs-git systemd-git

This allows for deduping and also knowing the directory the
cloned/downloaded package will be found in (#9)

I'll no doubt gain a few new bug reports with this change...
falconindy added a commit that referenced this issue Sep 4, 2020
@falconindy
Rip out all of our cancellation logic, use sd_event_exit
9e8b12c 
Originally, I wanted to address the following problem:

When a user callback indicates failure, calling CancelAll() means that
we end up (re)invoking a curl socket callback from a socket callback,
leading to a double free (either in sd-event or curl). One possible
backtrace looks like:

    (gdb) bt
->  #0  aur::AurImpl::DispatchSocketCallback (this=0x618000000480, s=<optimized out>, action=4, io=<optimized out>) at ../src/aur/aur.cc:330
    #1  0x00007ffff74fa4e1 in singlesocket () from /usr/lib/libcurl.so.4
    #2  0x00007ffff74fe622 in curl_multi_remove_handle () from /usr/lib/libcurl.so.4
    #3  0x000055555570e833 in aur::AurImpl::FinishRequest (this=<optimized out>, curl=0x623000005500, result=<optimized out>, dispatch_callback=<optimized out>) at ../src/aur/aur.cc:462
    #4  0x0000555555708cc1 in std::__do_visit<std::__detail::__variant::__deduce_visit_result<void>, aur::AurImpl::Cancel(const value_type&)::Visitor, const std::variant<void*, sd_event_source*>&> (__visitor=...) at /usr/include/c++/10.2.0/variant:869
    #5  std::visit<aur::AurImpl::Cancel(const value_type&)::Visitor, const std::variant<void*, sd_event_source*>&> (__visitor=...) at /usr/include/c++/10.2.0/variant:1710
    #6  aur::AurImpl::Cancel (this=0x618000000480, request=...) at ../src/aur/aur.cc:291
    #7  0x00005555557090f6 in aur::AurImpl::CancelAll (this=0x618000000480) at ../subprojects/abseil-cpp-20200225.2/absl/container/internal/raw_hash_set.h:311
    #8  0x000055555570f08d in aur::AurImpl::CheckFinished (this=0x618000000480) at ../src/aur/aur.cc:485
->  #9  0x000055555570f341 in aur::AurImpl::DispatchSocketCallback (this=0x618000000480, s=<optimized out>, action=4, io=<optimized out>) at ../src/aur/aur.cc:332
    #10 0x00007ffff74fb092 in Curl_multi_closed () from /usr/lib/libcurl.so.4
    #11 0x00007ffff74cc631 in Curl_closesocket () from /usr/lib/libcurl.so.4
    #12 0x00007ffff74df551 in Curl_disconnect () from /usr/lib/libcurl.so.4
    #13 0x00007ffff74fc354 in multi_done () from /usr/lib/libcurl.so.4
    #14 0x00007ffff74fcc91 in multi_runsingle () from /usr/lib/libcurl.so.4
    #15 0x00007ffff74fe1d1 in multi_socket () from /usr/lib/libcurl.so.4
    #16 0x00007ffff74fe354 in curl_multi_socket_action () from /usr/lib/libcurl.so.4
    #17 0x000055555570f60d in aur::AurImpl::OnCurlTimer (userdata=0x618000000480) at ../src/aur/aur.cc:401
    #18 0x00007ffff7466b3e in ?? () from /usr/lib/libsystemd.so.0
    #19 0x00007ffff746821e in sd_event_dispatch () from /usr/lib/libsystemd.so.0
    #20 0x00007ffff746a6a9 in sd_event_run () from /usr/lib/libsystemd.so.0
    #21 0x0000555555704bd3 in aur::AurImpl::Wait (this=0x618000000480) at ../src/aur/aur.cc:495
    #22 0x000055555563ee8a in auracle::Auracle::GetOutdatedPackages (this=<optimized out>, args=std::vector of length 0, capacity 0, packages=<optimized out>) at /usr/include/c++/10.2.0/bits/unique_ptr.h:421
    #23 0x0000555555656816 in auracle::Auracle::Outdated (this=<optimized out>, args=..., options=...) at ../src/auracle/auracle.cc:564
    #24 0x0000555555596c7f in main (argc=<optimized out>, argv=<optimized out>) at ../subprojects/abseil-cpp-20200225.2/absl/container/internal/raw_hash_set.h:311

We could do that by making CancelAll() merely schedule another event
that performs the actual cancellation at a later point in order to avoid
the recursion. However, our cancellation logic is all sorts of weird and
makes assumptions about how events are dispatched (i.e. there might be
multiple at a time). Let's just get rid of all of this and use the
sd-event mechanism of sd_event_exit instead.

This does, however (as did the original proposed solution), have the
side effect of logging multiple times because we potentially open up to
5 connections to the AUR at once, e.g.

  $ build/auracle --baseurl http://129.168.255.1 outdated
  error: UNKNOWN: Connection timed out after 10000 milliseconds
  error: UNKNOWN: Connection timed out after 10000 milliseconds

I suppose one way to fix this would be to do response merging on the
backend to match the request splitting. That way, the frontend only gets
one response. I think that comes with a lot of weird potential behaviors
though (handling of partial failures, to mention one). Would be nicer if
the AUR didn't have the crap behavior and could take POST requests in
order to extend the arg limit.

Whatever, this is a weird edge case.

Fixes #82.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@falconindy @rmarquis @AladW