enables multiple projects in the same knowledge base

[ivg]

This PR paves the road to the global knowledge base which will be storing information about several binaries (executables, libraries, object files, etc). It also brings a few new functional and convenience features. The quick summary is below:

1. enables multiple projects in the same knowledge base;
2. adds scoping of the procedures stored in the knowledge base;
3. introduces a new knowledge base class for code units;
4. extracts more information about binaries from the llvm loaders;
5. introduces unit biasing, fixes the base address is not respect by many components #1126
6. scopes information sources to the corresponding paths
7. renames the bap.std package to bap
8. adds bap compare command together with the new collators extension point
9. introduces rules for introspecting procedures stored in the knowledge base
10. joins symbolizers via the new possible-name property to collect all the aliases
11. moves logging and event facilities to the bap-main library
12. adds more parsers to the Bitvec module
13. updates all information sources to enable their cooperation with more than one file
14. rewrites the objdump plugin, it now also able to provide address from name
15. provides extensible abstractions for language, architecture, abi, etc
16. publishes and extends the Toplevel module
17. refines the project documentation

New features

The compare command and callgraph-collator

This is a new command that showcases the new ability to store multiple same or different files in the knowledge base. It comes with a new extension injection point called collators (sorry, the compare and comparators names are already too busy in OCaml). A collator is an object that takes two projects and compares them to its taste. Its interface is designed in such way that we don't need to store in the residential memory more than one project or to store information about more than two projects when we compare several files. As a demonstration, we implemented a collator that compares N binaries by comparing their callgraphs.

Knowledge Base Rules and their introspection

Procedures, stored in the knowledge base, are totally opaque so it is hard to guess which property is provided by which procedure and what are the dependencies. To enable transparency, we now explicitly and statically describe all rules, using the new rule description eDSL. The description doesn't affect the semantics and is purely for information purposes. To get all the rules, stored in the knowledge base, just type bap list rules, this is, for example, how symbolizers are described in terms of the knowledge base rules,

-- [Symbolizer.provide s] reflects [s] to KB.
bap:reflect-symbolizers(symbolizer) ::=
  core-theory:unit-bias
  core-theory:unit-path
  core-theory:label-unit
  core-theory:label-addr
  bap:arch
  -------------------------
  core-theory:possible-name

Implementation Details

The problem

When a project is created using the Project.create function it uses the internal knowledge base and stores all the information about the disassembled binary in it. When the Project.create function is called the second time, it most likely will end up in a conflicting state since we are indexing objects by their virtual addresses and is quite probable that two binaries will have different instructions at the same addresses. In addition, our old information sources, such as branchers, rooters, and symbolizers, albeit being deprecated are still playing an important role in our infrastructure and they were reflecting their information directly into the knowledge base every time a new file is opened, so that if several binaries were opened in a row they will keep computing roots and names for both of them at the same time, which also will lead eventually to the conflicting information.

Solution

We represent program labels as the knowledge base symbols, which were interned in the core-theory package. Instead, we can intern them in the current-package and set the current-package variable to a different value for different files projects. That enables the same addresses to be distinguished if the came from different files.

Scoping information sources is a bit harder. First of all, we introduced promising and providing operations to the Knowledge interface that temporary store procedures in the knowledge base. It is not, however, always possible to stretch the right scope, moreover, sometimes it is very convenient when the procedure is stored in the knowledge base indefinitely and provide information even after the project is reconstructed and maybe even after the knowledge base is persisted and loaded again.

To achieve these goals, we introduced the notion of code units. A code unit denotes a set of instructions that share common properties and attributed each instruction a unit. From the unit, we can obtain information about the file name, target architecture, and even the programming language from which the instruction was compiled. A well-behaved information provider, when it computes some instruction property, can now look into its unit and figure out its origin and check if it matches with its own source of information, e.g., when radare2 symbolizer is readings its symbols from the file named foo and the address comes from a unit that belongs to a file named bar it should not provide any symbolic information about that address. For the old information sources, such as rooter, symbolizer, and brancher, we enabled this behavior automatically, by adding the path property to the sources. When the source is provided to the knowledge base, the stored procedure checks if the path in the source matches the origin path of the address.

The objdump plugin is not using the old interface anymore and is fully rewritten. The idea is to devise the interface for providing information that will be used instead of the old symbolizers and rooters. It is still a work-in-progress that we have cherry-picked from the branch that delivers Ghidra support. The new implementation of the objdump plugin is a proof-of-concept that operates fully from the knowledge base (it is not even dependent on the Bap.Std interface). For each address, it obtains its unit and if it doesn't have any information about it, it opens and parses that file and then provides the obtained information as usual. It also provides a new service that is dual to the service that our symbolizers provide, it is able to resolve names to addresses. This latter service uncovered a long-term bug that we have in BAP but that went unnoticed. Namely, that --llvm-base option wasn't properly addressed by all our information providers. When this option was used, our llvm loader was rebasing the binary and, as a result, all addresses were different from what other information sources that rely on the original data were seeing. At the end, objdump, radare, and ida, were providing information for the real addresses not for the shifted. That also led to multiple conflicts.

The real culprit was the original design, as we shouldn't have this option at all on the llvm level, but instead handle it globally. But it is too late to change anything so we provided a more general solution. We introduced the bias property to our unit class, which denotes that all addresses in this unit are biased with respect to the real addresses. Then we updated our information providers to respect that bias, both when we need to go from biased to real addresses and back from real to biased. To minimize the impact, we added automated handling of biases to our old information sources, rooter, symbolizer, and brancher. We assume that they are all providing unbiased addresses and automatically subtract the bias before passing the addresses to them. Only the information sources that were obtained from the image source are considered biased, so this extra correction is not needed. Right now we do not allow users to create explicit biased sources as we don't see the real need for that, but later we may publish this interface.

[ivg]

@gitoleg, take a closer look at the stub-resolver, as I had to rewrite it substantially.