eng: the Engineering tool

What is it?

The eng tool is designed to perform project and product engineering tasks. Engineering tasks include:

• design
• development
• documentation
• release management
• bug tracking
• feature tracking
• task management
• requirements definitions
• configuration management

Most software developers and projects are familiar with the development process from an applied manner: projects have some sort of build description that lets a build tool create a runnable version of the project.

Some projects have documentation to varying degrees: README’s are common, guides for installation, usage, and quickstart are the next most common. Other documentation tends to be rare: requirements documents, design documents, API specifications, etc.

Many projects are maintained in a tool that provides varying amounts of additional capabilities such as release management and bug tracking, but those tend to exist in the tooling itself and separate from the project data. For example, Github provides issues (bug tracking), pull requests (development), and tags/artifacts (release management), but these are all part of Github itself and not available to anyone working with an offline clone of the repo.

The eng tool is intended to consolidate all these different aspects of engineering into the main portion of a project and provide a way to interact with them.

Discoverability

One of the keys to being able to use eng (or any tool) is discoverability. The ability to use a tool effectively means that the tool should be a supporting partner in its usage, for both beginning and experienced users. To this end, eng has been designed so that you can specify as little or as much as possible and eng can indicate what or how that command can be used.

When in doubt, just type eng, and don’t be afraid type type eng help or eng CMD to receive guidance.

What’s it for?

The eng tool is used when you want to think like an “eng”ineer.

Imagine a situation where you encounter a brand new project. One of your first questions is “where is the documentation”. While there is somewhat of a convention of looking for a README that provides some assistance, you could instead just enter $ eng doc, which would tell you that there is a list sub-command, so then you could type $ eng doc list to see all the different documentation available for the project.

The next thing you are likely to want to do is build the project. Instead of trying to figure out what build tool is used, and how to invoke it, just type $ eng dev. The project can define a “build” subcommand (via the _eng_ subdirectory) that will handle invoking the build tool. Maybe there are a couple of different targets that could be built; $ eng dev will show you those targets, and you can invoke each one.

What about testing the application? In the output of $ eng dev you can observe it reported subcommands for test and testlist. The $ eng dev testlist will display a list of the testsna and $ eng dev test will run them (the project-supplied _eng_ configuration provides the actual commands that eng will run to perform the tests).

When you think like an engineer, you want to “build” or “doc” or “test”. The eng tool provides the discoverable high-level coordination commands to do these things.

Note that eng will always echo the actual commands it is going to run to stdout. Nothing is hidden. This allows you to cut and paste the commands to modify them (adding or removing flags) or run them directly or in another environment. Using eng should not lock you in, using eng should feel like you have a trusty assistant helping you get things done.

Why not use ____?

Why not use make or a task management tool or a shell script or ____?

Simplicity

The eng tool is designed to be simple to use, and simple in what it does. It does not have lots of advanced features like internal scripting, dependency rebuild checking, etc. There are other tools that are better at those things (e.g. make/cmake/ninja/meson/bazel/etc. for dependency rebuild checking). Instead, eng simply collects those commands and their project-specific arguments into easy top-level operations.

This also means that eng has a large feature inertia. When considering whether to add a new feature to eng

Visibility

As mentioned above, eng always shows you what it does, so that you can run variations of those operations directly. It’s your assistant with a good memory of how to accomplish the main top-level tasks in the standard way.

Discoverability

While there is a way to embed help information in GNUmake, that is specific to GNUmake. Do you know how to embed that in your Makefile? Do you know how to get it when running make?

Do you know how to get help when running Meson? When running bazel? When running ninja? Or gradle? Or sbv? Or cargo? Or cabal? Or npm? Or gprbuild? There are lots of build tools, some are general and some are specific to language ecosystems, and sometimes a project even uses multiple of these.

So how do you figure out what you want to do? With eng, you can start by just typing eng. Then add more to the eng command line based on the help eng provides until you end up being able to do what you wanted to do without trying to read build configuration files.

Single purpose

As noted above, some projects use multiple build tools. Or try to adapt their build tool to do other things which don’t really match the build tool’s capabilities. And sometimes they supply scripts or other tools when they need additional functionality that the build tool cannot do.

The eng tool’s single purpose is to coordinate all the top-level activities. If some of those are achieved via a build tool and some are via a script, that’s all specified in the _eng_ configuration and eng provides a standard top-level interface to those operations.

Independence

The eng tool is not intended to replace any of the regular tools used by the project. It’s designed to coordinate their use from a top-level. Thus, it does not conflict with other tools, and it does not try to do things it is not well suited for.

Standardized interaction

Using eng changes very little across projects. Each project can have project-specific subcommands, but these are discoverable and self-documenting. If every project provided an eng configuration, then new users to a project (as well as long-time developers) can start with running $ eng and have good chance of being successful without devoting significant time to learning new tools and new configurations and processes.

And hey, why limit eng usage to humans? Now that eng knows how to coordinate the high-level project tasks like building and testing, just setup your CI to use eng.

Project-controlled Configurability

What eng is capable of doing is almost entirely specified by what the project places into the configuration files in the _eng_ directory. This means that it’s the responsibility of the project team to provide useful functionality via eng, but the _eng_ configuration is version controlled and can be updated/enhanced by both project developers and users because it uses a very simple text-based syntax and supports tracking changes to that configuration.

Installing

The only dependency for the eng tool is SWI Prolog. The SWI Prolog tool can be installed in most systems via the local package manager. The eng tool is OS independent and can be used under Linux, MacOS, or Windows.

To build and configure eng:

1. Download the eng repository and change to that directory
2. ./eng.sh dev fret_fetch
3. ./eng.sh dev build

   Note: use eng.bat instead of eng.sh under Windows.

The third step above creates and eng file, which is the executable file for the tool. You can copy this file to a location in your current PATH (e.g. /usr/local/bin, $HOME/bin, etc.).

Getting started

Once eng has been installed, the following commands will demonstrate the usage of eng and lead to further information:

$ eng
$ eng doc
$ eng doc list
$ eng doc info eng_tool
$ eng doc show eng_tool
$ eng help
$ eng help doc
$ eng system list
$ eng system gen Lando

How does it work?

Information for the eng tool is provided by adding .eng files to the project (normally in an _eng_ directory at the project’s top-level); each .eng file provides information related to one or more of the above areas, with data appropriate to that type of information. Although a single .eng file can contain multiple types of data, it’s more typical to use a different .eng file for each component of information. The .eng files are intended to be managed via the version control for the project just like the other sources.

The eng tool is run, specifying a sub-command. The tool will read the various eng files provided for the project to perform the requested sub-command. Each sub-command can use various portions of the information to perform an action associated with that sub-command. Some of the commands may create or update .eng files; this updates the project state managed by eng and these changes should be committed to the version control system to maintain accurate project state.

Syntax for .eng files

The syntax for the .eng files is associated with the type of information being provided; the “type” of information refers to a category of engineering tasks or concerns like to those described in the introduction section above. There may be (and usually are) multiple “components” of information in each category (e.g. multiple “bugs” in the bug tracking category). Each component is introduced by an “introductory phrase” that uniquely identifies the type of information; the introductory phrase is followed by additional phrases that together provide all the data for that component. If multiple components are presented in the same .eng file, each is started by a corresponding “introductory phrase” that terminates the information for any preceding component and begins the new component; multiple components in the same file do not necessarily need to have the same type.

Specific syntax is provided in detailed sections of the eqil_design documentation.

eng commands

The user invokes the eng tool with a sub-command to be performed in that particular run. The sub-commands are pre-defined, although many provide a way for the user (project developers) to extend or configure the command with regards to the specific actions it performs.

Many of the commands will update the .eng files; these updates should be preserved as they maintain the evolving engineering state of the project. Note that this means that “rolling back” to a previous commit for the project will also roll-back this state. Automated updates to the eng files are intended to be compatible with merge processes, but some sub-commands may perform parallel actions that will result in .eng conflicts when merged together; these situations represent Engineering practices that require manual resolution anyhow, so this is viewed as somewhat of a feature: resolving these conflicts in the .eng files helps resolve the associated Engineering issue. This latter statement is understandably vague; more information will be provided to clarify these in the description of the commands where these are likely to occur.

Tne various eng commands provided are described by simply issuing the eng command with no arguments. The eng help COMMAND can be used to get more information about a specific command.

Ease of use

The eng command is designed to be easily used to perform engineering management for a project. To facilitate this, it is utilized as a single file written in an interpreted language for which an interpreter is widely avaiable for multiple platforms. It is possible to copy the entire eng tool into the project itself as a single file, or else use an eng implementation referenced from the current PATH or other executable specification method.

All eng commands will provide help describing their action if passed the --help argument. Most commands provide a --dry-run argument that would describe what would be done without actually performing those actions, and the effects of any command which has impacts purely local to the working directory tree can be reversed by using the version control tool to discard or revert the changes; the same is not necessarily true for commands that have impact outside of the current working directory tree, although many of those commands provide a --revert argument; see their individual --help for more information.

Ouroboros

The eng tool is managed by: the eng tool. It is perhaps a more complex use of the tool than is useful for learning the tool, but it should be a good representative example of what can actually be done with the tool.