Skip to content

Packages

Jump to bottom Edit New page
Pedro Andrade edited this page Jul 18, 2017 · 93 revisions

Building TerraME Packages

Pedro R. Andrade, Raian V. Maretto

This tutorial works with TerraME version 2.0.0-RC4 or greater.

Summary

Introduction

Contributions to TerraME can be made in the form of packages. A package encapsulates a set of functionalities, models, and (or) data with a given purpose. This report describes the rules for building and validating packages in TerraME. A good way to learn about how to build packages in TerraME is by studying the code of some of the available packages. Additional information about packages can be found in the TerraME documentation, more specifically in UnitTest and Package.

General Structure

A package is stored as a directory. It might contain the following internal directories and files:

  • description.lua: Compulsory Lua file that describes the package. See section Package Description for more details.
  • license.txt: A file with the license of the package. See some options in http://www.r-project.org/Licenses/.
  • lua: Folder with the source code of the package. The functions in the files of this directory will be available for the simulation when the user loads the package using import() or getPackage(). See section Types, Models, and Functions for more details.
  • load.lua: Optional file that describes the order that the files in lua directory will be loaded. If this file does not exist, the files will be loaded in alphabetical order. If lua directory contains internal directories with Lua files, they will not be loaded unless explicitly described in load.lua.
  • tests: Folder with unit tests for the package. See section Tests for more details.
  • examples: Folder with examples of the package. See section Tests and Documentation for more details.
  • config.lua: A configuration file for testing the package. When running tests, if no configuration file is set, TerraME uses this file if it exists.
  • data: Folder with files storing data that can be used by tests or examples of the package. These files are typically used to build CellularSpaces, Societies, Neighborhoods, SocialNetworks, or placements, but can have any other usage. It can also have Lua scripts to create TerraView projects.
  • data.lua: A lua file that documents all data files. See section Documentation for more details.
  • font: Folder with files storing fonts that can be used to draw Agents in Maps.
  • font.lua: A lua file that documents all fonts of the package. See section Documentation for more details.
  • images: A directory that store image files related to the documentation.
  • log: A directory where signatures for the outputs of UnitTest:assertSnapshot(), UnitTest:assertFile(), and prints from examples are stored. The files can be stored directly within this directory or within internal directories with the name of the respective operational system ("windows", "linux", and/or "mac"). When using internal directories, the files directly within log are ignored.
  • logo.png: An optional image with the logo of the package to be shown in the HTML documentation. If this file does not exist, TerraME's logo will be shown in the HTML files.

Package Description

All the information about the package is documented in file description.lua. It is a common Lua file that defines some string variables, with the following semantics:

  • package: Name of the package.
  • title: Optional title for the HTML documentation of the package.
  • version: Current version of the package. It must be in the form <number>[.<number>]*. For example: 1, 0.2, 2.5.2.
  • date: Date of the current version. If this argument does not exist, then TerraME will automatically create it when building the package.
  • authors: Name of the author(s) of the package.
  • depends: A comma-separated list of package names which this package depends on. Those packages will be loaded before the current package when require is called. Each package name must be followed by a comment in parentheses specifying a version requirement. The comment must contain a comparison operator, whitespace and a valid version number. For example: "terrame (>= 1.2)".
  • contact: E-mail of one or more authors (optional).
  • content: A description of the package.
  • license: Name of the package's license.
  • url: An optional variable with a webpage of the package.

An example of description.lua is shown below.

package = "base"
version = "2.0"
date = "17 October 2014"
license = "LGPL 3.0"
depends = "terrame (>= 1.2), tube (>= 0.1)"
title = "TerraME Types and Functions"
url = "http://www.terrame.org"
authors = "INPE and TerraLAB/UFOP"
contact = "pedro.andrade@inpe.br, tiago@iceb.ufop.br"
content = "Detailed description of the package"

File Types

In a package, source code files group functions with similar purposes. There are three types of files in TerraME:

  • Model: File with the definition of a Model. The name of the file must have the same name of the model, plus ".lua". Each Model of a package must be implemented in a different file. Instances of the model can then be implemented in the examples of the package.
  • Type: Define a new type. Files with a function with the same name of the file will be considered as a new type by the package. This function is called constructor. All functions inside this file will be considered as functions related to the objects returned as result of calling the constructor. It means that the first argument of each function will be considered the instance of the new type. Every function belonging to a type must have the first argument named self for documentation purposes.
  • Functions: Describe a set of functions. The name of the file is independent from the names of the functions it contains, but in some way it should summarize its internal content.

In any of the files above, if it contains internal functions that should not be published by the package, such functions need to be declared as local.

Tests

Every global function of the source code must be tested in order to validate the package. In the directory tests, each file tests the functions of the file with the same name in the source code. Test files must return a table containing functions with names equal to the names of the functions of the respective file in the source code. Each of these functions get a UnitTest as argument and must have at least one assert. For example, the code below implements tests for functions belong() and levenshtein(), declared in the source code.

return{
    belong = function(unitTest)
        local mvector = {"a", "b", "c", "d"}

        unitTest:assert(belong("b", mvector))
        unitTest:assert(not belong("e", mvector))
    end,
    levenshtein = function(unitTest)
        unitTest:assertEquals(levenshtein("abv", "abc"), 1)
        unitTest:assertEquals(levenshtein("bvaacc", "bcaacac"), 2)
        unitTest:assertEquals(levenshtein("xwtaacc", "caacac"), 4)
    end
}

Each test function takes as argument an instance of UnitTest. This argument can be used to execute asserts, verifying results of the functions from the package. See the available asserts in the documentation of UnitTest.

Errors and Warnings

Functions usually require specific types or values in their arguments. If such requirements are not followed, they must stop the simulation with an error. In the documentation of Package, there are some useful functions to verify arguments of functions and to produce errors and warnings, such as customError(), verify(), and invalidFileExtensionError().

It might be interesting to test errors in your package to ensure that the package produces the right error messages. The way to do so is by generating the error in the source code functions and then use UnitTest:assertError() to check the error message. For example, the first argument of forEachElement() is mandatory:

function forEachElement(obj, func)
    if obj == nil then
        mandatoryArgumentError(1)
    --- ...
end

Every error function in Package has one or more associated functions ending with Msg that return the respective error message. Therefore, to test this error, we can use the code below:

return{
    forEachElement = function(unitTest)
        local error_func = function()
            forEachElement()
        end
        unitTest:assertError(error_func, mandatoryArgumentMsg(1))
    end
}

In the case of warnings, we can use UnitTest:assertWarning() to capture them. The code executed along an assertWarning() must execute without any error. Only a single warning must be produced. If a warning is found along tests outside an assertWarning(), it will be considered an error.

Random numbers

The internal random number generator of TerraME uses a seed with the current simulation time. It means that, as default, random numbers of two different simulations will not be the same. However, in the tests, it is necessary to have exactly the same random numbers to ensure that the asserts will not fail. Because of that, TerraME automatically sets the seed to zero before any test, to avoid that one test that uses random numbers affects the next ones. In any case, it is possible to set another seed along any test.

Data

Data tests require external files or connections to a DBMS. When the tests use files available in directory data, it is possible to implement tests using filePath(), available in TerraME (see its documentation in Package). For example, the following command creates a CellularSpace from a CSV file available in the package base, the core package of TerraME.

cs = CellularSpace{
    file = filePath("cs.csv", "base"),
    sep = ";"
}

unitTest:assertType(cs, "CellularSpace")
unitTest:assertEquals(2500, #cs)

When the tests use a database connection, they might use file config.lua through getConfig() (see Utils) to get information about how to connect to a database. Note that config.lua must be placed in the same directory TerraME will be executed for testing the package. Using this strategy guarantees that the tests could be run in different DBMSs or even in different machines without needing to change the test files. An example of config.lua is shown below:

password = ""
dbType = "mysql"

Using this file, one can create tests such as the one below:

CellularSpace = function(unitTest)
    local config = getConfig()
    local mdbType = config.dbType
    local mhost = config.host
    local muser = config.user
    local mpassword = config.password
    local mport = config.port
    local mdatabase = "cabeca"

    local cs = CellularSpace{
        dbType = mdbType,
        host = mhost,
        user = muser,
        password = mpassword,
        port = mport,
        database = mdatabase,
        theme = "cells90x90"
    }
}

Log

The directory log stores all files that can be used as signatures for some tests. Three situations use such directory:

  1. UnitTest:assertSnapshot()
  2. UnitTest:assertFile(), or
  3. examples that print() some output.

In each of these three cases, the tests verify if the file exist in the log/system directory, where system is the current operational system (windows, linux, or mac). If yes, it will compare the output of the test with the signature within the log directory. Otherwise, it will copy the output to log and show an error indicating that the test should be executed again.

Examples

Examples describe the main functionalities of a package. They are scripts that must run stand alone, needing to import() the package itself explicitly.

Examples can contain calls to print(). During the tests, some print call in an example, it redirects its output to a file with the same name of the example plus an extension .log. After that, it tries to find a file with the same name in the log directory. If it exists, it checks if both have the same content (the argument tolerance in the configuration file allows lines to be different). Otherwise, it will copy the file to log directory and show an error in the report pointing out that the tests must be run again to check if the same output will be produced again.

When the example creates Charts or Maps, TerraME automatically uses UnitTest:assertSnapshot(). In the case of Chart, it calls assert in the end of the simulation. For a Map, it creates an assert when the Map is created and another in the end of the simulation. It is also possible to have some difference between the log created when the assert is executed for the first time and the test by using tolerance in the configuration file.

Configuration file

It is possible to run only part of the tests, or configure parts of its execution, using a file that might contain at lease one of the following optional variables:

  • directory: When the package contains lots of functions, it might be interesting to have an internal structure of directories, grouping tests into classes. This argument contains a string or a vector of strings describing the directory(s) to be tested. These strings can describe the name or part of the name of the directory to be tested. For instance, if directory is "basic" then directories such as "basic/core" or "alternative/displaybasic" will be tested. If directory is nil, then all directories will be tested.
  • file: A string or vector of strings with the files to be tested. If nil, all files within the selected directories will be tested.
  • test: A string or vector of strings with the name of the function(s) to be tested. If nil, all tests within the selected files will be tested. It is also possible to set it as false, to run only the examples of the package.
  • notest: A string or vector of strings with the name of the function(s) not to be tested. It cannot be used with test.
  • examples: A boolean value, indicating whether the examples should be executed. As default, examples are executed only when all tests are executed.
  • lines: A boolean indicating whether TerraME should check if all the source code lines are executed along the tests. As default, it will not execute this check because it adds too many verifications to the tests. In any case, if there is a test function that the modeler wants to avoid such verification, it is possible to write debug.sethook() to avoid source code verification along the execution of such function. TerraME verifies the lines of code only when executing the test functions. Everything declared as global in the test scripts will not be taken into account.
  • time: A boolean value indicating whether TerraME should show the time each test takes. When true, it adds one more output line for each test. Each test that takes more than five seconds is printed with a yellow background, and with red background when it takes more than 20 seconds. The default value is false.
  • tolerance: A number between zero and one that indicates the error tolerance in the examples. It is valid for both the printed lines and the saved images. In the case of lines, each line has the given tolerance, which means that the number of characters that can be diffferent depends on the size of the line.

The code below shows an example of a config file:

examples = false
directory = "core"
file = {"CellularSpace.lua", "Society.lua"}
test = {"synchronize", "add"}

Executing tests

The command to test a package is:

terrame -package pkg -test

If package pkg is installed then TerraME runs its tests. If not, then TerraME checks if there is a local directory with this name storing the package. If directory tests does not exist when this command is run then TerraME will automatically create the directory and files according to the source code of the package. It fills each file with test functions, indicating where the modeler should define the tests.

It is also possible to add -color in order to have a coloured output, which helps to find errors:

terrame -color -package pkg -test

Under Mac and Linux, such colors are automatic in the terminal. Under Windows, you must install a software such as ansicon. To install it, just download, enter in the directory x64 or x86, according to your computer spec, and then run ansicon -i.

A configuration file can be used as optional argument, in the following way:

terrame [-color] -package pkg -test test-file.lua

The tool to test packages executes the following verifications:

  1. Check if the package can be loaded.
  2. Check if the package prints messages when loading.
  3. Check if every test function executes correctly, checking its asserts.
  4. Check if every test function has at least one assert.
  5. Check if every test function exists in the source code of the package. If the name of a file that contains a set of tests does not match any file in the source code, this verification is skipped for such file. In this sense, there can exist more functions in the tests than in the source code of the package.
  6. Check if any function creates global variables when executed. Global variables must be created only when the package is loaded.
  7. Check if the tests execute some print(). It is possible to print() only in the examples.
  8. Check if error messages point to the test files. This kind of error can occur when the package uses Lua functions error() or assert() directly, instead of the ones available in Package, or when there is some unexpected internal error in the package.
  9. Check if all functions of the source code have at least one test.
  10. Check if every assert was executed at least once. It is possible to ignore this verification for specific asserts by adding the word SKIP within a comment in such line. All asserts within comments must also have SKIP to be ignored, as it is understood that in principle asserts should not be commented.
  11. Check if all examples execute correctly. A package must have at least one example to be validated.

Source Code Verification

Some minor problems in the source code cannot be verified using test functionalities. For example, when one declares a variable and does not use it. As Lua is an interpreted language, it might be possible that the user declared a variable with a name and is using it with a different name, which might produce an error. TerraME allows a package to be verified to find this kind of bug. To execute this functionality just run:

terrame [-color] -package pkg -check

Documentation

The documentation of TerraME is written using a slightly changed version of LuaDoc. Every non-local function of the package must be documented. LuaDoc looks for the sequence of three minus signs (---) in the source code of the package. This sequence of characters indicates the beginning of a documented comment. The documentation ends with the first line of code found. The following code defines a function and its documentation.

--- Round a number given its value and a precision.
-- @arg num A number.
-- @arg idp The number of decimal places to be used.
-- @usage round(2.34566, 3)
function round(num, idp)
    -- ...
end

The first sentence between --- and the first period will be the resume. The other sentences until the first tag @ belong to the description of the function. It is possible to use character \ to separate paragraphs in the description of a function.

The documentation above uses two tags:

-- @arg <argument> <text>

Describe function arguments. It requires the name of the argument and its description (<text>). The function above has two arguments, num and idp.

-- @usage <text>

Describe an example on how to use the function. It must be a piece of code that can be executed alone. For instance, when writing a package, it must import the package in the beginning of each usage. It is possible to avoid running such script when building the documentation if the source code contains the text DONTRUN. The usage must also call the documented function, otherwise an error will be prompted while building the documentation. Usage is optional for deprecated functions.

The corresponding HTML documentation for this function will be:

Note that the function gets arguments using names in the source code, but the arguments are described as positions in the HTML documentation. It is documented this way because the user does not use the name of the arguments, only their position, to call a function.

In the case of functions with named arguments, they have a single table as argument. In this case, the way to document the parameters is by using <table>.<attribute> as name for the parameter. See an example below, where the argument attrs is a named table with four fields.

--- A second order function to numerically solve ordinary 
-- differential equations with a given initial value.
-- @arg attrs.method the name of a numeric algorithm to 
-- solve the ordinary differential equations.
-- @arg attrs.equation A differential equation or a vector of equations.
-- @arg attrs.a The beginning of the interval.
-- @arg attrs.b The end of the interval.
-- @usage v = integrate {
--     -- ...
-- }
function integrate(attrs)
    -- ...
end

The documentation of this function is shown as follows. Note that attrs. is not shown in the final documentation.

Types

When a file defines a type, the function with the name of the type documents the type. It will be shown in the beginning of the webpage of the type, followed by the other functions in alphabetical order. For instance:

--- Type to generate random numbers.
-- @arg data.seed A number to generate the pseudo-random numbers.
-- Default is the current time of the system.
-- @usage random = Random()
--
-- random = Random{seed = 0}
function Random(data)
    -- ...
end

The arguments are then described as <argument-name>. In the HTML, we have the following documentation:

The first argument of all functions of the type must be named self. It means that it will not be documented, as it will come from the element in the left of the : in the function call. For example:

--- Reset the seed to generate random numbers.
-- @arg seed A positive integer number.
-- @usage value = random:reSeed(1)
reSeed = function(self, seed)
    -- ...
end

will produce the following HTML documentation:

Files without type

Files without type are documented with @header:

-- @header <text>

The <text> describes the content of a file. This argument does not require the --- in the beginning of the comment. Files with functions must never use self as first argument in any of its functions, otherwise they will be ignored by the documentation.

Models

Models are documented in the same way of named functions. All the models of a package will be placed in a single webpage. Functions init() and check() should not be documented, as well as any other user-defined function in the file.

Data files

Data files are documented in a file called data.lua. This file must have only calls to data{} (to document files) or directory{} (to document directories). data{} gets the following named arguments:

  • file: A string (or vector of strings) with the name(s) of the documented file(s).
  • summary: A string describing the data.
  • source: Where the data was taken from.
  • image: A string with a file name (stored in images directory of the package) to be displayed in the documentation.
  • reference: A string describing a reference to be cited by those who want to use the file.
  • attributes: A named table, with names representing the attributes of the data and values describing describing the respective attributes. When documenting raster data, bands are described as attributes, using strings in their names. Note that, when the names cannot be used directly, such as 1 = ... (Lua does not allow attribute names starting with numbers), it is necessary to use ["1"] = ..., with the same semantics.

The files within a directory do not need to be documented. Function directory{} gets the following named arguments:

  • name: A string with the name of the directory.
  • summary: A string describing the directory.
  • source: Where the data within the directory was taken from.
  • reference: A string describing a reference to be cited by those who want to use the data within the directory.

An example of a data.lua is shown below:

data{
    file = "agents.csv",
    summary = "A simple set of four agents",
    attributes = {
        name = "Name of the agent", 
        age = "Age of the agent"
    }
}

data{
    file = {"emas.tif"},
    reference = "Almeida, Rodolfo M., et al. (2008) ...",
    summary = "Land cover data on Parque Nacional das Emas, Brazil",
    attributes = {
        ["1"] = "A band with the cover type. Zero means forest, one means deforested."
    }
}

directory{
    name = "test",
    summary = "Directory with files used only for internal tests.",
    source = "TerraME team"
}

When there is more than one attribute with the same description of a given data, it is possible to group them into a table. For example, a given file with attributes defor2000, defor2005, and defor2010 can be documented as follows:

data{
    file = "defor.shp",
    summary = "Deforestation data",
    attributes = {
        {["defor2000", "defor2005", "defor2010"]} = "Deforestation for the years 2000, 2005, and 2010."
    }
}

TerraME will automatically add more information from such files to the HTML documentation. They are:

  • The geometry of the data and the number of objects,
  • The geospatial projection, and
  • The type of each attribute or band available (for files), and
  • The number of files, and
  • The available file extensions (within directories).

In data directory, it is also possible to have lua files to create TerraView projects and cellular layers using terralib package. One can run all Lua scripts of a package using argument -project, on in the button Project in the graphical interface. The automatically created files will be documented by TerraME, and must not belong to data.lua. Lua files within data directory must also not be documented in data.lua.

Font files

Font files are documented in a file called font.lua. This file must have only calls to font{}, which gets the following named arguments:

  • name: The name of the font to be used as argument font for Map.
  • file: A string with the name of the file for the font in directory font.
  • source: Where the data was taken from.
  • summary: A description for the font.
  • symbol: A table mapping symbol names to their values in the font. The names can be used as value to argument symbol for Map.

An example of a font.lua is shown below:

font {
    name = "Pet Animals",
    file = "Pet Animals.ttf",
    source = "http://www.dafont.com/pet-animals.font",
    summary = "Pet animals by Zdravko Andreev, aka Z-Designs.",
    symbol = {
        fish   = 66,
        bird   = 77,
        horse  = 78,
        pig    = 80
    }
}

font {
    name = "JLS Smiles Sampler",
    file = "JLS Smiles Sampler.ttf",
    source = "http://www.dafont.com/jls-smiles-sampler.font",
    summary = "Font by Michael Adkins & James Stirling.",
    symbol = {
        smile = 65,
        pirate = 74,
        skeleton = 86,
        mustache = 99
    }
}

HTML tables

Every function with named arguments can have HTML tables. It is useful to describe arguments that represent strategies or options, described as strings.

-- @tabular <argument>

Tag @tabular builds a table for a given <argument> of a function with named arguments. It is possible to avoid referring to an argument by using NONE instead of an argument name. Tables are placed in the same order of its declaration. Values of tables are separated by & (columns) or \ (lines). The first line of a table is recognised as titles, so they will be displayed in bold font. If any of these names have the word arguments (not case sensitive) then all the elements of the column need to be an argument of the function. Additionally, LuaDoc will check if all arguments of the function are used in this column at least once. The example below describes a @tabular:

-- @tabular strategy
-- Strategy & Description & Compulsory Arguments & 
-- Optional Arguments \
-- "coord" & A bidirected relation. & target & name \
-- "function" & A Neighborhood based on a function. & 
-- filter & name \
-- "moore"(default) & Eight touching Cells. & &
-- name, self, wrap \    
-- "vonneumann" & Rook Neighborhood. & & name, self, wrap

The table below will then be created:

Internal links

It is possible to create internal links in the documentation. They can be created using <file-without-.lua>::<function-name>. The types defined in the package are identified along the documentation and links are automatically created in the HTML documentation. It is also possible to create links in the end of each function:

-- @see <text>

Refers to the documentation of other functions. Every reference in @see must also follow the standard <file-without-.lua>:<function-name>.

Images

It is possible to add images to the documentation of examples and models.

-- @image file.bmp

The file needs to be stored in the images directory of the package.

Deprecated functions

Deprecated functions are going to be removed from the package in the next versions. It is useful to have them to guarantee back compatibility and warn the user without producing any error at all.

-- @deprecated <link>

The <link> describes the function that can be used instead of the deprecated one. It can be an internal link or a description of what the user should use instead of the deprecated function.

Examples

Examples are documented using the tag @example.

-- @example <text>

The <text> describes the example. This argument does not require the --- in the beginning of the comment. It is possible to use character \ to separate paragraphs in the description of an example. Examples can also have tags @arg, describing global values that could be used as arguments for the script.

Building documentation

The command to build the documentation of a package is:

terrame [-color] -package pkg -doc

If package pkg is installed then TerraME builds its documentation. If not, then TerraME checks if there is a local directory with this name storing the package. The tool to build the documentation executes the following verifications:

  1. Check if every global function of the package is documented, as well as functions inside of global tables.
  2. Check if every non-named argument is documented.
  3. Check if every @tabular uses all available arguments of the function, in the columns that contain argument, and if all elements of @tabular are arguments of the function.
  4. Check if there is any unnecessary tag in the documentation.
  5. Check if every global function or functions inside of global tables have an @usage, and if each of them contains a call to the function itself.
  6. Check if all examples are documented.
  7. Check if all data files are documented.
  8. Check if every internal link is valid.

TerraME will automatically create a directory doc in the respective package and save the HTML documentation in such directory. It will create the following HTML webpages:

  1. An index.html, the main webpage describing the package using the content of description.lua. It also contains links to the other webpages.
  2. One webpage for each file in the directory that contain a type. The beginning of such webpages will describe the constructor of the type. Such files will be put in the group "Types" in the menu bar.
  3. One webpage for each file that contains functions (with @header).
  4. One webpage for the Models.
  5. One webpage for all examples.
  6. One webpage for the data.

Building Package

To build a package, we need to use the option -build:

terrame [-color] -package <pkg> -build <config> [-clean]

If package pkg is installed then TerraME builds the package. If not, then TerraME checks if there is a local directory with this name storing the package. When -build is executed, TerraME will first check if the directory contains the required information, then it builds the documentation, executes all the tests, and finally creates a zip file with the content of the package. It is possible to have a config file written in Lua to be used by the tests. This file can have a single value: lines. There is also an option in the command to build a package called -clean, to remove unnecessary files in the output, such as the directory log.

If you have comments, doubts or suggestions related to this document, please write a feedback to pedro.andrade <at> inpe.br.

Back to wiki or terrame.org.

Clone this wiki locally