Skip to content

Commit

Permalink
More updates to generic recognition guide
Browse files Browse the repository at this point in the history
  • Loading branch information
atruskie committed Feb 10, 2021
1 parent 2750ee7 commit abc1bf8
Show file tree
Hide file tree
Showing 19 changed files with 314 additions and 177 deletions.
40 changes: 20 additions & 20 deletions docs/basics/cli.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
---
title: Running commands
uid: cli
---

Expand Down Expand Up @@ -69,11 +70,11 @@ an option.
## Options

The command line options (all prefixed with a hyphen (`-`)), have a short and
long form. The short form is always shown to the left of the long form and using
long form. The short form (`-x`) is always shown to the left of the long form (`--long-option`) and using
either is equivalent. The value for the option should follow the option name,
separated by a space. Use double quotes to group values together.

We use long form options in this manual for clarity. See output from the `help`
We use long form options in this manual for clarity. See output from the [`help`](xref:help)
command for all options.

## Global options
Expand All @@ -88,25 +89,24 @@ to the command line:

Valid verbosity values are:

- `None` = 0 - show nothing
- `Error` = 1 - show only errors
- `Warn` = 2 - show only warnings
- `Info` = 3 - the standard level
- `Debug` = 4 - print some debug statements that show variable state and extra
information
- `Trace` = 5 - print many more debugging statements with detailed variable
values
- `Verbose` = 6 - print all stdout and stderr from associated tools
- `All` = 7 - print absolutely everything
- `None` = 0 - show nothing
- `Error` = 1 - show only errors
- `Warn` = 2 - show only warnings
- `Info` = 3 - the standard level
- `Debug` = 4 - print some debug statements that show variable state and extra
information
- `Trace` = 5 - print many more debugging statements with detailed variable
values
- `Verbose` = 6 - print all stdout and stderr from associated tools
- `All` = 7 - print absolutely everything

For example: `-l 4`, will give you the `debug` level of verbosity.

Alternatively, you can append one of the following switches to the command line:

- `-v` Sets the logging to verbose. Equivalent to LogLevel = `Debug` = 4
- `-vv` Sets the logging to very verbose. Equivalent to LogLevel = `Trace` = 5
- `-vvv` Sets the logging to extremely verbose. Equivalent to LogLevel = `All`
= 7
- `-v` Sets the logging to verbose. Equivalent to LogLevel = `Debug` = 4
- `-vv` Sets the logging to very verbose. Equivalent to LogLevel = `Trace` = 5
- `-vvv` Sets the logging to extremely verbose. Equivalent to LogLevel = `All` = 7

### Environment variables

Expand All @@ -117,14 +117,14 @@ Alternatively, you can append one of the following switches to the command line:

## Beware these Syntax Gotchas**

- **Never** finish a double quoted string argument with a backslash (\\). In
- **Never** finish a double quoted string argument with a backslash (\\). In
particular, do *not* end directory names like this:
“C:\\\\Path\\OutputDirectory\\”. The parsing rules for such cases are
complicated and outside of our control. See
[here](https://msdn.microsoft.com/en-us/library/system.environment.getcommandlineargs.aspx)
for details.
- You can test arguments on Windows with the `echoargs.EXE` program
- The arguments used are one of the first lines logged in _AP.exe_ log file
- If an input argument is an array (e.g. directoryinfo\[\]), any commas in the
- You can test arguments on Windows with the `echoargs.EXE` program
- The arguments used are one of the first lines logged in _AP.exe_ log file
- If an input argument is an array (e.g. directoryinfo\[\]), any commas in the
argument will delimit the values. For example, "Y:\\Results\\abc, 123,
doo-dah-dee" will be parsed as "Y:\\Results\\abc", " 123", " doo-dah-dee".
5 changes: 5 additions & 0 deletions docs/basics/config_file_name.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
```
<author>.<analysis>[.<tag>]*.yml
```

The `author` and `analysis` name sections are mandatory. The `tag` section is optional, ignored by _AP_, and can be repeated.
74 changes: 64 additions & 10 deletions docs/basics/config_files.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,10 @@ start with a hash symbol `#`
- Typically, you will only need to adjust a subset of the available parameters
- Most parameters have default values

You can find an introduction to YAML here:
<https://support.ehelp.edu.au/support/solutions/articles/6000055385-introduction-to-yaml>
You can find a introductions to YAML here:

- <https://support.ehelp.edu.au/support/solutions/articles/6000055385-introduction-to-yaml>
- <https://sweetohm.net/article/introduction-yaml.en.html>

You can validate YAML files (to check for syntax errors) here:
<http://yaml-online-parser.appspot.com/>
Expand Down Expand Up @@ -53,11 +55,7 @@ the naming format of the config files is now important. We use the name to deter
For any config file used by `audio2csv`/`AnalyzeLongRecording`
the name of the config file must follow this format:

```ebnf
<author>.<analysis>[.<tag>]*.yml
```

The `author` and `analysis` name sections are mandatory. The `tag` section is optional, ignored by _AP_, and can be repeated.
[!include[config_naming](./config_file_name.md)]

Here are some valid examples:

Expand Down Expand Up @@ -92,6 +90,8 @@ needs to be changed:
+ResampleRate: 16000
```

Note that the parameter name `ResampleRate` is followed by a colon, a space and then a value.

Most of our config files contain comments next to the parameters that explain what a parameter does. A comment is any
line that begins with an hash symbol (`#`). You can see the text that is a comment is coloured differently from the
parameter in the example below:
Expand All @@ -110,7 +110,7 @@ The most variable part of a config file is the `Profiles` section. Profiles
allow us to add extra sections to an analysis. This can be useful for dealing with:

- Geographical variation in calls.
Often a species call will vary between regions. The same detector can work for the different variants of a call but
Often a species' call will vary between regions. The same detector can work for the different variants of a call but
slightly different parameters are needed. In this case we add a profile for each regional variation of the call that
have slightly different parameters or thresholds.
- Generic recognition efforts.
Expand All @@ -130,7 +130,8 @@ Profiles:
SomeParameter: 123
AnotherParameter: "hello"
# more than one profile can be added
# We use the `!type` notation to tell AP what type of parameters we're giving it
# We use the `!type` notation to tell AP
# what type of parameters we're giving it
KoalaExhale: !OscillationParameters
ComponentName: Oscillation
SpeciesName: PhascolarctosCinereus
Expand All @@ -147,7 +148,8 @@ Profiles:
MinOcilFreq: 20
MaxOcilFreq: 55
EventThreshold: 0.2
# And another profile using the blob type (!BlobParameters) parameters
# And another profile using the blob type
# (!BlobParameters) parameters
KoalaInhale: !BlobParameters
ComponentName: Inhale
MinHertz: 800
Expand All @@ -162,3 +164,55 @@ Profiles can get complicated. Each configuration file should detail the differen
please let us know!
For more information on constructing generic recognizers see <xref:guides-generic-recognizers>.
## Indentation
Whenever a line is indented (add trailing spaces) in a YAML configuration file, it must be indented to a consistent level.
If it is inconsistent then _AP_ will not be able to read the config file properly.
Here are some examples. Note spaces are represented with a _middle dot_ (`·`).

### Good

```yml
Profiles:
····BoobookSyllable1: !ForwardTrackParameters
········MinHertz: 400
········MaxHertz: 1100
········MinDuration: 0.1
········MaxDuration: 0.499
```

Note that each indentation step uses four (4) spaces, and each line in the same level has the same indentation.

### Bad: inconsistent indentation

```yml
Profiles:
····BoobookSyllable1: !ForwardTrackParameters
······MinHertz: 400
········MaxHertz: 1100
········MinDuration: 0.1
········MaxDuration: 0.499
```

Note that the `MinHertz` entry is indented with six (6) spaces instead of eight (8). This means _AP_ will try to read
`MaxHertz` as a child (a sub-item) of `MinHertz` - which is incorrect because they are siblings belonging to the same item.


### Bad: mixing tabs and spaces

Note in this example the `⇥` symbol represents pressing the <kbd>Tab ↹</kbd> key.

```yml
Profiles:
····BoobookSyllable1: !ForwardTrackParameters
⇥⇥⇥⇥⇥MinHertz: 400
········MaxHertz: 1100
········MinDuration: 0.1
········MaxDuration: 0.499
```

Note that the `MinHertz` entry is indented with five (5) tabs instead of eight (8) spaces. Again, this means that
`MinHertz` and `MaxHertz` are not part of the same group (they do not have the same indentation), even though it looks like
they are aligned.
33 changes: 8 additions & 25 deletions docs/docfx.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@
"images/README.md",
"README.md",
"pdf/**",
"_ignore"
"_ignore",
"technical/apidoc/*"
]
}
],
Expand All @@ -72,17 +73,7 @@
]
}
],
"overwrite": [
{
"files": [
"technical/apidoc/**.md"
],
"exclude": [
"obj/**",
"_site/**"
]
}
],
"overwrite": "technical/apidoc/*",
"xrefService": [
"https://xref.docs.microsoft.com/query?uid={uid}"
],
Expand Down Expand Up @@ -140,7 +131,8 @@
"**/bin/**",
"**/obj/**",
"**/toc.yml",
"**/toc.md"
"**/toc.md",
"technical/apidoc/*"
]
},
{
Expand All @@ -162,7 +154,8 @@
],
"exclude": [
"images/README.md",
"README.md"
"README.md",
"technical/apidoc/*"
]
}

Expand All @@ -179,17 +172,7 @@
]
}
],
"overwrite": [
{
"files": [
"technical/apidoc/**.md"
],
"exclude": [
"**/bin/**",
"**/obj/**"
]
}
],
"overwrite": "technical/apidoc/*",
"wkhtmltopdf": {
"additionalArguments": "--enable-local-file-access --footer-center [page] --footer-font-size 10 --no-stop-slow-scripts"
},
Expand Down
26 changes: 13 additions & 13 deletions docs/guides/Ecosounds.NinoxBoobook.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---

# This is a non-functional example used for docs. Please see the real config file for the Boobook recognizer.
# Boobook Owl = Towsey.NinoxBoobook
# Resample rate must be 2 X the desired Nyquist
#ResampleRate: 22050
# SegmentDuration: units=seconds;
ResampleRate: 22050
# SegmentDuration: units=seconds;
SegmentDuration: 60
# SegmentOverlap: units=seconds;
SegmentOverlap: 0
Expand All @@ -12,14 +12,14 @@ SegmentOverlap: 0
# This profile is required for the species-specific recogniser and must have the current name.
Profiles:
BoobookSyllable: !ForwardTrackParameters
ComponentName: RidgeTrack
ComponentName: RidgeTrack
SpeciesName: NinoxBoobook
FrameSize: 1024
FrameStep: 256
WindowFunction: HANNING

# min and max of the freq band to search
MinHertz: 400
MinHertz: 400
MaxHertz: 1100
MinDuration: 0.17
MaxDuration: 1.2
Expand All @@ -28,15 +28,15 @@ Profiles:
- 6.0
- 9.0
- 12.0

#################### POST-PROCESSING of EVENTS ###################

PostProcessing:
# The following generic post-processing steps are determined by config settings.
# Step 1: Combine overlapping events - events derived from all profiles.
# Step 2: Combine possible syllable sequences and filter on excess syllable count.
# Step 3: Remove events whose bandwidth is too small or large.
# Step 4: Remove events that have excessive noise in their side-bands.
# The following generic post-processing steps are determined by config settings.
# Step 1: Combine overlapping events - events derived from all profiles.
# Step 2: Combine possible syllable sequences and filter on excess syllable count.
# Step 3: Remove events whose bandwidth is too small or large.
# Step 4: Remove events that have excessive noise in their side-bands.

# 1: Combine overlapping events
CombineOverlappingEvents: true
Expand All @@ -53,7 +53,7 @@ PostProcessing:
# 3: Remove events whose duration lies outside 3 SDs of an expected value.
#Duration:
ExpectedDuration: 0.14
DurationStandardDeviation: 0.01
DurationStandardDeviation: 0.01

# 4: Remove events whose bandwidth lies outside 3 SDs of an expected value.
# Bandwidth:
Expand Down
Loading

0 comments on commit abc1bf8

Please sign in to comment.