Add benchmark option `-b` for `test-avrdude` #1709

stefanrueger · 2024-02-26T10:26:14Z

test-avrdude -b executes five tests for typical programming tasks:

Write and verify a "difficult" sketch to flash: two code sections and one data section separated by holes of different sizes
Dump all flash, eg, to make a backup
Write and verify a "difficult" eeprom data file with holes
Dump all eeprom, eg, to make a backup
Chip erase (bootloaders are expected to page erase flash apart from the bootloader itself) and spot check whether flash was erased

The reported times are realistic times with overhead of starting avrdude, resetting the board via DTR/RTS, establishing comms, including erasing the flash before writing/verifying the sketch and disengaging the chip.

$ test-avrdude -b -d 0 \
  -p "u7.7/weu-jPrac -c urclock -P ch340 -p m328p -b 1000000" \
  -p "u7.7/-eu-jPrac -c urclock -P ch340 -p m328p -b 1000000"

Testing avrdude version 7.3-20240225 (20788712)
Prepare "u7.7/weu-jPrac -c urclock -P ch340 -p m328p -b 1000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.174 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   1.339 s: flash -U read all flash
✅   1.415 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   1.034 s: eeprom -U read all
✅   1.604 s: chip erase and spot check flash is actually erased
✅   7.566 s: benchmark for u7.7/weu-jPrac -c urclock -P ch340 -p m328p -b 1000000
Prepare "u7.7/-eu-jPrac -c urclock -P ch340 -p m328p -b 1000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.001 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   1.333 s: flash -U read all flash
✅   1.405 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   1.029 s: eeprom -U read all
✅   1.618 s: chip erase and spot check flash is actually erased
✅   7.386 s: benchmark for u7.7/-eu-jPrac -c urclock -P ch340 -p m328p -b 1000000

Note the benchmark line with the cumulative time for all five tasks

This executes five tests for typical programming tasks: - Write/verify a "difficult" sketch to flash: two code sections and one data section separated by "holes" of different sizes - Dump all flash, eg, to make a backup - Write/verify a "difficult" eeprom data file with holes - Dump all eeprom, eg, to make a backup - Chip erase (bootloaders are expected to page erase flash apart from the bootloader itself) and spot check whether flash was erased The reported times are realistic times with overhead of starting avrdude, resetting the board via DTR/RTS, establishing comms, including erasing the flash before writing/verifying the sketch and disengaging the chip. $ test-avrdude -b -d 0 \ -p "u7.7/weu-jPrac -c urclock -P ch340 -p m328p -b 1000000" \ -p "u7.7/-eu-jPrac -c urclock -P ch340 -p m328p -b 1000000" Testing avrdude version 7.3-20240225 (2078871) Prepare "u7.7/weu-jPrac -c urclock -P ch340 -p m328p -b 1000000" and press 'enter' or 'space' to continue. Press any other key to skip ✅ 2.174 s: flash -U write/verify holes_rjmp_loops_32768B.hex ✅ 1.339 s: flash -U read all flash ✅ 1.415 s: eeprom -U write/verify holes_pack_my_box_1024B.hex ✅ 1.034 s: eeprom -U read all ✅ 1.604 s: chip erase and spot check flash is actually erased ✅ 7.566 s: benchmark for u7.7/weu-jPrac -c urclock -P ch340 -p m328p -b 1000000 Prepare "u7.7/-eu-jPrac -c urclock -P ch340 -p m328p -b 1000000" and press 'enter' or 'space' to continue. Press any other key to skip ✅ 2.001 s: flash -U write/verify holes_rjmp_loops_32768B.hex ✅ 1.333 s: flash -U read all flash ✅ 1.405 s: eeprom -U write/verify holes_pack_my_box_1024B.hex ✅ 1.029 s: eeprom -U read all ✅ 1.618 s: chip erase and spot check flash is actually erased ✅ 7.386 s: benchmark for u7.7/-eu-jPrac -c urclock -P ch340 -p m328p -b 1000000 Note the benchmark line with the cumulative time for all five tasks

mcuee · 2024-02-26T14:57:45Z

This PR does not work too well under MSYS2 mingw64 (WIndows)

MINGW64 /c/work/avr/avrdude_test/avrdude_sr/tools
$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t . -v -p "u7.7/-eu-jPrac -c urclock -P COM20 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3
Prepare "u7.7/-eu-jPrac -c urclock -P COM20 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
Cannot detect flash; check that "u7.7/-eu-jPrac -c urclock -P COM20 -p m328p -b 2000000" are valid avrdude options; skipping this test

$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t . -v -p "-c urclock -P COM20 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3
Prepare "-c urclock -P COM20 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.721 s: flash -U write/verify holes_rjmp_loops_32768B.hex
❌   0.626 s: flash -U read all flash (failed command below)
$ ./avrdude.exe -qq -c urclock -P COM20 -p m328p -b 2000000 -Uflash:r:/dev/null
avrdude OS error: file /dev/null is not writeable: No such file or directory
avrdude OS error: cannot open output file /dev/null: No such file or directory
avrdude error: write to file /dev/null failed
✅   1.248 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
❌   0.462 s: eeprom -U read all (failed command below)
$ ./avrdude.exe -qq -c urclock -P COM20 -p m328p -b 2000000 -Ueeprom:r:/dev/null
avrdude OS error: file /dev/null is not writeable: No such file or directory
avrdude OS error: cannot open output file /dev/null: No such file or directory
avrdude error: write to file /dev/null failed
✅   1.474 s: chip erase and spot check flash is actually erased
❌   5.531 s: benchmark for -c urclock -P COM20 -p m328p -b 2000000

One or more AVRDUDE "-c urclock -P COM20 -p m328p -b 2000000" tests failed. Do you want to retry this particular test? (y/n): n

$ ./avrdude -c urclock -p m328p -P COM20 -b 2000000 -xshowall
avrdude: AVR device initialized and ready to accept instructions
0000ffffffff 0000-00-00 00.00  application 0 store 0 meta 1 boot 384 u7.7 -eu-jPrac vector 25 (SPM_Ready) ATmega328P

mcuee · 2024-02-26T15:07:26Z

The above test is using avrdude 7.3 MSVC64 binary.

Same results if I use the github action MSVC64 binary of this PR.

dev/null can not be used.
The first example shows the possibility to mention u7.7/-eu-jPrac but it does not seem to work.

 MINGW64 /c/work/avr/avrdude_test/avrdude_sr/tools
$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t . -v -p "u7.7/-eu-jPrac -c urclock -P COM20 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240226 (8919a22)
Prepare "u7.7/-eu-jPrac -c urclock -P COM20 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
Cannot detect flash; check that "u7.7/-eu-jPrac -c urclock -P COM20 -p m328p -b 2000000" are valid avrdude options; skipping this test

$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t . -v -p "-c urclock -P COM20 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240226 (8919a22)
Prepare "-c urclock -P COM20 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.723 s: flash -U write/verify holes_rjmp_loops_32768B.hex
❌   0.637 s: flash -U read all flash (failed command below)
$ ./avrdude.exe -qq -c urclock -P COM20 -p m328p -b 2000000 -Uflash:r:/dev/null
avrdude OS error: file /dev/null is not writeable: No such file or directory
avrdude OS error: cannot open output file /dev/null: No such file or directory
avrdude error: write to file /dev/null failed
✅   1.252 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
❌   0.450 s: eeprom -U read all (failed command below)
$ ./avrdude.exe -qq -c urclock -P COM20 -p m328p -b 2000000 -Ueeprom:r:/dev/null
avrdude OS error: file /dev/null is not writeable: No such file or directory
avrdude OS error: cannot open output file /dev/null: No such file or directory
avrdude error: write to file /dev/null failed
✅   1.473 s: chip erase and spot check flash is actually erased
❌   5.535 s: benchmark for -c urclock -P COM20 -p m328p -b 2000000

One or more AVRDUDE "-c urclock -P COM20 -p m328p -b 2000000" tests failed. Do you want to retry this particular test? (y/n): n

$ ./avrdude -c urclock -p m328p -P COM20 -b 2000000 -xshowall
avrdude: AVR device initialized and ready to accept instructions
0000ffffffff 0000-00-00 00.00  application 0 store 0 meta 1 boot 384 u7.7 -eu-jPrac vector 25 (SPM_Ready) ATmega328P

mcuee · 2024-02-26T15:20:17Z

Hmm, strange, the github action MSVC64 binary (go to https://github.com/avrdudes/avrdude/actions/runs/8047336152, then click avrdude-msvc-x64) does not seem to have the right commit ID. Local Windows MSVC64 and MSYS2 mingw64 build do not have this issue.

I download the other MSYS2 mingw64 binary and it has the same issue.

Testing ./avrdude.exe version 7.3-20240226 (8919a22)

Edit -- this is normal.

Run actions/checkout@v3
Syncing repository: avrdudes/avrdude
Getting Git version info
Temporarily overriding HOME='D:\a\_temp\33e54ae3-40b4-4b11-a5a0-8a6c5c8e624e' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
"C:\Program Files\Git\bin\git.exe" config --global --add safe.directory D:\a\avrdude\avrdude
Deleting the contents of 'D:\a\avrdude\avrdude'
Initializing the repository
Disabling automatic garbage collection
Setting up auth
Fetching the repository
Determining the checkout info
Checking out the ref
"C:\Program Files\Git\bin\git.exe" log -1 --format='%H'
'8919a22feb5b2e255de0235a2245f87c208d346f'

mcuee · 2024-02-26T23:44:17Z

@stefanrueger

This PR works very well under Linux. The following tests are carried out using an Arduino Uno clone with ATmega16U2 USB to Serial chip (same as the official Arduino Uno Rev 3 design), it can go to 2M baud.

From the following test results, we can see the speed gain from 115200bps to 500kbps is significant. Then the jump from 500kbps to 1Mbps is less significant. The jump from 1Mbps to 2Mbps is even less important.

mcuee@UbuntuSwift3 ~/build/avr/avrdude_sr/tools (benchmark)$ avrdude -c urclock -P /dev/ttyACM0 -p m328p -xshowall
avrdude: AVR device initialized and ready to accept instructions
ffffffffffff 0000-00-00 00.00  application 0 store 0 meta 1 boot 384 u7.7 -eu-jPrac vector 25 (SPM_Ready) ATmega328P

mcuee@UbuntuSwift3 ~/build/avr/avrdude_sr/tools (benchmark)$ ./test-avrdude -b -d 0 -v -p "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 115200"
Testing avrdude version 7.3-20240225 (20788712)
Prepare "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 115200" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.750 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   3.464 s: flash -U read all flash
✅   1.331 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.801 s: eeprom -U read all
✅   1.962 s: chip erase and spot check flash is actually erased
✅  10.308 s: benchmark for u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 115200

mcuee@UbuntuSwift3 ~/build/avr/avrdude_sr/tools (benchmark)$ ./test-avrdude -b -d 0 -v -p "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 500000"
Testing avrdude version 7.3-20240225 (20788712)
Prepare "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 500000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.920 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   1.141 s: flash -U read all flash
✅   1.259 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.506 s: eeprom -U read all
✅   1.572 s: chip erase and spot check flash is actually erased
✅   6.398 s: benchmark for u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 500000

mcuee@UbuntuSwift3 ~/build/avr/avrdude_sr/tools (benchmark)$ ./test-avrdude -b -d 0 -v -p "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 1000000"
Testing avrdude version 7.3-20240225 (20788712)
Prepare "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 1000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.790 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.780 s: flash -U read all flash
✅   1.253 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.463 s: eeprom -U read all
✅   1.498 s: chip erase and spot check flash is actually erased
✅   5.784 s: benchmark for u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 1000000

mcuee@UbuntuSwift3 ~/build/avr/avrdude_sr/tools (benchmark)$ ./test-avrdude -b -d 0 -v -p "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 2000000"
Testing avrdude version 7.3-20240225 (20788712)
Prepare "u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.721 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.609 s: flash -U read all flash
✅   1.242 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.448 s: eeprom -U read all
✅   1.469 s: chip erase and spot check flash is actually erased
✅   5.489 s: benchmark for u7.7/-eu-jPrac -c urclock -P /dev/ttyACM0 -p m328p -b 2000000

mcuee · 2024-02-26T23:45:57Z

The reported times are realistic times with overhead of starting avrdude, resetting the board via DTR/RTS, establishing comms, including erasing the flash before writing/verifying the sketch and disengaging the chip.

@stefanrueger

I agree. I will use this method once this PR is merged. Thanks for providing the nice improvements for avrdude.

Uppercase names are typically used for environment variables in bash

mcuee · 2024-02-27T01:00:02Z

dev/null can not be used.

A quick and dirty fix.

MINGW64 /c/work/avr/avrdude_test/avrdude_sr/tools
$ git diff
diff --git a/tools/test-avrdude b/tools/test-avrdude
index 7ecabe8c..836d2728 100755
--- a/tools/test-avrdude
+++ b/tools/test-avrdude
@@ -375,7 +375,7 @@ for (( p=0; p<$arraylength; p++ )); do

     if [[ $benchmark -eq 1 ]]; then
       specify="flash -U read all flash"
-      command=(${avrdude[@]} -Uflash:r:/dev/null)
+      command=(${avrdude[@]} -Uflash:r:./tmp.hex:i)
       execute "${command[@]}"
       result [ $? == 0 ]
       bench_t_flrd=$bench_t
@@ -406,7 +406,7 @@ for (( p=0; p<$arraylength; p++ )); do
         bench_t_eewr=$bench_t

         specify="eeprom -U read all"
-        command=(${avrdude[@]} -Ueeprom:r:/dev/null)
+        command=(${avrdude[@]} -Ueeprom:r:./tmp.hex:i)
         execute "${command[@]}"
         result [ $? == 0 ]
         bench_t_eerd=$bench_t

$ ./avrdude.exe -c urclock -P COM20 -p m328p -xshowall
avrdude: AVR device initialized and ready to accept instructions
0000ffffffff 0000-00-00 00.00  application 0 store 0 meta 1 boot 384 
u7.7 weu-jPrac vector 25 (SPM_Ready) ATmega328P

$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t . -v -p "-c urclock -P COM20 -p m328p -b 1000000"
Testing ./avrdude.exe version 7.3-20240226 (8919a22)
Prepare "-c urclock -P COM20 -p m328p -b 1000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.968 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.801 s: flash -U read all flash
✅   1.243 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.464 s: eeprom -U read all
✅   1.528 s: chip erase and spot check flash is actually erased
✅   6.004 s: benchmark for -c urclock -P COM20 -p m328p -b 1000000

$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t . -v -p "-c urclock -P COM20 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240226 (8919a22)
Prepare "-c urclock -P COM20 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.910 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.636 s: flash -U read all flash
✅   1.235 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.454 s: eeprom -U read all
✅   1.471 s: chip erase and spot check flash is actually erased
✅   5.706 s: benchmark for -c urclock -P COM20 -p m328p -b 2000000

@stefanrueger

I think one way to fix is to use another command line option to specify the scratch file name to be used, at least for Windows, similar to -t option to specify the temp directory.

The cumulative time for typical user tasks depends on the flash and EEPROM size of the part. This commit computes a size-normalised time. Lower is better. This avrbench number still depends on the part, but less so than the cumulative time of the tasks. The summary line is put in vertical bars to make creation of markdown tables easier.

stefanrueger · 2024-02-27T01:23:40Z

/dev/null can not be used.

Should be fixed now

The first example shows the possibility to mention u7.7/-eu-jPrac but it does not seem to work.

I have no idea why this is so in MingW. This is probably a pre-existing problem?

I will use this method once this PR is merged.

Neat! The latest commit writes the summary line in markdown, so you can copy that line into a larger table.

mcuee · 2024-02-27T01:51:10Z

/dev/null can not be used.

Should be fixed now

Not yet. /dev/null is present but just can not be used in this case (as temporary reading file), similar to /dev/shm issue previously.

$ ls -la /dev/null
crw-rw-rw- 1 XFCHEN WorkGroup 1, 3 Feb 27 09:47 /dev/null

$ ls -la /dev/shm/
total 15
drwxr-xr-x 1 XFCHEN WorkGroup)   0 Feb  1 11:19 .
drwxr-xr-x 1 XFCHEN WorkGroup   0 Mar 22  2021 ..
-rw-r--r-- 1 XFCHEN WorkGroup  12 Apr  8  2021 fish_shmem_1189162
-rw-r--r-- 1 XFCHEN WorkGroup 382 Jan 30 15:33 test-avrdude.log.pMXXoV
-rw-r--r-- 1 XFCHEN WorkGroup 382 Jan 30 15:47 test-avrdude.log.Rv8F1B
-rw-r--r-- 1 XFCHEN WorkGroup 846 Jan 30 15:11 test-avrdude_mod.log.9Ays8m
-rw-r--r-- 1 XFCHEN WorkGroup 846 Jan 30 15:13 test-avrdude_mod.log.x0wLUy

$ ./test-avrdude -b -e ./avrdude.exe -d 0 -t . -v -p "-c urclock -P COM5 -p m328p -b 1000000"
Testing ./avrdude.exe version 7.3-20240227 (cfc26fd)
Prepare "-c urclock -P COM5 -p m328p -b 1000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.055 s: flash -U write/verify holes_rjmp_loops_32768B.hex
❌   0.884 s: flash -U read all flash (failed command below)
$ ./avrdude.exe -qq -c urclock -P COM5 -p m328p -b 1000000 -Uflash:r:/dev/null
avrdude OS error: file /dev/null is not writeable: No such file or directory
avrdude OS error: cannot open output file /dev/null: No such file or directory
avrdude error: write to file /dev/null failed
✅   1.335 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
❌   0.550 s: eeprom -U read all (failed command below)
$ ./avrdude.exe -qq -c urclock -P COM5 -p m328p -b 1000000 -Ueeprom:r:/dev/null
avrdude OS error: file /dev/null is not writeable: No such file or directory
avrdude OS error: cannot open output file /dev/null: No such file or directory
avrdude error: write to file /dev/null failed
✅   1.599 s: chip erase and spot check flash is actually erased
|❌| 6.423 s|  0.999 s|-c urclock -P COM5 -p m328p -b 1000000|

One or more AVRDUDE "-c urclock -P COM5 -p m328p -b 1000000" tests failed. Do you want to retry this particular test? (y/n): n

The first example shows the possibility to mention u7.7/-eu-jPrac but it does not seem to work.

I have no idea why this is so in MingW. This is probably a pre-existing problem?

Hmm, then this must be the case. I actually do not know this feature and have never used until today.

stefanrueger · 2024-02-27T02:01:27Z

/dev/null is present but just can not be used in this case

I suspect bash emulates /dev/null but as soon as you have a program that tries to r/w there then the OS fails...

stefanrueger · 2024-02-27T02:05:10Z

I actually do not know this feature and have never used until today

This "feature" relies on the observation that AVRDUDE simply ignores non-option arguments:

$ avrdude -t -qq AVRDUDE rocks -c dryrun -p m328p

It could be that the MingW getopts() function stops option parsing with the first non-option argument?

mcuee · 2024-02-27T02:41:07Z

I actually do not know this feature and have never used until today

This "feature" relies on the observation that AVRDUDE simply ignores non-option arguments:

I see.

$ avrdude -t -qq AVRDUDE rocks -c dryrun -p m328p
It could be that the MingW getopts() function stops option parsing with the first non-option argument?

I tested both MSVC64 and mingw64 binaries from github action and both have the same results.

$ ./avrdude -t -qq AVRDUDE rocks -c dryrun -p m328p
avrdude error: no programmer has been specified on the command line or in the
        config file(s); specify one using the -c option and try again

$ ./avrdude_mingw64 -t -qq AVRDUDE rocks -c dryrun -p m328p
avrdude_mingw64 error: no programmer has been specified on the command line or in the
                config file(s); specify one using the -c option and try again

For MSVC, there is no getopt so avrdude is using a replacement here, adapted from mingw64 implementation.
https://github.com/avrdudes/avrdude/blob/main/src/msvc/getopt.c
https://github.com/avrdudes/avrdude/blob/main/src/msvc/getopt.h

stefanrueger · 2024-02-27T03:36:48Z

I don't think it is worth our while to improve MingW parsing, but the /dev/null problem should be solved now.

mcuee · 2024-02-27T03:42:05Z

I don't think it is worth our while to improve MingW parsing, but the /dev/null problem should be solved now.

I agree. mingw/msvc option parsing does not need to be changed. I think very few people know the trick in reality.

And yes /dev/null issue has been sorted out. I think this PR should be good to go now.

MINGW64 /c/work/avr/avrdude_test/avrdude_sr/tools
$ ./test-avrdude -b -e ./avrdude.exe -d 0 -t . -v -p "-c urclock -P COM5 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240227 (cdecaee)
Prepare "-c urclock -P COM5 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.001 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.706 s: flash -U read all flash
✅   1.331 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.532 s: eeprom -U read all
✅   1.565 s: chip erase and spot check flash is actually erased
|✅| 6.135 s|  0.947 s|-c urclock -P COM5 -p m328p -b 2000000|

mcuee · 2024-02-27T03:46:53Z

@stefanrueger

BTW, now we have the option -s to skip EEPROM testing for bootloaders. Can we have another option to skip the chip erase testing? Or maybe just to skip chip erase testing as well when the user specify -s.

Can we extend similar option to programmers as well?

$ ./test-avrdude -h
Syntax: test-avrdude {<opts>}
Function: test AVRDUDE for certain programmer and part combinations
Options:
    -b                          benchmark only summarising overall and avrbench time
    -c <configuration spec>     additional configuration options used for all runs
    -d <sec>                    delay between test commands (default 0.75 seconds)
    -e <avrdude path>           set path of AVRDUDE executable (default avrdude)
    -l                          list test commands but do not execute them
    -p <programmer/part specs>  can be used multiple times, overrides default tests
    -s                          skip EEPROM tests for bootloaders
    -t <dir>                    temporary directory (default /dev/shm)
    -T                          Add dryrun/dryboot test cases to test test-avrdude
    -v                          verbose: show AVRDUDE error and warning messages
    -? or -h                    show this help text
Note: some Windows environments require the option -t . or similar
Examples:
    $ test-avrdude -s -p "-c dryboot -p t13" -p "-c dryrun -p m4809"
    $ test-avrdude -d 2 -p "-cjtag2updi -patmega4809 -Pusb:2341:0058 -r"

stefanrueger · 2024-02-27T03:59:41Z

Can we have another option to skip the chip erase testing?

We can drop that test altogether.

The new avrbench figure next to the cumulative time

Ignores the chip erase time anyway (as it's already subsumed in writing & verifying a sketch)
Is a weighted sum of flash and eeprom times (more weight for flash)
Normalises for different flash and eeprom sizes
Attempts the "right thing" when there is no eeprom

mcuee · 2024-02-27T04:30:46Z

Can we have another option to skip the chip erase testing?

We can drop that test altogether.

Good idea to drop to the test for -b.

The new avrbench figure next to the cumulative time

Ignores the chip erase time anyway (as it's already subsumed in writing & verifying a sketch)

Is a weighted sum of flash and eeprom times (more weight for flash)

Normalises for different flash and eeprom sizes

Attempts the "right thing" when there is no eeprom

I see. Sounds good to me.

MINGW64 /c/work/avr/avrdude_test/avrdude_sr/tools
$ ./test-avrdude -b -e ./avrdude.exe -d 0 -s -t . -v -p "-c urclock -P COM5 -p m328p"
Testing ./avrdude.exe version 7.3-20240227 (cdecaee)
Prepare "-c urclock -P COM5 -p m328p" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.855 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   3.573 s: flash -U read all flash
✅   2.073 s: chip erase and spot check flash is actually erased
|✅| 8.501 s|  2.100 s|-c urclock -P COM5 -p m328p|

$ ./test-avrdude -b -e ./avrdude.exe -d 0 -t . -v -p "-c urclock -P COM5 -p m328p"
Testing ./avrdude.exe version 7.3-20240227 (cdecaee)
Prepare "-c urclock -P COM5 -p m328p" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.857 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   3.572 s: flash -U read all flash
✅   1.439 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.885 s: eeprom -U read all
✅   2.075 s: chip erase and spot check flash is actually erased
|✅|10.828 s|  1.813 s|-c urclock -P COM5 -p m328p|

$ ./test-avrdude -b -e ./avrdude.exe -d 0 -s -t . -v -p "-c urclock -P COM5 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240227 (cdecaee)
Prepare "-c urclock -P COM5 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.820 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.708 s: flash -U read all flash
✅   1.567 s: chip erase and spot check flash is actually erased
|✅| 4.095 s|  0.826 s|-c urclock -P COM5 -p m328p -b 2000000|

$ ./test-avrdude -b -e ./avrdude.exe -d 0 -t . -v -p "-c urclock -P COM5 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240227 (cdecaee)
Prepare "-c urclock -P COM5 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.824 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.705 s: flash -U read all flash
✅   1.325 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.540 s: eeprom -U read all
✅   1.556 s: chip erase and spot check flash is actually erased
|✅| 5.950 s|  0.910 s|-c urclock -P COM5 -p m328p -b 2000000|

mcuee · 2024-02-27T13:21:57Z

Test under Windows MSYS2 mingw64 build environment.

MINGW64 /c/work/avr/avrdude_test/avrdude_sr/tools
$ ./test-avrdude -b -e ./avrdude.exe -d 0 -t . -v -p "-c urclock -P COM20 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240227 (61ed3dd1)
Prepare "-c urclock -P COM20 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.739 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.622 s: flash -U read all flash
✅   1.254 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.450 s: eeprom -U read all
|✅| 4.065 s|avrbench  0.842 s|-c urclock -P COM20 -p m328p -b 2000000|

$ ./test-avrdude -b -e ./avrdude.exe -d 0 -s -t . -v -p "-c urclock -P COM20 -p m328p -b 2000000"
Testing ./avrdude.exe version 7.3-20240227 (61ed3dd1)
Prepare "-c urclock -P COM20 -p m328p -b 2000000" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.729 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   0.621 s: flash -U read all flash
|✅| 2.350 s|avrbench  0.768 s|-c urclock -P COM20 -p m328p -b 2000000|

$ ./test-avrdude -b -e ./avrdude.exe -d 0 -t . -v -p "-c usbasp -p m168p -B 0.25"
Testing ./avrdude.exe version 7.3-20240227 (61ed3dd1)
Prepare "-c usbasp -p m168p -B 0.25" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.456 s: flash -U write/verify holes_rjmp_loops_16384B.hex
✅   2.815 s: flash -U read all flash
✅   1.631 s: eeprom -U write/verify holes_pack_my_box_512B.hex
✅   0.426 s: eeprom -U read all
|✅| 6.328 s|avrbench  2.622 s|-c usbasp -p m168p -B 0.25|

$ ./test-avrdude -b -e ./avrdude_mingw64.exe -d 0 -t . -v -p "-c usbasp -p m168p -B 0.25"
Testing ./avrdude_mingw64.exe version 7.3-20240227 (61ed3dd1)
Prepare "-c usbasp -p m168p -B 0.25" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.517 s: flash -U write/verify holes_rjmp_loops_16384B.hex
✅   2.936 s: flash -U read all flash
✅   1.726 s: eeprom -U write/verify holes_pack_my_box_512B.hex
✅   0.516 s: eeprom -U read all
|✅| 6.695 s|avrbench  2.774 s|-c usbasp -p m168p -B 0.25|

$ ./test-avrdude -b -e ./avrdude.exe -d 0 -t . -v -p "-c usbasp -p m328p -B 0.25"
Testing ./avrdude.exe version 7.3-20240227 (61ed3dd1)
Prepare "-c usbasp -p m328p -B 0.25" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.505 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   5.407 s: flash -U read all flash
✅   3.033 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.682 s: eeprom -U read all
|✅|11.627 s|avrbench  2.409 s|-c usbasp -p m328p -B 0.25|

$ ./test-avrdude -b -e ./avrdude_mingw64.exe -d 0 -t . -v -p "-c usbasp -p m328p -B 0.25"
Testing ./avrdude_mingw64.exe version 7.3-20240227 (61ed3dd1)
Prepare "-c usbasp -p m328p -B 0.25" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.513 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   5.597 s: flash -U read all flash
✅   3.155 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   0.800 s: eeprom -U read all
|✅|12.065 s|avrbench  2.500 s|-c usbasp -p m328p -B 0.25|

mcuee · 2024-02-27T13:25:06Z

@stefanrueger

I was thinking about avrbench. I see two options.

keep the current way: weighted sum of flash and eeprom times (more weight for flash)
split into two number -- avrbench_flash and avrbench_eeprom.

I actually think maybe we should go for Option 2 because some people may not care about EEPROM at all.

How do you like this idea?

stefanrueger · 2024-02-27T23:03:07Z

split into two number -- avrbench_flash and avrbench_eeprom

@mcuee Good shout. Let me think about this a bit.

Invariably, whenever you have a benchmark, people try to sort everything according to a single number. This invariably is some sort of weighted average. We could try to create a table with five numbers and let the user select what they like and sort accordingly (the column caption affords two little triangles to sort up or down according to that triangle).

So, we could have

Five columns: flash wr, flash rd, eeprom wr, eeprom rd, avrbench
Four columns: flash wr, flash rd, eeprom wr, eeprom rd
Two columns: flash avrbench, eeprom avrbench
One column: avrbench

Normalisation is always an issue; we have the choice of normalising by flash and eeprom size or we could decide to create tables only with 32k parts and show raw times.

Thinking about it I have the ever so slight preference for ditching avrbench, go for four columns and do no normalisation (that's the most transparent and does not pass judgement on what people may or may not be interested in). It would be fabulous if we could create a sortable table in the wiki to give users the power to sort themselves.

I could create markdown tables incl header automatically (in the test-avrdude order), but it would need the skills of someone like @mcuee to make them sortable

mcuee · 2024-02-27T23:40:30Z

Thinking about it I have the ever so slight preference for ditching avrbench, go for four columns and do no normalisation (that's the most transparent and does not pass judgement on what people may or may not be interested in). It would be fabulous if we could create a sortable table in the wiki to give users the power to sort themselves.

@stefanrueger
Yes, I agree this is the way to go.

mcuee · 2024-02-27T23:55:35Z

I could create markdown tables incl header automatically (in the test-avrdude order), but it would need the skills of someone like @mcuee to make them sortable

I did a Google search and it seems github does not support sortable markdown table.

There are Chrome Extensions which can do that. I just tried the following and it seems to work wekk well. The extension is not pouplar though.
https://chromewebstore.google.com/detail/ekjbdpnpokbkkebdgjgoadgpkegefkab

On the other hand, it is not difficult for the users to import the table to their favorite spreadsheet program (eg: Excel or LibreOffice Calc) and sort the table. There are also shell scripts which can do that.

Shell script example:
https://gist.github.com/domargan/5141965cd19e37047bfd34ddc4c4ab53

stefanrueger · 2024-02-28T02:03:53Z

OK, here is a mockup of how this could look like

test-avrdude -b -d 0 \
  -p "-c urclock -P ch340 -p m328p -b 1000000 u7.7/weu-jPrac" \
  -p "-c urclock -P ch340 -p m328p -b 1000000 u7.7/-eu-jPrac" | \
  tee /tmp/bb

Testing avrdude version 7.3-20240225 (20788712)

The benchmark shows the wall clock time of avrdude carrying out the following tasks
  - Program Sketch: erase flash, then write to flash and verify a sketch in three sections separated by holes
  - Backup Flash: read the full flash memory and write to file
  - Program Data: write to EEPROM and verify a file in two data sections separated by a hole
  - Backup EEPROM: read the full EEPROM memory and write to file

The sketch and data payload is roughly one sixth of the respective memory size

| | Programmer | Part | Program Sketch | Backup Flash | Program Data | Backup EEPROM | Comments |
|:-:|--:|--:|--:|--:|--:|--:|:--|
Prepare "-c urclock -P ch340 -p m328p -b 1000000 u7.7/weu-jPrac" and press 'enter' or 'space' to continue. Press any other key to skip
✅   2.163 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   1.320 s: flash -U read all flash
✅   1.393 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   1.003 s: eeprom -U read all
|✅|urclock|m328p| 2.163 s| 1.320 s| 1.393 s| 1.003 s| -b 1000000 u7.7/weu-jPrac|
Prepare "-c urclock -P ch340 -p m328p -b 1000000 u7.7/-eu-jPrac" and press 'enter' or 'space' to continue. Press any other key to skip
✅   1.983 s: flash -U write/verify holes_rjmp_loops_32768B.hex
✅   1.344 s: flash -U read all flash
✅   1.422 s: eeprom -U write/verify holes_pack_my_box_1024B.hex
✅   1.012 s: eeprom -U read all
|✅|urclock|m328p| 1.983 s| 1.344 s| 1.422 s| 1.012 s| -b 1000000 u7.7/-eu-jPrac|

The output of -b can then be filtered by the following command

$ grep -v ^'[^T |]'  /tmp/bb

to obtain the table including legend:

Testing avrdude version 7.3-20240225 (2078871)

The benchmark shows the wall clock time of avrdude carrying out the following tasks

Program Sketch: erase flash, then write to flash and verify a sketch in three sections separated by holes
Backup Flash: read the full flash memory and write to file
Program Data: write to EEPROM and verify a file in two data sections separated by a hole
Backup EEPROM: read the full EEPROM memory and write to file

The sketch and data payload is roughly one sixth of the respective memory size

	Programmer	Part	Program Sketch	Backup Flash	Program Data	Backup EEPROM	Comments
✅	urclock	m328p	2.163 s	1.320 s	1.393 s	1.003 s	-b 1000000 u7.7/weu-jPrac
✅	urclock	m328p	1.983 s	1.344 s	1.422 s	1.012 s	-b 1000000 u7.7/-eu-jPrac

mcuee · 2024-02-28T05:28:16Z

@stefanrueger

Looks very nice. Thanks.

stefanrueger · 2024-02-28T09:45:33Z

OK, @mcuee have a look.

yes xxx | test-avrdude -Tsb | grep -v ^'[^T |]' outputs the following table sraight away:

Testing avrdude version 7.3-20240225 (2078871)

The benchmark shows the wall clock time of avrdude carrying out the following tasks

Fl-ewv: erase flash, then write a sketch with three sections separated by holes to flash and verify
Fl-r: read the full flash memory and write to file
EE-wv: write data with two data sections separated by a hole to EEPROM and verify
EE-r: read the full EEPROM memory and write to file

The sketch and data payload is roughly one sixth of the respective memory size

	`-c pgm`	`-p part`	Fl-ewv	Fl-r	EE-wv	EE-r
✅	dryrun	at90can32	0.07 s	0.06 s	0.07 s	0.07 s
✅	dryboot	at90s4414	0.06 s	0.06 s	--	--
✅	dryboot	atmega103	0.07 s	0.07 s	--	--
✅	dryboot	atmega163	0.07 s	0.07 s	--	--
✅	dryboot	atmega16hva	0.07 s	0.07 s	--	--
✅	dryboot	atmega256rfr2	0.07 s	0.07 s	--	--
✅	dryboot	atmega4808	0.07 s	0.07 s	--	--
✅	dryboot	atmega64	0.07 s	0.07 s	--	--
✅	dryboot	atmega64rfr2	0.07 s	0.07 s	--	--
✅	dryboot	atmega8	0.07 s	0.07 s	--	--
✅	dryrun	attiny13	0.07 s	0.06 s	0.06 s	0.06 s
✅	dryrun	attiny202	0.07 s	0.06 s	0.06 s	0.06 s
✅	dryrun	attiny28	0.07 s	0.07 s	--	--
✅	dryrun	attiny3224	0.07 s	0.07 s	0.07 s	0.07 s
✅	dryboot	attiny461	0.07 s	0.07 s	--	--
✅	dryrun	attiny4	0.07 s	0.07 s	--	--
✅	dryrun	attiny828	0.07 s	0.06 s	0.06 s	0.06 s
✅	dryboot	atxmega128a4	0.07 s	0.07 s	--	--
✅	dryboot	atxmega128c3	0.07 s	0.07 s	--	--
✅	dryboot	atxmega16e5	0.07 s	0.07 s	--	--
✅	dryboot	atxmega192c3	0.07 s	0.07 s	--	--
✅	dryboot	atxmega256c3	0.07 s	0.07 s	--	--
✅	dryboot	atxmega32e5	0.07 s	0.07 s	--	--
✅	dryboot	atxmega64a4	0.07 s	0.07 s	--	--
✅	dryboot	atxmega8e5	0.07 s	0.07 s	--	--
✅	dryboot	avr16ea28	0.07 s	0.07 s	--	--
✅	dryboot	avr32ea28	0.07 s	0.07 s	--	--
✅	dryboot	avr64dd14	0.07 s	0.07 s	--	--
✅	dryboot	avr64ea28	0.07 s	0.07 s	--	--
✅	dryboot	lgt8f328p	0.07 s	0.07 s	--	--

And here a more realistic example (you need to press enter and then again once you see the first result and have changed the board):

test-avrdude -b -d 0 \
  -p "-c urclock -P ch340 -p m328p -b 1000000 u7.7/weu-jPrac" \
  -p "-c urclock -P ch340 -p m328p -b 1000000 u7.7/-eu-jPrac" | \
  grep -v ^'[^T |]'

Testing avrdude version 7.3-20240225 (2078871)

The benchmark shows the wall clock time of avrdude carrying out the following tasks

Fl-ewv: erase flash, then write a sketch with three sections separated by holes to flash and verify
Fl-r: read the full flash memory and write to file
EE-wv: write data with two data sections separated by a hole to EEPROM and verify
EE-r: read the full EEPROM memory and write to file

The sketch and data payload is roughly one sixth of the respective memory size

	`-c pgm`	`-p part`	Fl-ewv	Fl-r	EE-wv	EE-r	Comments
✅	urclock	m328p	2.16 s	1.33 s	1.40 s	1.03 s	-b 1000000 u7.7/weu-jPrac
✅	urclock	m328p	1.99 s	1.32 s	1.42 s	1.02 s	-b 1000000 u7.7/-eu-jPrac

stefanrueger · 2024-02-28T10:56:22Z

@mcuee I think the CI failure is spurious... other than that I am now finished with this PR.

I would suggest closing the new benchmark discussion and either put a growing table into a locked new discussion item or into the wiki. Perhaps best to only report on existing programmers and published bootloaders that are in actual use.
That way it's easy to look up resuts...

mcuee · 2024-02-28T12:23:55Z

@mcuee I think the CI failure is spurious... other than that I am now finished with this PR.

Yes, that is due to some repo download issues. I have restarted the failed Linux build jobs and now it is okay.

mcuee · 2024-02-28T12:26:01Z

I would suggest closing the new benchmark discussion and either put a growing table into a locked new discussion item or into the wiki. Perhaps best to only report on existing programmers and published bootloaders that are in actual use. That way it's easy to look up resuts...

I agree.

I will create a new Wiki page first. If required, I will create a new locked discussion thread.

mcuee · 2024-02-28T12:49:48Z

This PR is nice. I think this PR can be merged.

The following is with avrdude MSVC64 binary, Arduino Uno Rev 3 clone using ATmega16U2 as the USB to Serial chip (fastest USB to Serial chip so far).

MINGW64 /c/work/avr/avrdude_test/avrdude_sr/tools
$ ./avrdude -c urclock -P COM20 -p ATmega328P -xshowall
avrdude: AVR device initialized and ready to accept instructions
0000ffffffff 2024-02-01 18.38 holes_rjmp_loops_32768B.hex 24576 store 7769 meta 39 boot 384 
u7.7 -eu-jPrac vector 25 (SPM_Ready) ATmega328P

$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t .\
 -p "-c urclock -P COM20 -p ATmega328P -b 2000000" \
 -p "-c urclock -P COM20 -p ATmega328P -b 1000000" \
 -p "-c urclock -P COM20 -p ATmega328P -b 500000" \
 -p "-c urclock -P COM20 -p ATmega328P -b 250000" \
 -p "-c urclock -P COM20 -p ATmega328P -b 115200" | grep -v ^'[^T |]'
Testing ./avrdude.exe version 7.3-20240228 (71a9cb47)

The benchmark shows the wall clock time of avrdude carrying out the following tasks
  - Fl-ewv: erase flash, then write a sketch with three sections separated by holes, and verify
  - Fl-r: read the entire flash memory and write to file
  - EE-wv: write data to EEPROM with two data sections separated by a hole, and verify
  - EE-r: read the entire EEPROM memory and write to file

The sketch and data payload is roughly one sixth of the respective memory size

| | `-c pgm` | `-p part` | Fl-ewv | Fl-r | EE-wv | EE-r | Comments |
|:-:|--:|--:|--:|--:|--:|--:|:--|
|✅|urclock|ATmega328P| 1.73 s| 0.62 s| 1.25 s| 0.45 s| -b 2000000|
|✅|urclock|ATmega328P| 1.79 s| 0.80 s| 1.25 s| 0.46 s| -b 1000000|
|✅|urclock|ATmega328P| 1.92 s| 1.15 s| 1.25 s| 0.50 s| -b 500000|
|✅|urclock|ATmega328P| 2.17 s| 1.86 s| 1.28 s| 0.58 s| -b 250000|
|✅|urclock|ATmega328P| 2.76 s| 3.49 s| 1.34 s| 0.81 s| -b 115200|

Copy/paste the result -- it becomes a nice table.

	`-c pgm`	`-p part`	Fl-ewv	Fl-r	EE-wv	EE-r	Comments
✅	urclock	ATmega328P	1.73 s	0.62 s	1.25 s	0.45 s	-b 2000000
✅	urclock	ATmega328P	1.79 s	0.80 s	1.25 s	0.46 s	-b 1000000
✅	urclock	ATmega328P	1.92 s	1.15 s	1.25 s	0.50 s	-b 500000
✅	urclock	ATmega328P	2.17 s	1.86 s	1.28 s	0.58 s	-b 250000
✅	urclock	ATmega328P	2.76 s	3.49 s	1.34 s	0.81 s	-b 115200

mcuee · 2024-02-28T13:27:21Z

One more result.

$ ./test-avrdude -e ./avrdude.exe -b -d 0 -t . \
 -p "-c usbasp -p ATmega328P -B 0.25" \
 -p "-c usbasp -p ATmega328P -B 0.5" \
 -p "-c usbasp -p ATmega328P -B 1" \
 -p "-c usbasp -p ATmega328P -B 2" \
 -p "-c usbasp -p ATmega328P -B 4" | grep -v ^'[^T |]'
Testing ./avrdude.exe version 7.3-20240228 (71a9cb47)

The benchmark shows the wall clock time of avrdude carrying out the following tasks
  - Fl-ewv: erase flash, then write a sketch with three sections separated by holes, and verify
  - Fl-r: read the entire flash memory and write to file
  - EE-wv: write data to EEPROM with two data sections separated by a hole, and verify
  - EE-r: read the entire EEPROM memory and write to file

The sketch and data payload is roughly one sixth of the respective memory size

| | `-c pgm` | `-p part` | Fl-ewv | Fl-r | EE-wv | EE-r | Comments |
|:-:|--:|--:|--:|--:|--:|--:|:--|
|✅|usbasp|ATmega328P| 2.53 s| 5.43 s| 3.03 s| 0.69 s| -B 0.25|
|✅|usbasp|ATmega328P| 2.49 s| 5.40 s| 3.03 s| 0.64 s| -B 0.5|
|✅|usbasp|ATmega328P| 3.10 s| 6.72 s| 3.03 s| 0.68 s| -B 1|
|✅|usbasp|ATmega328P| 3.81 s| 8.77 s| 3.10 s| 0.78 s| -B 2|
|✅|usbasp|ATmega328P| 4.80 s| 9.82 s| 3.12 s| 0.85 s| -B 4|

Table format.

	`-c pgm`	`-p part`	Fl-ewv	Fl-r	EE-wv	EE-r	Comments
✅	usbasp	ATmega328P	2.53 s	5.43 s	3.03 s	0.69 s	-B 0.25
✅	usbasp	ATmega328P	2.49 s	5.40 s	3.03 s	0.64 s	-B 0.5
✅	usbasp	ATmega328P	3.10 s	6.72 s	3.03 s	0.68 s	-B 1
✅	usbasp	ATmega328P	3.81 s	8.77 s	3.10 s	0.78 s	-B 2
✅	usbasp	ATmega328P	4.80 s	9.82 s	3.12 s	0.85 s	-B 4

mcuee · 2024-02-28T13:39:56Z

Wiki page created
https://github.com/avrdudes/avrdude/wiki/avrdude-benchmark-for-Flash-and-EEPROM-programming

mcuee · 2024-02-29T05:28:30Z

@stefanrueger

A minor thing, maybe we want to remove the mentioning of avrbench in the -b help text now that it is not really used.

MINGW64 /c/work/avr/avrdude_test/avrdude_main/tools
$ ./test-avrdude -h
Syntax: test-avrdude {<opts>}
Function: test AVRDUDE for certain programmer and part combinations
Options:
    -b                          benchmark only summarising overall and avrbench time
    -c <configuration spec>     additional configuration options used for all runs
    -d <sec>                    delay between test commands (default 0.75 seconds)
    -e <avrdude path>           set path of AVRDUDE executable (default avrdude)
    -l                          list test commands but do not execute them
    -p <programmer/part specs>  can be used multiple times, overrides default tests
    -s                          skip EEPROM tests for bootloaders
    -t <dir>                    temporary directory (default /dev/shm)
    -T                          Add dryrun/dryboot test cases to test test-avrdude
    -v                          verbose: show AVRDUDE error and warning messages
    -? or -h                    show this help text
Note: some Windows environments require the option -t . or similar
Examples:
    $ test-avrdude -s -p "-c dryboot -p t13" -p "-c dryrun -p m4809"
    $ test-avrdude -d 2 -p "-cjtag2updi -patmega4809 -Pusb:2341:0058 -r"

stefanrueger · 2024-02-29T07:28:07Z

Good catch, pushed a commit directly on git main

mcuee added the enhancement New feature or request label Feb 26, 2024

Use alternative for /dev/null when not available

6e6404c

Lowercase variable names in test-avrdude

3b56768

Uppercase names are typically used for environment variables in bash

Clarify -b option

f088a9b

Change /dev/null to temp file to be Windows compatible

cd41d2b

stefanrueger added 3 commits February 27, 2024 22:51

Widen chip erase tests to also capture -c urclock emulation

efd6bab

Drop chip erase and checking test for test-avrdude -b

82c001b

Mention avrbench legend in test-avrdude -b summary

61ed3dd

stefanrueger added 2 commits February 28, 2024 22:07

Remove avrbench and generate 4-number benchmark table with legend

64c5d1d

Change 1-byte file upload for emulated chip erase

e35714b

Keep case of part in test-avrdude -b -p "-p part ..."

71a9cb4

stefanrueger merged commit 076e2e9 into avrdudes:main Feb 28, 2024
12 checks passed

stefanrueger deleted the benchmark branch February 28, 2024 22:41

mcuee added this to the AVRDUDE 8.0 milestone Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add benchmark option `-b` for `test-avrdude` #1709

Add benchmark option `-b` for `test-avrdude` #1709

stefanrueger commented Feb 26, 2024 •

edited

Loading

mcuee commented Feb 26, 2024

mcuee commented Feb 26, 2024

mcuee commented Feb 26, 2024 •

edited

Loading

mcuee commented Feb 26, 2024 •

edited

Loading

mcuee commented Feb 26, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024 •

edited

Loading

stefanrueger commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024 •

edited

Loading

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 28, 2024 •

edited

Loading

mcuee commented Feb 28, 2024

stefanrueger commented Feb 28, 2024

stefanrueger commented Feb 28, 2024

mcuee commented Feb 28, 2024 •

edited

Loading

mcuee commented Feb 28, 2024

mcuee commented Feb 28, 2024 •

edited

Loading

mcuee commented Feb 28, 2024

mcuee commented Feb 28, 2024

mcuee commented Feb 29, 2024

stefanrueger commented Feb 29, 2024

Add benchmark option -b for test-avrdude #1709

Add benchmark option -b for test-avrdude #1709

Conversation

stefanrueger commented Feb 26, 2024 • edited Loading

mcuee commented Feb 26, 2024

mcuee commented Feb 26, 2024

mcuee commented Feb 26, 2024 • edited Loading

mcuee commented Feb 26, 2024 • edited Loading

mcuee commented Feb 26, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024 • edited Loading

stefanrueger commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024 • edited Loading

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 27, 2024

mcuee commented Feb 27, 2024

mcuee commented Feb 27, 2024

stefanrueger commented Feb 28, 2024 • edited Loading

mcuee commented Feb 28, 2024

stefanrueger commented Feb 28, 2024

stefanrueger commented Feb 28, 2024

mcuee commented Feb 28, 2024 • edited Loading

mcuee commented Feb 28, 2024

mcuee commented Feb 28, 2024 • edited Loading

mcuee commented Feb 28, 2024

mcuee commented Feb 28, 2024

mcuee commented Feb 29, 2024

stefanrueger commented Feb 29, 2024

Add benchmark option `-b` for `test-avrdude` #1709

Add benchmark option `-b` for `test-avrdude` #1709

stefanrueger commented Feb 26, 2024 •

edited

Loading

mcuee commented Feb 26, 2024 •

edited

Loading

mcuee commented Feb 26, 2024 •

edited

Loading

mcuee commented Feb 27, 2024 •

edited

Loading

mcuee commented Feb 27, 2024 •

edited

Loading

stefanrueger commented Feb 28, 2024 •

edited

Loading

mcuee commented Feb 28, 2024 •

edited

Loading

mcuee commented Feb 28, 2024 •

edited

Loading