add support for limiting builds via 'buildtest build --limit'

[shahzebsiddiqui]

without limit

(buildtest)  ~/Documents/github/buildtest/ [buildtest_build_limit*] buildtest bd -t python          
╭───────────────────────────────────── buildtest summary ──────────────────────────────────────╮                                                                                                                                           
│                                                                                              │                                                                                                                                           
│ User:               siddiq90                                                                 │                                                                                                                                           
│ Hostname:           DOE-7086392                                                              │                                                                                                                                           
│ Platform:           Darwin                                                                   │                                                                                                                                           
│ Current Time:       2023/02/09 14:06:15                                                      │                                                                                                                                           
│ buildtest path:     /Users/siddiq90/Documents/github/buildtest/bin/buildtest                 │                                                                                                                                           
│ buildtest version:  1.2                                                                      │                                                                                                                                           
│ python path:        /Users/siddiq90/.local/share/virtualenvs/buildtest-Ir4AdrfC/bin/python3  │                                                                                                                                           
│ python version:     3.7.3                                                                    │                                                                                                                                           
│ Configuration File: /Users/siddiq90/Documents/github/buildtest/buildtest/settings/config.yml │                                                                                                                                           
│ Test Directory:     /Users/siddiq90/Documents/github/buildtest/var/tests                     │                                                                                                                                           
│ Report File:        /Users/siddiq90/Documents/github/buildtest/var/report.json               │                                                                                                                                           
│ Command:            /Users/siddiq90/Documents/github/buildtest/bin/buildtest bd -t python    │                                                                                                                                           
│                                                                                              │                                                                                                                                           
╰──────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                                                           
─────────────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────────────
                          Discovered buildspecs                          
╔═══════════════════════════════════════════════════════════════════════╗
║ buildspec                                                             ║
╟───────────────────────────────────────────────────────────────────────╢
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml ║
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml ║
╚═══════════════════════════════════════════════════════════════════════╝
                        Buildspecs By Tag=python                         
╔═══════════════════════════════════════════════════════════════════════╗
║ buildspec                                                             ║
╟───────────────────────────────────────────────────────────────────────╢
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml ║
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml ║
╚═══════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  2
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  2
─────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 2
Invalid Buildspecs: 0
/Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml: VALID
/Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml: VALID
Total builder objects created: 2
                                                                                      Builders by type=script                                                                                       
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder               ┃ type   ┃ executor           ┃ compiler ┃ nodes ┃ procs ┃ description                             ┃ buildspecs                                                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ circle_area/3277599a  │ script │ generic.local.bash │ None     │ None  │ None  │ Calculate circle of area given a radius │ /Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml │
├───────────────────────┼────────┼────────────────────┼──────────┼───────┼───────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ python_hello/678e14ec │ script │ generic.local.bash │ None     │ None  │ None  │ Hello World python                      │ /Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml │
└───────────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴─────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
circle_area/3277599a: Creating test directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/3277599a
circle_area/3277599a: Creating the stage directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/3277599a/stage
circle_area/3277599a: Writing build script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/3277599a/circle_area_build.sh
python_hello/678e14ec: Creating test directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/678e14ec
python_hello/678e14ec: Creating the stage directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/678e14ec/stage
python_hello/678e14ec: Writing build script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/678e14ec/python_hello_build.sh
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 1 processes for processing builders
─────────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────
circle_area/3277599a does not have any dependencies adding test to queue
python_hello/678e14ec does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [circle_area/3277599a, python_hello/678e14ec]
circle_area/3277599a: Current Working Directory : /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/3277599a/stage
circle_area/3277599a: Running Test via command: bash --norc --noprofile -eo pipefail circle_area_build.sh
circle_area/3277599a: Test completed in 0.140125 seconds
circle_area/3277599a: Test completed with returncode: 0
circle_area/3277599a: Writing output file -  /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/3277599a/circle_area.out
circle_area/3277599a: Writing error file - /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/3277599a/circle_area.err
python_hello/678e14ec: Current Working Directory : /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/678e14ec/stage
python_hello/678e14ec: Running Test via command: bash --norc --noprofile -eo pipefail python_hello_build.sh
python_hello/678e14ec: Test completed in 0.122984 seconds
python_hello/678e14ec: Test completed with returncode: 0
python_hello/678e14ec: Writing output file -  /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/678e14ec/python_hello.out
python_hello/678e14ec: Writing error file - /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-hello/python_hello/678e14ec/python_hello.err
                                                    Test Summary                                                     
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ builder               ┃ executor           ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime  ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ circle_area/3277599a  │ generic.local.bash │ PASS   │ N/A N/A N/A                         │ 0          │ 0.140125 │
├───────────────────────┼────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
│ python_hello/678e14ec │ generic.local.bash │ PASS   │ N/A N/A N/A                         │ 0          │ 0.122984 │
└───────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘



Passed Tests: 2/2 Percentage: 100.000%
Failed Tests: 0/2 Percentage: 0.000%


Adding 2 test results to /Users/siddiq90/Documents/github/buildtest/var/report.json
Writing Logfile to: /Users/siddiq90/Documents/github/buildtest/var/logs/buildtest_x1ltfop9.log

with limit

(buildtest)  ~/Documents/github/buildtest/ [buildtest_build_limit*] buildtest bd -t python --limit=1
╭───────────────────────────────────────── buildtest summary ─────────────────────────────────────────╮                                                                                                                                    
│                                                                                                     │                                                                                                                                    
│ User:               siddiq90                                                                        │                                                                                                                                    
│ Hostname:           DOE-7086392                                                                     │                                                                                                                                    
│ Platform:           Darwin                                                                          │                                                                                                                                    
│ Current Time:       2023/02/09 14:06:04                                                             │                                                                                                                                    
│ buildtest path:     /Users/siddiq90/Documents/github/buildtest/bin/buildtest                        │                                                                                                                                    
│ buildtest version:  1.2                                                                             │                                                                                                                                    
│ python path:        /Users/siddiq90/.local/share/virtualenvs/buildtest-Ir4AdrfC/bin/python3         │                                                                                                                                    
│ python version:     3.7.3                                                                           │                                                                                                                                    
│ Configuration File: /Users/siddiq90/Documents/github/buildtest/buildtest/settings/config.yml        │                                                                                                                                    
│ Test Directory:     /Users/siddiq90/Documents/github/buildtest/var/tests                            │                                                                                                                                    
│ Report File:        /Users/siddiq90/Documents/github/buildtest/var/report.json                      │                                                                                                                                    
│ Command:            /Users/siddiq90/Documents/github/buildtest/bin/buildtest bd -t python --limit=1 │                                                                                                                                    
│                                                                                                     │                                                                                                                                    
╰─────────────────────────────────────────────────────────────────────────────────────────────────────╯                                                                                                                                    
─────────────────────────────────────────────────────────────────────────────────────────────────────────  Discovering Buildspecs ─────────────────────────────────────────────────────────────────────────────────────────────────────────
                          Discovered buildspecs                          
╔═══════════════════════════════════════════════════════════════════════╗
║ buildspec                                                             ║
╟───────────────────────────────────────────────────────────────────────╢
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml ║
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml ║
╚═══════════════════════════════════════════════════════════════════════╝
                        Buildspecs By Tag=python                         
╔═══════════════════════════════════════════════════════════════════════╗
║ buildspec                                                             ║
╟───────────────────────────────────────────────────────────────────────╢
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml ║
║ /Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml ║
╚═══════════════════════════════════════════════════════════════════════╝


Total Discovered Buildspecs:  2
Total Excluded Buildspecs:  0
Detected Buildspecs after exclusion:  2
─────────────────────────────────────────────────────────────────────────────────────────────────────────── Parsing Buildspecs ────────────────────────────────────────────────────────────────────────────────────────────────────────────
Valid Buildspecs: 2
Invalid Buildspecs: 0
/Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml: VALID
/Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml: VALID
Total builder objects created: 2
                                                                                      Builders by type=script                                                                                       
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ builder               ┃ type   ┃ executor           ┃ compiler ┃ nodes ┃ procs ┃ description                             ┃ buildspecs                                                            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ circle_area/8fbf0081  │ script │ generic.local.bash │ None     │ None  │ None  │ Calculate circle of area given a radius │ /Users/siddiq90/Documents/github/buildtest/tutorials/python-shell.yml │
├───────────────────────┼────────┼────────────────────┼──────────┼───────┼───────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────┤
│ python_hello/6b29e933 │ script │ generic.local.bash │ None     │ None  │ None  │ Hello World python                      │ /Users/siddiq90/Documents/github/buildtest/tutorials/python-hello.yml │
└───────────────────────┴────────┴────────────────────┴──────────┴───────┴───────┴─────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────┘
Limit number of tests to 1 for Building and Running. 
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Building Test ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
circle_area/8fbf0081: Creating test directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8fbf0081
circle_area/8fbf0081: Creating the stage directory: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8fbf0081/stage
circle_area/8fbf0081: Writing build script: /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8fbf0081/circle_area_build.sh
────────────────────────────────────────────────────────────────────────────────────────────────────────────── Running Tests ──────────────────────────────────────────────────────────────────────────────────────────────────────────────
Spawning 1 processes for processing builders
─────────────────────────────────────────────────────────────────────────────────────────────────────────────── Iteration 1 ───────────────────────────────────────────────────────────────────────────────────────────────────────────────
circle_area/8fbf0081 does not have any dependencies adding test to queue
In this iteration we are going to run the following tests: [circle_area/8fbf0081]
circle_area/8fbf0081: Current Working Directory : /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8fbf0081/stage
circle_area/8fbf0081: Running Test via command: bash --norc --noprofile -eo pipefail circle_area_build.sh
circle_area/8fbf0081: Test completed in 0.14818 seconds
circle_area/8fbf0081: Test completed with returncode: 0
circle_area/8fbf0081: Writing output file -  /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8fbf0081/circle_area.out
circle_area/8fbf0081: Writing error file - /Users/siddiq90/Documents/github/buildtest/var/tests/generic.local.bash/python-shell/circle_area/8fbf0081/circle_area.err
                                                   Test Summary                                                    
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┓
┃ builder              ┃ executor           ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime ┃
┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━┩
│ circle_area/8fbf0081 │ generic.local.bash │ PASS   │ N/A N/A N/A                         │ 0          │ 0.14818 │
└──────────────────────┴────────────────────┴────────┴─────────────────────────────────────┴────────────┴─────────┘



Passed Tests: 1/1 Percentage: 100.000%
Failed Tests: 0/1 Percentage: 0.000%


Adding 1 test results to /Users/siddiq90/Documents/github/buildtest/var/report.json
Writing Logfile to: /Users/siddiq90/Documents/github/buildtest/var/logs/buildtest_tsl1697a.log

[shahzebsiddiqui]

@prathmesh4321 @Xiangs18 hey i am implementing a new feature for buildtest build to limit number of tests, can you take a look at this feature and review this.

[shahzebsiddiqui]

@Xiangs18 i asked black in psf/black#3562 regarding the CI check failure. I locally updated black once #1370 was merged it and it turns out the new version of black was the issue.