Source Code in Repo

[billywhizz]

I think it would be useful and involve less friction if i could push latest techempower source for my framework to my own external repository and then have the dockerfile that runs in tfb pick up the latest changes from my repo whenever container is built during the test runs. I can understand concerns about this re. lack of oversight/visibility but is this really an issue once a framework has initially been accepted and as long as source is available for others to view/verify in an external repo?

if this was allowed (i'm not sure it isn't as i can see other frameworks who already pull down source code and patches from external sites in their dockerfiles) then it would mean a lot less work/toil for TE folks reviewing changes and delays for framework maintainers who want to push changes out between runs of the benchmark.

@bhauer @nbrady-techempower any opinions on what is allowed/possible re. this? thx.

[billywhizz]

e.g. not picking on lithium maintainers but i noticed this and there are already people who do this and don't check any source code into TE repo - is this ok? if so, i think i will do similar myself.
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/lithium/compile.sh

#! /bin/sh

DB_FLAG=$1
COMMIT="$2"

if [ "$COMMIT" = "" ]; then
  COMMIT="a046b3345098157849d9e2ab49a475aaabf4a90f"
fi

if [ $DB_FLAG = "TFB_MYSQL" ]; then
  CXX_FLAGS="-I /usr/include/mariadb  -lmariadbclient "
  wget https://raw.githubusercontent.com/matt-42/lithium/$COMMIT/single_headers/lithium_mysql.hh
elif [ $DB_FLAG = "TFB_PGSQL" ]; then
  CXX_FLAGS="-lpthread  -L/usr/lib -lpq -I/postgres-bab150045bd9766869f471ede88734ea0989261c/src/include"
  wget https://raw.githubusercontent.com/matt-42/lithium/$COMMIT/single_headers/lithium_pgsql.hh
fi

COMPILER="clang++"

wget https://raw.githubusercontent.com/matt-42/lithium/$COMMIT/single_headers/lithium_http_server.hh

if [ $COMPILER = "clang++" ]; then
  clang++ -fprofile-instr-generate=./profile.prof -flto -DLITHIUM_SERVER_NAME=l -DPROFILE_MODE -DNDEBUG -D$DB_FLAG -O3 -march=native -std=c++17 ./lithium.cc $CXX_FLAGS -lpthread -lboost_context -lssl -lcrypto -o /lithium_tbf
  /lithium_tbf tfb-database 8081
  llvm-profdata-10 merge -output=./profile.pgo  ./profile.prof
  clang++ -fprofile-instr-use=./profile.pgo -flto -DLITHIUM_SERVER_NAME=l -DNDEBUG -D$DB_FLAG -O3 -march=native -std=c++17 ./lithium.cc $CXX_FLAGS -lpthread -lboost_context -lssl -lcrypto -o /lithium_tbf
else
  g++ -flto -DLITHIUM_SERVER_NAME=l -DNDEBUG -D$DB_FLAG -O3 -march=native -std=c++17 ./lithium.cc $CXX_FLAGS -lpthread -lboost_context -lssl -lcrypto -o /lithium_tbf
fi

/lithium_tbf tfb-database 8080

[NateBrady23]

You don’t need to commit source code to the repo but when you pull it in you have to point to a specific commit or version like lithium is doing.

The only code that needs to be committed into the repo is the code that implements the test cases.

Pulling “latest” or anything like that can cause a test that was passing CI initially to break. The goal is that any commit in the repo should be able to reproduce results at that time.

[billywhizz]

thanks for clarifying @nbrady-techempower.