Real Time Bidding (RTB) - Demand Side Platform framework
open-source library utilizing modern C++11/14/17 features and latest Boost.
What makes us different from other open-source RTB projects on GitHub :
- Our stack is fairly small and easy to integrate with any cmake project.
- Partitioned targeting data
- Monolithic bidder
- Code generation for targeting matchers and bidder executable
- Minimum dependency on outside vendors.
- Decoupled by C++ templates
- Very high throughput up to 105K QPS on 16 cores Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
- GUI for editing campaign budget
As a model cmake project please visit https://github.com/vanilla-rtb/rapid-bidder
We provide
and custom targetings and bidder executable generators https://github.com/vanilla-rtb/extensions
Multi-bidder-model-with-communicator-for-Win-notifications
best performance compared to other stacks - 105K QPS
runs on cloud with docker - see instructions
Recommended build environment: Linux or macOS, CMake >= 3.9.2, GCC >= 7.0, Clang >= 5.0 , Boost >= 1.67
Structure :
-
/ -- the root directory
- benchmarks/ -- optionaly built benchmarks for IPC caches, json parsers and low overhead IPC audit logger
- CRUD/ -- C++11 high performance HTTP-restful handlers based on boost.ASIO and CRUD API
- docker/ -- vanilla docker files and instructions on how to build and run
- jsonv/ -- DSL mapper of json encoded objects to C++ structures
- parsers/ -- fast zero copy, zero memory allocation parsers
- rapidjson/ -- very fast json tokenizer including SAX / DOM API but requires more memory
- rtb/ -- RTB framework
- core/ -- generic structures shared in the project ( RTB specific )
- common/ -- generic RTB agnostic structures
- datacache/ -- IPC cache generic classes for fast targeting and other lookups
- exchange/ -- exchange handlers implementation
- DSL/ -- DSL formats for jsonv , boost::any and rapidjson
- examples/ -- root to our sandbox with examples
- bidder/ -- collection of application specific classes to support targeting
- bidder_experimental/ -- bidder based on chained matchers model
- loader/ -- collection of application specific classes to support campaign loading
- campaign/ -- add/modify/delete campaign API + UI ( work in progress )
- datacache/ -- IPC cache implementation based on rtb/datacache model
- UI/ -- HTML and javascript for campaign budget management
-
[CMakeLists.txt] - cmake file
The stack of vanilla-rtb includes other C++11/14 projects and is referencing them via gh-subree. To update to the latest version of json-voorhees or CRUD we use the following commands :
- git subtree pull --prefix jsonv git@github.com:tgockel/json-voorhees.git master --squash
- git subtree pull --prefix CRUD git@github.com:venediktov/CRUD.git master --squash
using Selector = vanilla::ad_selector<vanilla::BudgetManager, Ad>;
using DSLT = DSL::GenericDSL<std::string, DSL::rapid_mapper> ;
//Return from each lambda becomes input for next lambda in the tuple of functions
auto retrieve_domain_f = [&cacheLoader](const std::string& dom, auto&& ...) {
Domain domain;
if(!cacheLoader.retrieve(domain,dom)) {
return boost::optional<uint32_t>();
}
return boost::optional<uint32_t>(domain.dom_id);
};
auto retrieve_ico_campaign_f = [&cacheLoader](boost::optional<uint32_t> dom_id, auto&& ...) {
std::vector<ICOCampaign> ico_campains;
if (!cacheLoader.retrieve(ico_campains,*dom_id)) {
return boost::optional<decltype(ico_campains)>();
}
return boost::optional<decltype(ico_campains)>(ico_campains);
};
vanilla::core::Banker<BudgetManager> banker;
auto retrieve_campaign_ads_f = [&](boost::optional<std::vector<ICOCampaign>> campaigns, auto && req, auto && imp) {
std::vector<Ad> retrieved_cached_ads;
for (auto &campaign : *campaigns) {
if (!cacheLoader.retrieve(retrieved_cached_ads, campaign.campaign_id, imp.banner.get().w, imp.banner.get().h)) {
continue;
}
auto budget_bid = banker.authorize(cacheLoader.get<CampaignCache<BidderConfig>>(), campaign.campaign_id);
std::transform(std::begin(retrieved_cached_ads),
std::end(retrieved_cached_ads),
std::begin(retrieved_cached_ads), [budget_bid](Ad & ad){
ad.auth_bid_micros = std::min(budget_bid, ad.max_bid_micros);
return ad;
});
}
return retrieved_cached_ads;
};
//creating bidder endpoint utilizing self-referencing pattern
exchange_handler<DSLT> bid_handler(std::chrono::milliseconds(10));
bid_handler
.logger([](const std::string &data) {
LOG(debug) << "bid request=" << data ;
})
.error_logger([](const std::string &data) {
LOG(debug) << "bid request error " << data ;
})
.auction_async([&](const BidRequest &request) {
thread_local vanilla::Bidder<DSLT, Selector> bidder(std::move(Selector()));
return bidder.bid(request,
request.site.get().ref,
//chained matchers lambdas defined above
retrieve_domain_f,
retrieve_ico_campaign_f,
retrieve_campaign_ads_f
);
});(installing dependencies before building vanilla stack)
$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release .. -G "Unix Makefiles"
$ make -j4 install
####### Creating Debug build #######
$ cd ..
$ mkdir Debug
$ cd Debug
$ cmake -DCMAKE_BUILD_TYPE=Debug .. -G "Unix Makefiles"
$ make -j4 installsame steps as above for linux , only difference is depending on your environment either Visual Studio or NMake project can be used
######### for NMake ####################
cd Release
cmake -DCMAKE_BUILD_TYPE=Release .. -G "NMake Makefiles"
cd ../Debug
cmake -DCMAKE_BUILD_TYPE=Debug .. -G "NMake Makefiles"
######### for Visual Studio ############
cd Release
cmake -DCMAKE_BUILD_TYPE=Release .. -G "Visual Studio 14 2015"
cd ../Debug
cmake -DCMAKE_BUILD_TYPE=Debug .. -G "Visual Studio 14 2015"For the reliable results it is suggested to have the build directory out of source tree.
The process involves creating a build directory, generating an Xcode project in that directory with CMake,
opening the project file generated in the build directory with Xcode, and lastly, adjusting project
settings as requried and kicking off the build.
To generate an Xcode project invoke cmake from an empty build directory with command line similar to cmake -G Xcode -DCMAKE_BUILD_TYPE=Release.
$ xcode-select --install
$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew doctor
$ brew install cmake
$ brew install boost
$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release .. -G "Unix Makefiles"
$ make -j4 install$ brew doctor
$ brew install cmake
$ brew install boost
$ brew install --with-clang llvm
$ mkdir Release
$ cd Release
$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=/usr/local/opt/llvm/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/opt/llvm/bin/clang++ -DCMAKE_RANLIB=/usr/local/opt/llvm/bin/llvm-ranlib -DCMAKE_AR=/usr/local/opt/llvm/bin/llvm-ar .. -G "Unix Makefiles"When building on Linux and Mac OS X with Make it's possible to automatically adjust the concurreny of the build using nproc for Linux and sysctl -n hw.physicalcpu for Mac OS X command line tools that returns number of CPUs available to the Make execution context:
Linux :
$ make -j $(nproc) ...
Mac OS X :
make -j $(sysctl -n hw.physicalcpu) ...
It's also possible to specifying the target Load Average with -l flag to prevent machine overloading:
$ make -l $(nproc) ...
And lastly, on Linux, it's possible to run the build with the BATCH scheduling mode (throughput oriented) as:
$ chrt --batch 0 make ...
All above considered the ultimate Make invocation combo on Linux would be something like:
$ chrt --batch 0 make -j$(nproc) -l$(nproc)
- HTTP-Bidder
- vanilla-rtb/Release/examples/bin$ ./http_bidder_test --config etc/config.cfg
- test with curl and apache benchmark
- vanilla-rtb/Release/examples/bin$ ./curl.sh --bidder
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --bidder
- Start Exchange Handler with HTTP handler
- vanilla-rtb/Release/examples/bin$ ./exchange_handler_test --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction-any
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --auction-rapid
- Start Exchange Handler distributing to multi-bidders via communicator
- vanilla-rtb/Release/examples/bin$ ./exchange_handler_test --config etc/config.cfg
- Mock-bidders starting multiple in one swoop, currently configured as 5 bidders in config
- vanilla-rtb/Release/examples/bin$ ./mock_bidder_test --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./ab.sh -n30000 -c10 --mock-bidders
- multi-bidders starting multiple in one swoop (currently configured as 3 bidders in config ) and starting exchange handler
- vanilla-rtb/Release/examples/bin$ ./multi_bidder --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./multi_exchange_handler --config etc/config.cfg
Testing cache loader of ad campaigns and ad budgets ( needs to be extended to get cache update events from outside )
- Cache loader
- vanilla-rtb/Release/examples/bin$ ./cache_loader_test --config etc/config.cfg
- Mock exchange - emulating bid requests
- To run mock exchange you need any python, and python "requests" library installed.
- for simple exchange please run
- vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py
- for various geo and size rotation please run
- vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py --geo "Russia:Moscow USA:NY USA:Washington USA:Chicago" --size '100:300 240:400 420:280'
- for more info please run
- vanilla-rtb/Release/examples/bin/mock_exchange$ python mock-x.py --help
-
Campaign manager - Budget
- vanilla-rtb/Release/examples/bin$ ./campaign_manager_test --config etc/config.cfg
-
Notification service and Slave-Banker
- vanilla-rtb/Release/examples/bin$ ./notification_service_test --config etc/config.cfg
- vanilla-rtb/Release/examples/bin$ ./slavebanker_service_test --config etc/config.cfg
-
To add/delete/modify campign budgets fire up UI by connecting to manager via browser http:://localhost:11081/campaign/index.html
We hope you like our project , fork our repo or chip in for couple of 🍻!
This project exists thanks to all the people who contribute. [Contribute].
Thank you to all our backers! 🙏 [Become a backer]
Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]
