Context-Generic Programming
Using CGP for static dispatch #3
Replies: 17 comments 17 replies
-
|
Thank you for starting the discussion and your elaboration. I had a look at the CGP code, and overall, it looks solid to me; I just filled a PR with docs, as it was mentioned on the homepage under To your comments:
I just wrote the first draft of the specfile, which roughly elaborates the background, context, and goals: https://gist.github.com/marvin-hansen/5f7ffb9ff6e2afe0df77da767bb92db2 As for code, I see if I can extract the core out of my monorepo next week. This is an internal project, so there is no way I can share the entire repo, but I think I can extract enough to experiment around with a CGP implementation. Will post an update probably later on Monday. Thank you for all your help. |
Beta Was this translation helpful? Give feedback.
-
|
I made a public repo with my current code: https://github.com/marvin-hansen/cgp-exploration/tree/cgp/queng_integration The main branch still contains the previous implementation (minus enum_dispatch, as I have removed that part) whereas the CGP branch has been trimmed down to the essence of the integration task at hand. The relevant folder is queng_integration with the rest being largely internal dependencies or build related files. Over the next few days, I am going to convert that code into CGP and may reach out in case I bump into an issue, |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing your code! I have taken a quick look, and I have some ideas on how you can better design your interfaces. If my understanding is correct, the main goal is to remove the need for custom macros like I will share my sketch in the next few days when I have the time. |
Beta Was this translation helpful? Give feedback.
-
|
Here is a rough sketch of how you could structure the interfaces and implementations for your project: pub mod traits {
use std::collections::HashSet;
use cgp::prelude::*;
#[cgp_component {
provider: ApiUrlGetter,
}]
pub trait HasApiUrl {
fn api_url(&self) -> &str;
fn api_wss_url(&self) -> &str;
}
#[cgp_component {
name: SymbolTypeComponent,
provider: ProvideSymbolType,
}]
pub trait HasSymbolType: Async {
type Symbol: Async;
}
#[cgp_component {
name: TimeResolutionTypeComponent,
provider: ProvideTimeResolutionType,
}]
pub trait HasTimeResolutionType: Async {
type TimeResolution: Async;
}
#[cgp_component {
provider: SymbolFetcher,
}]
#[async_trait]
pub trait CanFetchExchangeSymbols: HasSymbolType + HasErrorType {
async fn fetch_exchange_symbols(&self) -> Result<HashSet<Self::Symbol>, Self::Error>;
}
#[cgp_component {
provider: SymbolValidator,
}]
#[async_trait]
pub trait CanValidateSymbols: HasSymbolType + HasErrorType {
async fn vaidate_symbols(&self, symbols: &[Self::Symbol]) -> Result<bool, Self::Error>;
}
#[cgp_component {
provider: EventProcessor,
}]
#[async_trait]
pub trait CanProcessEvent: Async + HasErrorType {
async fn process_event(&self, data: &[Vec<u8>]) -> Result<(), Self::Error>;
}
#[cgp_component {
provider: OhlcvDataStarter,
}]
#[async_trait]
pub trait CanStartOhlcvData: HasSymbolType + HasTimeResolutionType + HasErrorType {
async fn start_ohlcv_data(
&self,
symbols: &[Self::Symbol],
time_resolution: &Self::TimeResolution,
) -> Result<(), Self::Error>;
}
#[cgp_component {
provider: OhlcvDataStopper,
}]
#[async_trait]
pub trait CanStopOhlcvData: HasSymbolType + HasErrorType {
async fn stop_ohlcv_data(&self, symbols: &[Self::Symbol]) -> Result<(), Self::Error>;
}
}
pub mod impls {
use core::fmt::Display;
use cgp::core::Async;
use cgp::prelude::HasErrorType;
use super::traits::*;
pub struct UseBinanceUsdFuturesMainnetUrl;
impl<Context> ApiUrlGetter<Context> for UseBinanceUsdFuturesMainnetUrl {
fn api_url(_context: &Context) -> &str {
"https://dapi.binance.com/dapi/v1"
}
fn api_wss_url(_context: &Context) -> &str {
"wss://dstream.binance.com/ws"
}
}
pub struct UseBinanceUsdFuturesTestnetUrl;
impl<Context> ApiUrlGetter<Context> for UseBinanceUsdFuturesTestnetUrl {
fn api_url(_context: &Context) -> &str {
"https://testnet.binancefuture.com/api/v3"
}
fn api_wss_url(_context: &Context) -> &str {
"wss://dstream.binancefuture.com"
}
}
pub struct UseBinanceDataIntegration;
impl<Context> OhlcvDataStarter<Context> for UseBinanceDataIntegration
where
Context:
HasSymbolType<Symbol: Display> + HasTimeResolutionType + HasApiUrl + CanProcessEvent,
{
async fn start_ohlcv_data(
_context: &Context,
_symbols: &[Context::Symbol],
_time_resolution: &Context::TimeResolution,
) -> Result<(), Context::Error> {
todo!()
}
}
pub struct PrintEvent;
impl<Context> EventProcessor<Context> for PrintEvent
where
Context: Async + HasErrorType,
{
async fn process_event(
_context: &Context,
_data: &[Vec<u8>],
) -> Result<(), Context::Error> {
Ok(())
}
}
}Here are a few improvements that you could consider:
With this design, you would define different concrete contexts that are wired with different event processors and different API URLs. Hope this can give you a starting point to consider how to use CGP with your project. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @soareschen Okay, I've re-designed the old interfaces as minimal traits using CGP. The old design used the core-binance-integration as a kinda template in the sense that Now the idea is, that the URL is composed into the specific integration using the delegate component pattern. That makes sense to me and actually centralizes all maintenance related to API urls in one single file, so that is a big plus. However, what isn't so clear to me is the following: The start/stop data method actually share state i.e active symbols & tokio handlers, see the old struct: #[derive(Default)]
pub struct ImsBinanceDataIntegration {
api_base_url: String,
api_wss_url: String,
http_client: Client,
symbols_active_trade: RwLock<Vec<String>>,
symbols_active_ohlcv: RwLock<Vec<String>>,
symbol_cache: RwLock<Option<(HashSet<String>, Instant)>>,
trade_handlers: RwLock<HashMap<String, JoinHandle<()>>>,
ohlcv_handlers: RwLock<HashMap<String, JoinHandle<()>>>,
}When I implement the actual providers of the new CanStartTradeData / CanStopTradeData traits, can I just keep the fields of |
Beta Was this translation helpful? Give feedback.
-
|
Okay, somehow I've overlooked the hello world example with field access right on the homepage. Basically the way to access fields from the context seems to be the HasField macro. '''rust impl Greeter for GreetHello How is the get_field accessor implemented? Does it translate to an ordinary getter? |
Beta Was this translation helpful? Give feedback.
-
|
There are many ways you can organize the interface of the field accessors to decouple the implementation from the concrete context. The most naive and straightforward way is to define a single trait that contains all the field accessor methods: #[cgp_component {
provider: BinanceIntegrationFieldsGetter,
}]
pub trait HasBinanceIntegrationFields {
fn http_client(&self) -> &Client;
fn symbols_active_trade(&self) -> &RwLock<Vec<String>>;
fn symbols_active_ohlcv(&self) -> &RwLock<Vec<String>>;
fn symbol_cache(&self) -> &RwLock<Option<(HashSet<String>, Instant)>>;
fn trade_handlers(&self) -> &RwLock<HashMap<String, JoinHandle<()>>>;
fn ohlcv_handlers(&self) -> &RwLock<HashMap<String, JoinHandle<()>>>;
}You can also split the field accessor methods into multiple traits, depending on your use cases. This would allow some contexts to skip providing some of the fields, in case they do not use certain providers. Or it could allow you to use the same accessor traits with other non-Binance applications. For example: #[cgp_component {
provider: HttpClientGetter,
}]
pub trait HasHttpClient {
fn http_client(&self) -> &Client;
}
#[cgp_component {
provider: BinanceSymbolsGetter,
}]
pub trait HasBinanceSymbols {
fn symbols_active_trade(&self) -> &RwLock<Vec<String>>;
fn symbols_active_ohlcv(&self) -> &RwLock<Vec<String>>;
fn symbol_cache(&self) -> &RwLock<Option<(HashSet<String>, Instant)>>;
}
#[cgp_component {
provider: JoinHandlesGetter,
}]
pub trait HasJoinHandles {
fn trade_handlers(&self) -> &RwLock<HashMap<String, JoinHandle<()>>>;
fn ohlcv_handlers(&self) -> &RwLock<HashMap<String, JoinHandle<()>>>;
}I don't yet have the time to write the book section for impl<Context> HttpClientGetter<Context> for UseContext
where
Context: HasField<symbol!("http_client"), Value = Client>,
{
fn http_client(context: &Context) -> &Client {
context.get_field(PhantomData)
}
}The other purpose for So it is up to you to decide how granular you want to define your interfaces. If you find the definition of many small traits too tedious, you are free to define monolithic accessor traits, or skip them altogether and use |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @soareschen Ultimately, I settled for a single trait for all getters. These fields reflect the particular implementation and should not leak to the outside world to preserve the option to update or improve the implementation without being constrained by preserving leaked internals. However, I hit a snag while implementing the CanFetchExchangeSymbols trait. The trait is defined as: #[cgp_component {provider: SymbolFetcher,}]
#[async_trait]
pub trait CanFetchExchangeSymbols: HasSymbolType + HasErrorType + Async {
async fn fetch_exchange_symbols(&self) -> Result<HashSet<Self::Symbol>, Self::Error>;
}Note, this is an async trait thus requires an async implementation. The general idea is, that I have to:
So I went on and drafted the following implementation: impl HasSymbolType for ImsBinanceDataIntegration { type Symbol = String; }
impl HasErrorType for ImsBinanceDataIntegration { type Error = (); }
impl<Context> CanFetchExchangeSymbols<Context> for ImsBinanceDataIntegration
where
Context: HasSymbolType<Symbol = String>
+ HasErrorType
+ HasBinanceIntegrationFields
+ HasApiUrl
+ Async
+ Sync
+ Send,
{
async fn fetch_exchange_symbols(
context: &Context,
) -> Result<HashSet<Context::Symbol>, Context::Error> {
// Check cache first
if let Some((symbols, timestamp)) = &*context.symbol_cache().read().await {
if timestamp.elapsed() < SYMBOL_CACHE_DURATION {
return Ok(symbols.clone());
}
}
....
Ok(symbols)
}However, when I try to compile this, I get the following error: I don't know what's happening here, but it looks complex to me. Any idea how to fix that? |
Beta Was this translation helpful? Give feedback.
-
|
The name of the provider trait is |
Beta Was this translation helpful? Give feedback.
-
|
Dang, this is it. Problem solved. Thank you. I guess it takes a bit time to get used to all the different parts and how they relate to each other. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, I think I got the SymbolFetchProvider and the SymbolValidatorProvider correctly implemented. Specifically, the injected EventProccessorProvider triggers a scope warning in the spawened tokio task because context would move. let handle = tokio::spawn(async move {
let mut reconnect_time = Instant::now() + RECONNECT_INTERVAL;
let mut ws_stream = ws_stream;
'connection: loop {
loop {
// Check if we need to reconnect
if Instant::now() >= reconnect_time {
break;
}
// Process messages with timeout
match tokio::time::timeout(Duration::from_secs(1), ws_stream.next()).await {
Ok(Some(Ok(msg))) => {
if let Message::Text(text) = msg {
let bar = utils::extract_ohlcv_bar_from_json(
text.as_str(),
&symbol_clone,
)
.await;
if let Some(bar) = bar {
let (_, data) = OHLCVBar::encode_to_sbe(bar)
.expect("Failed to encode OHLCV data");
// Previously, that worked c/b processor was actually an arc reference to the actual processor
// if let Err(e) = processor.process_event(&[data]).await {
// Scope violation: context` escapes the associated function body here
// argument requires that `'1` must outlive `'static
if let Err(e) = context.process_event(&[data]).await {
eprintln!("Error processing OHLCV data: {}", e);
return;
}
}
}
}Previously, the event processor was a cloned Arc reference so it was safe to move into the newly spawned task. Now, with the EventProcessor basically attached to context, I get the following compile error: rror[E0521]: borrowed data escapes outside of associated function
--> queng_integration/data/binance_core_data_integration/src/ohlcv_data_integration.rs:93:26
|
62 | context: &Context,
| ------- - let's call the lifetime of this reference `'1`
| |
| `context` is a reference that is only valid in the associated function body
...
93 | let handle = tokio::spawn(async move {
| __________________________^
94 | | let mut reconnect_time = Instant::now() + RECONNECT_INTERVAL;
95 | | let mut ws_stream = ws_stream;
... |
177 | | }
178 | | });
| | ^
| | |
| |______________`context` escapes the associated function body here
| argument requires that `'1` must outlive `'static`The compiler is correct here, but I am not so sure how to call the processor from within the task without moving context. Any idea how to get around this issue? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you need to add additional constraint in the provider implementation that Usually, we define CGP contexts with the following boilerplate to allow cloning and using them inside spawned tasks: #[derive(Clone)]
pub struct AppContext {
pub fields: Arc<AppContextFields>,
}
#[derive(HasField)]
pub struct AppContextFields {
// place your fields here instead.
}
// Implementing Deref will help propagating `HasField` implementation
// via `HasField`'s blanket implementation.
impl Deref for AppContext {
type Target = AppContextFields;
fn deref(&self) -> &AppContextFields {
&self.fields
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Actually, just cloning the context without Arc did the trick. |
Beta Was this translation helpful? Give feedback.
-
|
@soareschen When I try to wire together a specific integration, say Spot, using the default providers implemented in the core integration, I observe a few strange things:
Lets start with the example code triggering the issue: #[tokio::main]
async fn main() -> Result<(), Error> {
let integration = ImsBinanceSpotDataIntegration::new();
// This works
let api_url = integration.api_url();
println!("api_url: {}", api_url);
let api_wss_url = integration.api_wss_url();
println!("api_wss_url: {}", api_wss_url);
// This doesn't work and throws an error
let res = integration.fetch_exchange_symbols().await;
if res.is_ok() {
let symbols = res.unwrap();
println!("symbols: {:#?}", symbols);
} else {
println!("Error: {}", res.unwrap_err());
}
Ok(())
}The ImsBinanceSpotDataIntegration is implemented as following: #[derive(Default, Copy, Clone)]
pub struct ImsBinanceSpotDataIntegration {}
impl ImsBinanceSpotDataIntegration {
pub fn new() -> Self {
Self {}
}
}
pub struct ImsBinanceSpotDataIntegrationComponents;
impl HasComponents for ImsBinanceSpotDataIntegration {
type Components = ImsBinanceSpotDataIntegrationComponents;
}
delegate_components! {
ImsBinanceSpotDataIntegrationComponents {
ApiUrlComponent: UseBinanceSpotMainnetUrl,
[
// These are always the same for all Binance integrations
SymbolFetchComponent,
SymbolValidatorComponent,
OhlcvDataStreamComponent,
TradeDataStreamComponent,
SymbolTypeComponent,
TimeResolutionTypeComponent,
]: UseImsBinanceDataIntegration,
}
}The error I get is: I suspect, the problem may arise from the custom error type that I use, which is defined as an enum: use std::error::Error;
#[derive(Debug, Clone, Eq, PartialEq, Hash)]
pub enum ImsDataIntegrationError {
FailedToFetchSymbols(String),
FailedToDeserializeJsonSymbols(String),
FailedToExtractSymbolsFromResponse(String),
FailedToValidateSymbols(String),
SymbolNotFound(String),
}
impl Error for ImsDataIntegrationError {}
impl std::fmt::Display for ImsDataIntegrationError {
...The symbol provider in question is implemented as: impl<Context> SymbolFetchProvider<Context> for UseImsBinanceDataIntegration
where
Context: HasSymbolType<Symbol = String>
+ HasErrorType<Error = ImsDataIntegrationError>
+ HasBinanceIntegrationFields
+ HasApiUrl
+ Async
+ Sync
+ Send,
{
async fn fetch_exchange_symbols(
context: &Context,
) -> Result<HashSet<Context::Symbol>, Context::Error> {
// Check cache first
if let Some((symbols, timestamp)) = &*context.symbol_cache().read().await {
if timestamp.elapsed() < SYMBOL_CACHE_DURATION {
return Ok(symbols.clone());
}
}
// Cache is stale or doesn't exist, fetch symbols from API
let url = format!("{}/exchangeInfo", context.api_url());
let response = context
.http_client()
.get(&url)
.send()
.await
.map_err(|e| ImsDataIntegrationError::FailedToFetchSymbols(e.to_string()).into())?;
let data: Value = response.json().await.map_err(|e| {
ImsDataIntegrationError::FailedToDeserializeJsonSymbols(e.to_string()).into()
})?;
let symbols = data["symbols"]
.as_array()
.ok_or_else(|| {
ImsDataIntegrationError::FailedToExtractSymbolsFromResponse("".to_string()).into()
})?
.iter()
.filter_map(|s| s["symbol"].as_str().map(String::from))
.collect::<HashSet<_>>();
// Update cache
*context.symbol_cache().write().await = Some((symbols.clone(), Instant::now()));
Ok(symbols)
}
}I understand that the HasErrorType specialization somehow needs to be made explicit in the SpotDataInteagetion. I just could not figure this one out. The other thing is, I did read though the error chapter, but TBH, I found it way too complicated for my humble use case that I did not implemented the illustrated path and instead opted for a much simpler error enum. I think the book even mentioned that its better to start with a simple approach at the beginning, which I am trying here but, somehow that is not working yet. Any advice? |
Beta Was this translation helpful? Give feedback.
-
|
I think you also need to implement pub struct UseImsDataIntegrationError;
impl<Context> ProvideErrorType<Context> for UseImsDataIntegrationError {
type Error = ImsDataIntegrationError;
}
delegate_components! {
ImsBinanceSpotDataIntegrationComponents {
ErrorTypeComponent: UseImsDataIntegrationError,
}
}There is also a currently undocumented helper that you can use to simplify the error type provider implementation: delegate_components! {
ImsBinanceSpotDataIntegrationComponents {
ErrorTypeComponent: WithType<ImsDataIntegrationError>,
}
} |
Beta Was this translation helpful? Give feedback.
-
|
Alright, the error issue has been solved that way. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I have to circle back to the field / getter issue because that started to haunt me badly: The way I understand your proposal a few days ago is to basically establish a public interface, just like all the other components.
The last point really hit me hard because the provider is implemented against a generic context meaning, even if I were to set fields against the implementing struct, there is no way I can access the fields as these require self, but you have only context, which brings me back to square one, I have to expose my fields as public trait with all the problems described in 1). Initially, I thought I can get away with a private trait within the implementation, but that breaks the wiring of components due to an unsatisfied trait bound error because the private trait is not part of the public interface, hence the interface and provider do not match, and that again brings me back to square one, I have to expose my internal fields as public trait with all the problems described in 1). This is a very practical problem, because right now, I cannot wire together my Spot Integration because the underlying SymbolFetchProvider requires a bunch of fields that cause exactly the described problem. Maybe I am missing something or I am not understanding something, but I have to ask this explicitly: Is there any way to hide internal fields required for a component provider with CGP? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the main way of doing this with CGP is to define lower level implementations and interfaces that interact with the fields, so that the high level core logic never interact with the fields directly. The idea is that you should have a collection of core logic that do not know anything about the concrete context or its fields. You can then implement the lower level components that either make use of the fields directly, or replaceable with other components that are implemented in other ways.
This is the main reason why each field should be in their own accessor trait, because it is usually not possible upfront to know which field is optional, and which is required. For example, one way you can decouple the pub trait HasSymbolCache: HasSymbolType {
fn symbol_cache(&self) -> &RwLock<Option<(HashSet<Self::Symbol>, Instant)>>;
}
pub struct FetchSymbolFromCache<InFetcher>(pub PhantomData<InFetcher>);
impl<Context, InFetcher> SymbolFetcher<Context> for FetchSymbolFromCache<InFetcher>
where
Context: HasSymbolCache + HasErrorType,
InFetcher: SymbolFetcher<Context>,
Context::Symbol: Clone,
{
async fn fetch_exchange_symbols(
context: &Context,
) -> Result<HashSet<Context::Symbol>, Context::Error> {
let symbol_cache = context.symbol_cache();
if let Some((symbols, timestamp)) = &*symbol_cache.read().await {
if timestamp.elapsed() < SYMBOL_CACHE_DURATION {
return Ok(symbols.clone());
}
}
let symbols = InFetcher::fetch_exchange_symbols(context).await?;
*symbol_cache.write().await = Some((symbols.clone(), Instant::now()));
Ok(symbols)
}
}You can then statically inspect that If you really want to be sure that a trait like |
Beta Was this translation helpful? Give feedback.
-
|
When I try to use the HasField macro to get access to internal fields, as suggested here: #[derive(Clone)]
pub struct UseImsBinanceDataIntegration {
pub fields: Arc<ImsBinanceDataContextFields>,
}
impl UseImsBinanceDataIntegration {
pub fn new() -> Self {
Self {
fields: Arc::new(ImsBinanceDataContextFields::new()),
}
}
}
// Implementing Deref will help propagating `HasField` implementation
// via `HasField`'s blanket implementation.
impl Deref for UseImsBinanceDataIntegration {
type Target = ImsBinanceDataContextFields;
fn deref(&self) -> &ImsBinanceDataContextFields {
&self.fields
}
}
#[derive(HasField, Default)]
pub struct ImsBinanceDataContextFields {
http_client: Client,
symbols_active_trade: RwLock<Vec<String>>,
symbols_active_ohlcv: RwLock<Vec<String>>,
symbol_cache: RwLock<Option<(HashSet<String>, Instant)>>,
trade_handlers: RwLock<HashMap<String, JoinHandle<()>>>,
ohlcv_handlers: RwLock<HashMap<String, JoinHandle<()>>>,
}
impl ImsBinanceDataContextFields {
pub fn new() -> Self {
Self {
http_client: Client::new(),
symbols_active_trade: RwLock::new(Vec::with_capacity(50)),
symbols_active_ohlcv: RwLock::new(Vec::with_capacity(50)),
symbol_cache: RwLock::new(None),
trade_handlers: RwLock::new(HashMap::with_capacity(50)),
ohlcv_handlers: RwLock::new(HashMap::with_capacity(50)),
}
}
}Then I define the getter interface: #[cgp_component {
name: BinanceIntegrationFieldsComponent,
provider: BinanceIntegrationFieldsProvider,
}]
pub trait HasBinanceIntegrationFields {
fn http_client(&self) -> &Client;
fn symbols_active_trade(&self) -> &RwLock<Vec<String>>;
fn symbols_active_ohlcv(&self) -> &RwLock<Vec<String>>;
....
}And implement the getters for context: impl<Context> BinanceIntegrationFieldsProvider<Context> for UseImsBinanceDataIntegration
where
Context: HasField<symbol!("http_client"), Value = Client>,
Context: HasField<symbol!("symbols_active_trade"), Value = RwLock<Vec<String>>>,
Context: HasField<symbol!("symbols_active_ohlcv"), Value = RwLock<Vec<String>>>,
Context: HasField<symbol!("symbol_cache"), Value = RwLock<Option<(HashSet<String>, Instant)>>>,
Context: HasField<symbol!("trade_handlers"), Value = RwLock<HashMap<String, JoinHandle<()>>>>,
Context: HasField<symbol!("ohlcv_handlers"), Value = RwLock<HashMap<String, JoinHandle<()>>>>,
{
fn http_client(context: &Context) -> &Client where <Context as Deref>::Target: Deref{
context.get_field(PhantomData::<Client>)
}
fn symbols_active_trade(context: &Context) -> &RwLock<Vec<String>> {
context.get_field(PhantomData::<RwLock<Vec<String>>>)
}
....However, then I get 99 errors that Deref is not implemented... I mean, all I want is to access an simple field, so why is this is so hard? |
Beta Was this translation helpful? Give feedback.
-
|
It looks like you have confused the #[derive(Clone)]
pub struct ImsBinanceDataContext {
pub fields: Arc<ImsBinanceDataContextFields>,
}
impl Deref for ImsBinanceDataContext {
type Target = ImsBinanceDataContextFields;
fn deref(&self) -> &ImsBinanceDataContextFields {
&self.fields
}
}
pub struct ImsBinanceDataContextComponents;
impl HasComponents for ImsBinanceDataContext {
type Components = ImsBinanceDataContextComponents;
}
delegate_components! {
ImsBinanceDataContextComponents {
BinanceIntegrationFieldsComponent: UseImsBinanceDataIntegration,
}
}Unfortunately since the CGP concepts are not native to Rust, we don't have separate namespaces to avoid mixing up the use of component names, providers, and contexts. So it can be easy to mix them up and use them at the wrong position if you are not too familiar with the separation. |
Beta Was this translation helpful? Give feedback.
-
|
Also, the method impl<Context> BinanceIntegrationFieldsProvider<Context> for UseImsBinanceDataIntegration
where
Context: HasField<symbol!("http_client"), Value = Client>,
Context: HasField<symbol!("symbols_active_trade"), Value = RwLock<Vec<String>>>,
Context: HasField<symbol!("symbols_active_ohlcv"), Value = RwLock<Vec<String>>>,
Context: HasField<symbol!("symbol_cache"), Value = RwLock<Option<(HashSet<String>, Instant)>>>,
Context: HasField<symbol!("trade_handlers"), Value = RwLock<HashMap<String, JoinHandle<()>>>>,
Context: HasField<symbol!("ohlcv_handlers"), Value = RwLock<HashMap<String, JoinHandle<()>>>>,
{
fn http_client(context: &Context) -> &Client {
context.get_field(PhantomData::<symbol!("http_client")>)
}
fn symbols_active_trade(context: &Context) -> &RwLock<Vec<String>> {
context.get_field(PhantomData::<symbol!("symbols_active_ohlcv")>)
}
}Hopefully I can get some time to add the chapter for |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately in this case, Rust is also giving the wrong error diagnostic for |
Beta Was this translation helpful? Give feedback.
-
|
Okay, thank you so much for the detailed explanation.
Let me work through all this and get back to you tomorrow. It's quite a
steep learning curve, tbh, but your explanation makes sense. It's still
mind boggling to think through the higher order effects of a generalized
context.
…On Sun, Jan 5, 2025 at 18:28 Soares Chen ***@***.***> wrote:
Unfortunately in this case, Rust is also giving the wrong error diagnostic
for HasField, due to it misunderstanding the use of the blanket
implementation for Deref. I will have to check whether I can add a
#[diagnostics::on_unimplemented] attribute for HasField to improve its
error message.
—
Reply to this email directly, view it on GitHub
<#3 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFYR7XGMFSY5X3TKFVB5HF32JECMRAVCNFSM6AAAAABUKF23WSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNZTHA2DSMY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
-
|
No problem. Thanks to your questions and feedback, I get to know better on the perspective of the learners of the challenges and common mistakes that they face. Hopefully I can improve the documentation later to make things less confusing. But yeah, the learning curve for CGP can be steep, since it makes use of many advanced concepts from already advanced topics like generic programming, functional programming, typeclasses and type-level programming, and add even more concepts on top. But hopefully by learning CGP, you can also indirectly learn about these other advanced topics and improve your overall programming skills. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, so I added the ImsBinanceDataContext with a Deref impl in a seperate file called context: impl HasComponents for ImsBinanceDataContext {
type Components = ImsBinanceDataContextComponents;
}
#[derive(Clone)]
pub struct ImsBinanceDataContext {
pub fields: Arc<ImsBinanceDataContextFields>,
}
impl ImsBinanceDataContext {
pub fn new() -> Self {
Self {
fields: Arc::new(ImsBinanceDataContextFields::new()),
}
}
}
impl Deref for ImsBinanceDataContext {
type Target = ImsBinanceDataContextFields;
fn deref(&self) -> &ImsBinanceDataContextFields {
&self.fields
}
}
#[derive(HasField, Default)]
pub struct ImsBinanceDataContextFields {
http_client: Client,
symbols_active_trade: RwLock<Vec<String>>,
symbols_active_ohlcv: RwLock<Vec<String>>,
symbol_cache: RwLock<Option<(HashSet<String>, Instant)>>,
trade_handlers: RwLock<HashMap<String, JoinHandle<()>>>,
ohlcv_handlers: RwLock<HashMap<String, JoinHandle<()>>>,
}
impl ImsBinanceDataContextFields {
pub fn new() -> Self {
Self {
http_client: Client::new(),
symbols_active_trade: RwLock::new(Vec::with_capacity(50)),
symbols_active_ohlcv: RwLock::new(Vec::with_capacity(50)),
symbol_cache: RwLock::new(None),
trade_handlers: RwLock::new(HashMap::with_capacity(50)),
ohlcv_handlers: RwLock::new(HashMap::with_capacity(50)),
}
}
}And then updated the getter as you said: #[cgp_component {
name: BinanceIntegrationFieldsComponent,
provider: BinanceIntegrationFieldsProvider,
}]
pub trait HasBinanceIntegrationFields {
fn http_client(&self) -> &Client;
fn symbols_active_trade(&self) -> &RwLock<Vec<String>>;
fn symbols_active_ohlcv(&self) -> &RwLock<Vec<String>>;
fn symbol_cache(&self) -> &RwLock<Option<(HashSet<String>, Instant)>>;
fn trade_handlers(&self) -> &RwLock<HashMap<String, JoinHandle<()>>>;
fn ohlcv_handlers(&self) -> &RwLock<HashMap<String, JoinHandle<()>>>;
}
impl<Context> BinanceIntegrationFieldsProvider<Context> for UseImsBinanceDataIntegration
where
Context: Deref,
Context: HasField<symbol!("http_client"), Value = Client>,
Context: HasField<symbol!("symbols_active_trade"), Value = RwLock<Vec<String>>>,
Context: HasField<symbol!("symbols_active_ohlcv"), Value = RwLock<Vec<String>>>,
Context: HasField<symbol!("symbol_cache"), Value = RwLock<Option<(HashSet<String>, Instant)>>>,
Context: HasField<symbol!("trade_handlers"), Value = RwLock<HashMap<String, JoinHandle<()>>>>,
Context: HasField<symbol!("ohlcv_handlers"), Value = RwLock<HashMap<String, JoinHandle<()>>>>,
{
fn http_client(context: &Context) -> &Client {
context.get_field(PhantomData::<symbol!("http_client")>)
}
fn symbols_active_trade(context: &Context) -> &RwLock<Vec<String>> {
context.get_field(PhantomData::<symbol!("symbols_active_trade")>)
}
....I assume, the delegation to the BinanceIntegrationFieldsComponent has to happen within the binance_core_integration crate, so I placed it into the lib.rs like so: #[derive(Clone)]
pub struct UseImsBinanceDataIntegration {}
pub struct ImsBinanceDataContextComponents;
delegate_components! {
ImsBinanceDataContextComponents {
BinanceIntegrationFieldsComponent: UseImsBinanceDataIntegration,
}
}However, when I compile that, I get an error that the proc macro wiring the spot integration together has paniced: The binance_spot_data_integration is the same as before: |
Beta Was this translation helpful? Give feedback.
-
|
It looks like you have a typo of delegate_components! {
ImsBinanceSpotDataIntegrationComponents {
ApiUrlComponent: UseBinanceSpotMainnetUrl,
ErrorTypeComponent: UseImsDataIntegrationError,
[
BinanceIntegrationFieldsComponent,
SymbolFetchComponent,
SymbolValidatorComponent,
OhlcvDataStreamComponent,
TradeDataStreamComponent,
SymbolTypeComponent,
TimeResolutionTypeComponent,
]: UseImsBinanceDataIntegration,
}
} |
Beta Was this translation helpful? Give feedback.
-
|
I'll probably have to rewrite the proc macros at some point in time with better error message shown on syntax errors. Currently they are more of at a quick hack MVP level just to get the minimal working, so that I could continue on other work. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the proc macros definitely need improvements b/c at this stage, almost all errors I get are so misleading that it is impossible to diagnose anything let alone derive a fix so I am basically stuck in the mud even after fixing the typo... |
Beta Was this translation helpful? Give feedback.
-
|
As for the learning, the only feedback I have is that the topic about fields and getters in a context needs to be covered way earlier in the book. Associated types, error handling, and debugging really can wait, but field access needs to be part of the first few chapters so that people get an idea what they are getting themselves into. This would have helped me a lot and saved a lot of time. |
Beta Was this translation helpful? Give feedback.
-
|
That's a good point. I have just published a new chapter for that here: https://patterns.contextgeneric.dev/field-accessors.html |
Beta Was this translation helpful? Give feedback.
-
|
I went back to the drawing board yesterday, sketching out the exact problem I am solving. Here is the high level summary:
The enum_dispatch crate did worked on 1 -3, but failed on 4 & 5 because, by its very design, it introduces something akin to a central super crate on which all binaries depend and, worse, that rebuilds with each added integration thus triggering a full rebuild of all other binaries even though none is affected by the underlying change,. From the exploration of CGP, I learned a few thing important things:
That said, the website clearly stated that CGP is early material and probably only suitable for hackers and early contributors, and I fully agree with the assessment. I knew that I am touching something hot out of the kitchen sink and that is okay. No objection here. You only know what something is when you use it and see what it does. The requirements I have thrown at CGP are actually on the lower end of the complexity in my project so that gives some perspective. I have also learned a lot, for example:
What needs improvements:
However, at this point, I had to make the hard decision to cut my losses, stop working with CGP, and move on as its current state does not get me anywhere near the basic requirements let alone the absurd time I spent for basically no outcome. I am no stranger to, say, alternative usage of the type system; when I implemented the deep_causality crate, I stretched Rust generics and type extensions to the upper limit and from that experience, I see that context generics has a place, but to get there I think it needs to be simpler to use. In any case, please continue the work and keep improving CGP b/c at a later stage, CGP may aces those tasks I work on with ease and shows its actual capabilities in plain sight. |
Beta Was this translation helpful? Give feedback.
-
|
No problem. I probably should have warned you beforehand to start small and not try to migrate large code bases to use CGP all at once. We are currently still in very early stage of development, so we definitely still need a bit more time to improve the development experience to make CGP more usable for newcomers.
Now that I think about it, I probably should have suggested you to put all fields in a common struct, and define a single accessor to get the whole struct as a field. Something like: pub trait HasImsBinanceDataFields {
fn ims_binance_data_fields(&self) -> &ImsBinanceDataFields;
}
impl<Context> HasImsBinanceDataFields for Context
where
Context: HasField(symbol!("ims_binance_data_fields"), Value = ImsBinanceDataFields>,
{
fn ims_binance_data_fields(&self) -> &ImsBinanceDataFields {
self.get_field(PhantomData)
}
}All the remaining CGP field patterns are more for later advanced stages, that you could make use of by the time you run into the need to break down Moving forward, I'll try to put clearer disclaimers to help set people's expections on using CGP at the current stage. Here are some of the tips:
I hope you will still find time to explore CGP on your own with smaller projects, when you have the free time! |
Beta Was this translation helpful? Give feedback.
-
|
All good. As I said, Im quite fond of the component model and I think one or two releases later, those issues around fields and debugging might be already be resolved. I'm certainly preserving the CGP branch I've worked on to see if the next version of CGP works out. I actually have some requirements in thr next milestone where CGP would greatly help to facilitate modular composition, but that's probably due in about the second half of the year. The hard truth is, whenever you try something new, it never works on the first try. However a few iterations later might already work out. Lastly, I want to express my sincere gratitude for all your dedication, help and support to get me started. I hope that CGP keeps evolving so that I can finally lift it into my code base later this year. |
Beta Was this translation helpful? Give feedback.
-
|
@marvin-hansen I have just published a new version of Based on your feedback, I have also significantly revised the chapter for Field Accessors, so that the easy approaches are shown first, and with updated guides to use the new macros. So I'd also like to thank you again for willing to try CGP at this early stage, and provide me valuable feedback on how to make CGP more approachable for newcomers. |
Beta Was this translation helpful? Give feedback.
-
Moving the discussion from: contextgeneric/cgp#44 (comment)
@marvin-hansen wrote:
Great to hear about your interest in considering about the use of CGP for static dispatch in place of
enum_dispatch. I can share a bit about the current status of CGP for this kind of use case.You mentioned that your use case would involve integration of your application with up to 50 data sources. Whether CGP would be useful depends on how many of the data sources would be used for specific use cases during runtime. CGP excels in compile-time wiring. So for example, if a specific program run only requires the use of 5 our of the 50 data sources, then using CGP to selectively implement the wiring could provide significant improvement in the runtime performance.
Note that, however, this would require Rust code to be written for each specific wiring, and have the code recompiled. So CGP would not be that useful in the case if you want to ship a single executable, and allow end users to choose from which of the 50 data sources to use at runtime based on a configuration file.
As for dynamic dispatch, I have some plans on how to enable composition of sub-applications at compile time, while still allowing some form of dynamic dispatch. The main advantage is that the composition would be type-safe across multiple application-specific types. Let's say each of your data source provides different
InputandOutputtype, a CGP-style composition would ensure that a data source handler cannot accept anInputvalue coming from different data source, or produce anOutputvalue that belongs to a different data source. However, this pattern is still in development, and require more advanced CGP constructs to be implemented. So it is probably not yet ready for you if that is what you are looking for.But perhaps I just misunderstood your use case, and what you are looking for could be solved in much simpler way. Feel free to share a link your code here, if it is publicly available, so that we can better understand the problem you want to solve.
Beta Was this translation helpful? Give feedback.
All reactions