How to rate-limit a call/delay call requests?

[oco-adam]

Hi, I'm trying to crawl a Shopify ecommerce site, so I need to rate limit the crawl so that there's a delay of 0.5s between requests.

So I'm using with_delay(500) in the website builder to try and achieve this.

Also, the Shopify site has thousands of pages so I'm subscribing to results.

However, it doesn't seem like the delay is being respected inconsistently. In particular it seems like the delay is respected for the first few pages of the crawl when I crawl with website.crawl().await;, and not at all when I crawl with website.crawl_sitemap().await;...

Any help with this would be much appreciated: my code looks like this

    let mut website = Website::new(&url)
        .with_delay(500)
        .with_crawl_id(crawl_id)
        .with_retry(100)
        .build()
        .unwrap();
        
    let mut rx = website.subscribe(100000).unwrap();
    tokio::spawn(async move {
        while let Ok(res) = rx.recv().await {
           let url = res.get_url();
           // parse html etc...
           ...
        }
    });

    website.crawl().await;
    // website.crawl_sitemap().await;
    website.unsubscribe();