Skip to content

Commit

Permalink
fix: flowctl catalog does need pagination after all
Browse files Browse the repository at this point in the history
Was having a weird issue where `flowctl catalog pull-specs --captures=true --collections=true` wasn't returning any captures, but `flowctl catalog pull-specs --captures=true --collections=false` was, and realized that the first command was returning exactly 1000 specs. Turns out that even though we're paginating the inputs, we still need to paginate the queries themselves.
  • Loading branch information
jshearer committed Sep 13, 2024
1 parent b174a36 commit 4629768
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions crates/flowctl/src/catalog/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,13 @@ mod test;
use crate::{
api_exec, api_exec_paginated, controlplane,
output::{to_table_row, CliOutput, JsonCell},
pagination::into_items,
};
use anyhow::Context;
use futures::stream::{FuturesUnordered, StreamExt};
use futures::{
stream::{FuturesUnordered, StreamExt},
TryStreamExt,
};
use itertools::Itertools;
use models::{CatalogType, RawValue};
use serde::{Deserialize, Serialize};
Expand Down Expand Up @@ -260,7 +264,7 @@ where
let builder = builder.clone().in_("catalog_name", batch);
async move {
// No need for pagination because we're paginating the inputs.
api_exec::<Vec<T>>(builder).await
into_items::<T>(builder).try_collect::<Vec<T>>().await
}
})
.collect::<FuturesUnordered<_>>();
Expand Down

0 comments on commit 4629768

Please sign in to comment.