Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RSS handoff, Nexus doing svc management, DNS integration #1216

Closed
wants to merge 59 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
7f1087d
Pull in RSS changes from 'use-dns' branch
smklein Jun 9, 2022
eca5484
RSS performs config by itself, mostly
smklein Jun 9, 2022
565862e
RSS side of handoff to Nexus mostly complete
smklein Jun 9, 2022
dfa614b
Handoff to Nexus is hacky, but working
smklein Jun 10, 2022
dc3b84b
Add bg work user, rack insert populate, patch tests
smklein Jun 12, 2022
e265f0d
Await RSS handoff, even in tests
smklein Jun 12, 2022
b5ca139
Partway through service allocation - still very WIP
smklein Jun 13, 2022
7e986b8
v1 of nexus-managed services is code complete; no tests yet
smklein Jun 14, 2022
95a5873
Add indices, add tests, fix bugs
smklein Jun 15, 2022
2a28eb9
It's hacky, but it's working. I'm seeing services be re-balanced corr…
smklein Jun 15, 2022
248d4cb
Merge branch 'dns-client' into rss-handoff
smklein Jun 15, 2022
b07322c
clippy, fmt
smklein Jun 15, 2022
7f41e42
Strongly-typed DNS service names
smklein Jun 15, 2022
a68de33
Populate DNS records
smklein Jun 16, 2022
746114b
Fix dns client bug, start shortening timeouts
smklein Jun 16, 2022
1b019b1
clippy
smklein Jun 16, 2022
94b4b46
Concurrent provisioning
smklein Jun 16, 2022
a02e009
Dynamic oximeter config
smklein Jun 16, 2022
a5be4d0
Allow oximeter to use config-provided addresses
smklein Jun 17, 2022
59dc382
Fix command-based tests
smklein Jun 17, 2022
81bf2d4
Nexus lazily accessing timeseries DB
smklein Jun 20, 2022
aed3ba6
Cleanup TODOs
smklein Jun 20, 2022
8fce9a1
Box resolver to make clippy happy
smklein Jun 20, 2022
d26ee14
Internal DNS tests
smklein Jun 20, 2022
4b5dab7
Clean up test code
smklein Jun 20, 2022
db2b545
no retry in client library
smklein Jun 20, 2022
027fb3b
Fix internal-dns
smklein Jun 20, 2022
bccb416
Merge branch 'dns-client' into rss-handoff
smklein Jun 20, 2022
bce58f4
Merge branch 'rack-id' into rss-handoff
smklein Jun 20, 2022
089623e
Merge branch 'background-work-user' into rss-handoff
smklein Jun 20, 2022
e33fb4b
fix typos, warnings
smklein Jun 20, 2022
9a9ca35
Merge branch 'rss-set-dns' into rss-handoff
smklein Jun 20, 2022
36135b2
Merge branch 'oximeter-resolves-nexus-address' into rss-handoff
smklein Jun 20, 2022
a82b653
Merge branch 'rack-populate' into rss-handoff
smklein Jun 21, 2022
9fc4994
Cleanup imports
smklein Jun 21, 2022
11ebb7b
[nexus] Add tests for rack endpoints
smklein Jun 21, 2022
19356a8
Merge branch 'rack-populate' into rss-handoff
smklein Jun 21, 2022
e6dc594
Merge branch 'nexus-resolves-clickhouse' into rss-handoff
smklein Jun 21, 2022
1822762
[nexus] Add tunable to disable background tasks
smklein Jun 21, 2022
55feaf6
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 21, 2022
d2536d7
Delete out-dated docs
smklein Jun 21, 2022
7fb947b
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 21, 2022
6c5e035
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 21, 2022
768221d
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 21, 2022
b6434e3
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 22, 2022
c5bd827
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 22, 2022
0ff033a
renamed opctx
smklein Jun 22, 2022
4d7a46c
in tests too
smklein Jun 22, 2022
5fc64aa
merge
smklein Jun 24, 2022
286699c
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 24, 2022
eed9437
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 24, 2022
900cd38
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 24, 2022
f540d3a
Merge branch 'nexus-service-management' into rss-handoff
smklein Jun 26, 2022
8cf9ad4
Merge branch 'nexus-service-management' into rss-handoff
smklein Jul 6, 2022
1a7bb40
partial merge
smklein Jul 11, 2022
ba1a731
Merge branch 'nexus-service-management' into rss-handoff
smklein Jul 11, 2022
6047b93
remove unused
smklein Jul 11, 2022
eba4486
Finish merge
smklein Jul 11, 2022
9b07d55
Merge branch 'nexus-service-management' into rss-handoff
smklein Jul 27, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clean up test code
smklein committed Jun 20, 2022
commit 4b5dab7556167c5d3781655723bb27491376f7dc
182 changes: 77 additions & 105 deletions internal-dns-client/src/multiclient.rs
Original file line number Diff line number Diff line change
@@ -19,75 +19,37 @@ use trust_dns_resolver::TokioAsyncResolver;

type DnsError = crate::Error<crate::types::Error>;

// A structure which instructs the client APIs how to access
// DNS servers.
//
// These functions exist in a separate struct for comparison
// with the test-utility, [`LocalAddressGetter`].
struct FromReservedRackSubnet {}

const FROM_RESERVED_RACK_SUBNET: FromReservedRackSubnet =
FromReservedRackSubnet {};

impl FromReservedRackSubnet {
fn subnet_to_ips(
subnet: Ipv6Subnet<AZ_PREFIX>,
) -> impl Iterator<Item = IpAddr> {
ReservedRackSubnet::new(subnet)
.get_dns_subnets()
.into_iter()
.map(|dns_subnet| IpAddr::V6(dns_subnet.dns_address().ip()))
}

fn subnet_to_dropshot_server_addrs(
&self,
subnet: Ipv6Subnet<AZ_PREFIX>,
) -> impl Iterator<Item = SocketAddr> {
Self::subnet_to_ips(subnet)
.map(|address| SocketAddr::new(address, DNS_SERVER_PORT))
}

fn subnet_to_dns_server_addrs(
&self,
subnet: Ipv6Subnet<AZ_PREFIX>,
) -> impl Iterator<Item = SocketAddr> {
Self::subnet_to_ips(subnet)
.map(|address| SocketAddr::new(address, DNS_PORT))
}
/// Describes how to find the DNS servers.
///
/// In production code, this is nearly always [`Ipv6Subnet<AZ_PREFIX>`],
/// but it allows a point of dependency-injection for tests to supply their
/// own address lookups.
pub trait DnsAddressLookup {
fn dropshot_server_addrs(&self) -> Vec<SocketAddr>;

fn dns_server_addrs(&self) -> Vec<SocketAddr>;
}

// A test-only alternative to [`FromReservedRackSubnet`].
//
// Rather than inferring DNS server addresses from the rack subnet,
// they may be explicitly supplied. This results in easier-to-test code.
#[cfg(test)]
#[derive(Default)]
struct LocalAddressGetter {
addrs: Vec<(SocketAddr, SocketAddr)>,
fn subnet_to_ips(
subnet: Ipv6Subnet<AZ_PREFIX>,
) -> impl Iterator<Item = IpAddr> {
ReservedRackSubnet::new(subnet)
.get_dns_subnets()
.into_iter()
.map(|dns_subnet| IpAddr::V6(dns_subnet.dns_address().ip()))
}

#[cfg(test)]
impl LocalAddressGetter {
fn add_dns_server(
&mut self,
dns_address: SocketAddr,
server_address: SocketAddr,
) {
self.addrs.push((dns_address, server_address));
}

fn subnet_to_dropshot_server_addrs(
&self,
) -> impl Iterator<Item = SocketAddr> + '_ {
self.addrs
.iter()
.map(|(_dns_address, dropshot_address)| *dropshot_address)
impl DnsAddressLookup for Ipv6Subnet<AZ_PREFIX> {
fn dropshot_server_addrs(&self) -> Vec<SocketAddr> {
subnet_to_ips(*self)
.map(|address| SocketAddr::new(address, DNS_SERVER_PORT))
.collect()
}

fn subnet_to_dns_server_addrs(
&self,
) -> impl Iterator<Item = SocketAddr> + '_ {
self.addrs.iter().map(|(dns_address, _dropshot_address)| *dns_address)
fn dns_server_addrs(&self) -> Vec<SocketAddr> {
subnet_to_ips(*self)
.map(|address| SocketAddr::new(address, DNS_PORT))
.collect()
}
}

@@ -105,26 +67,14 @@ pub struct Updater {
}

impl Updater {
/// Creates a new "Updater", capable of communicating with all
/// DNS servers within the AZ.
pub fn new(subnet: Ipv6Subnet<AZ_PREFIX>, log: Logger) -> Self {
let addrs =
FROM_RESERVED_RACK_SUBNET.subnet_to_dropshot_server_addrs(subnet);
pub fn new(address_getter: &impl DnsAddressLookup, log: Logger) -> Self {
let addrs = address_getter.dropshot_server_addrs();
Self::new_from_addrs(addrs, log)
}

// Creates a new updater, using test-supplied DNS servers.
#[cfg(test)]
fn new_for_test(address_getter: &LocalAddressGetter, log: Logger) -> Self {
let dns_addrs = address_getter.subnet_to_dropshot_server_addrs();
Self::new_from_addrs(dns_addrs, log)
}

fn new_from_addrs(
addrs: impl Iterator<Item = SocketAddr>,
log: Logger,
) -> Self {
fn new_from_addrs(addrs: Vec<SocketAddr>, log: Logger) -> Self {
let clients = addrs
.into_iter()
.map(|addr| {
info!(log, "Adding DNS server: {}", addr);
crate::Client::new(&format!("http://{}", addr), log.clone())
@@ -271,28 +221,18 @@ pub struct Resolver {
}

impl Resolver {
/// Creates a DNS resolver, looking up DNS server addresses based on
/// the provided subnet.
pub fn new(subnet: Ipv6Subnet<AZ_PREFIX>) -> Result<Self, ResolveError> {
let dns_addrs =
FROM_RESERVED_RACK_SUBNET.subnet_to_dns_server_addrs(subnet);
Self::new_from_addrs(dns_addrs)
}

// Creates a new resolver, using test-supplied DNS servers.
#[cfg(test)]
fn new_for_test(
address_getter: &LocalAddressGetter,
pub fn new(
address_getter: &impl DnsAddressLookup,
) -> Result<Self, ResolveError> {
let dns_addrs = address_getter.subnet_to_dns_server_addrs();
let dns_addrs = address_getter.dns_server_addrs();
Self::new_from_addrs(dns_addrs)
}

fn new_from_addrs(
dns_addrs: impl Iterator<Item = SocketAddr>,
dns_addrs: Vec<SocketAddr>,
) -> Result<Self, ResolveError> {
let mut rc = ResolverConfig::new();
for socket_addr in dns_addrs {
for socket_addr in dns_addrs.into_iter() {
rc.add_name_server(NameServerConfig {
socket_addr,
protocol: Protocol::Udp,
@@ -312,7 +252,7 @@ impl Resolver {
pub fn new_from_ip(address: Ipv6Addr) -> Result<Self, ResolveError> {
let subnet = Ipv6Subnet::<AZ_PREFIX>::new(address);

Resolver::new(subnet)
Resolver::new(&subnet)
}

/// Looks up a single [`Ipv6Addr`] based on the SRV name.
@@ -416,6 +356,41 @@ mod test {
}
}

// A test-only way to infer DNS addresses.
//
// Rather than inferring DNS server addresses from the rack subnet,
// they may be explicitly supplied. This results in easier-to-test code.
#[derive(Default)]
struct LocalAddressGetter {
addrs: Vec<(SocketAddr, SocketAddr)>,
}

impl LocalAddressGetter {
fn add_dns_server(
&mut self,
dns_address: SocketAddr,
server_address: SocketAddr,
) {
self.addrs.push((dns_address, server_address));
}
}

impl DnsAddressLookup for LocalAddressGetter {
fn dropshot_server_addrs(&self) -> Vec<SocketAddr> {
self.addrs
.iter()
.map(|(_dns_address, dropshot_address)| *dropshot_address)
.collect()
}

fn dns_server_addrs(&self) -> Vec<SocketAddr> {
self.addrs
.iter()
.map(|(dns_address, _dropshot_address)| *dns_address)
.collect()
}
}

// The resolver cannot look up IPs before records have been inserted.
#[tokio::test]
async fn lookup_nonexistent_record_fails() {
@@ -428,7 +403,7 @@ mod test {
dns_server.dropshot_server_address(),
);

let resolver = Resolver::new_for_test(&address_getter)
let resolver = Resolver::new(&address_getter)
.expect("Error creating localhost resolver");

let err = resolver
@@ -489,10 +464,9 @@ mod test {
dns_server.dropshot_server_address(),
);

let resolver = Resolver::new_for_test(&address_getter)
let resolver = Resolver::new(&address_getter)
.expect("Error creating localhost resolver");
let updater =
Updater::new_for_test(&address_getter, logctx.log.clone());
let updater = Updater::new(&address_getter, logctx.log.clone());

let record = TestServiceRecord::new(
AAAA::Zone(Uuid::new_v4()),
@@ -527,10 +501,9 @@ mod test {
dns_server.dropshot_server_address(),
);

let resolver = Resolver::new_for_test(&address_getter)
let resolver = Resolver::new(&address_getter)
.expect("Error creating localhost resolver");
let updater =
Updater::new_for_test(&address_getter, logctx.log.clone());
let updater = Updater::new(&address_getter, logctx.log.clone());

let cockroach_addrs = [
SocketAddrV6::new(
@@ -652,10 +625,9 @@ mod test {
dns_server.dropshot_server_address(),
);

let resolver = Resolver::new_for_test(&address_getter)
let resolver = Resolver::new(&address_getter)
.expect("Error creating localhost resolver");
let updater =
Updater::new_for_test(&address_getter, logctx.log.clone());
let updater = Updater::new(&address_getter, logctx.log.clone());

// Insert a record, observe that it exists.
let mut record = TestServiceRecord::new(
2 changes: 1 addition & 1 deletion nexus/src/app/background/services.rs
Original file line number Diff line number Diff line change
@@ -92,7 +92,7 @@ pub struct ServiceBalancer {
impl ServiceBalancer {
pub fn new(log: Logger, nexus: Arc<Nexus>) -> Self {
let dns_updater = DnsUpdater::new(
nexus.az_subnet(),
&nexus.az_subnet(),
log.new(o!("component" => "DNS Updater")),
);

2 changes: 1 addition & 1 deletion nexus/src/context.rs
Original file line number Diff line number Diff line change
@@ -145,7 +145,7 @@ impl ServerContext {
Ipv6Subnet::<AZ_PREFIX>::new(config.deployment.subnet.net().ip());
info!(log, "Setting up resolver on subnet: {:?}", az_subnet);
let resolver =
internal_dns_client::multiclient::Resolver::new(az_subnet)
internal_dns_client::multiclient::Resolver::new(&az_subnet)
.map_err(|e| format!("Failed to create DNS resolver: {}", e))?;

// Set up DB pool
9 changes: 5 additions & 4 deletions sled-agent/src/rack_setup/service.rs
Original file line number Diff line number Diff line change
@@ -313,9 +313,10 @@ impl ServiceInner {
) -> Result<(), SetupServiceError> {
info!(self.log, "Handing off control to Nexus");

let resolver =
internal_dns_client::multiclient::Resolver::new(config.az_subnet())
.expect("Failed to create DNS resolver");
let resolver = internal_dns_client::multiclient::Resolver::new(
&config.az_subnet(),
)
.expect("Failed to create DNS resolver");
let ip = resolver
.lookup_ip(SRV::Service(ServiceName::Nexus))
.await
@@ -546,7 +547,7 @@ impl ServiceInner {
.collect::<Result<_, SetupServiceError>>()?;

let dns_servers = internal_dns_client::multiclient::Updater::new(
config.az_subnet(),
&config.az_subnet(),
self.log.new(o!("client" => "DNS")),
);
self.dns_servers