We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from July 3rd, 2023.
On this week's show, Adam Leventhal posed questions from Hacker News (mostly) to Oxide founders Bryan Cantrill and Steve Tuck.
They were also joined by Steve Klabnik.
Hey, you--yes you! We could use your help with these notes. There are two sections below: one for Q&A with Steve and Bryan, and one for Q&A with the Oxide team from the live chat. You could help us out a ton by filling in the *responses from Steve and Bryan and by going through the chat archive.
Questions for Steve and Bryan:
[@6:38] Q:
Congrats to the team, but after hearing about Oxide for literal years since the beginning of the company and repeatedly reading different iterations of their landing page, I still don't know what their product actually is. It's a hypervisor host? Maybe? So I can host VMs on it? And a network switch? So I can....switch stuff? (*)
A:
Steve: A rack-scale computer; "A product that allows the rest of the market that runs on-premises IT access to cloud computing."
Bryan: agrees
[@8:46] Q:
It's like an on prem AWS for devs. I don't understand the use case but the hardware is cool. (*)
I didn’t understand the business opportunity of Oxide at all. Didn’t make sense to me.However if they’re aiming at the companies parachuting out of the cloud back to data centers and on prem then it makes a lot of sense.It’s possible that the price comparison is not with comparable computing devices, but simply with the 9 cents per gigabyte egress fee from major clouds. (*)
A:
Bryan: "Elastic infrastructure is great and shouldn't be cloistered to the public cloud"; Good reasons to run on-prem: compliance, security, risk management, latency, economics; "Once you get to a certain size, it really makes sense to own"
Steve: As more things move onto the internet, need for on-prem is going to grow; you should have the freedom to own
[@13:31] Q:
Somebody help me understand the business value. All the tech is cool but I don't get the business model, it seems deeply impractical.
- You buy your own servers instead of renting, which is what most people are doing now. They argue there's a case for this, but it seems like a shrinking market. Everything has gone cloud.
- Even if there are lots of people who want to leave the cloud, all their data is there. That's how they get you -- it costs nothing to bring data in and a lot to transfer it out. So high cost to switch.
- AWS and others provide tons of other services in their clouds, which if you depend on you'll have to build out on top of Oxide. So even higher cost to switch.
- Even though you bought your own servers, you still have to run everything inside VMs, which introduce the sort of issues you would hope to avoid by buying your own servers! Why is this? Because they're building everything on Illumos (Solaris) which is for all practical purposes is dead outside Oxide and delivering questionable value here.
- Based on blogs/twitter/mastodon they have put a lot of effort into perfecting these weird EE side quests, but they're not making real new hardware (no new CPU, no new fabric, etc). I am skeptical any customers will notice or care and would have not noticed had they used off the shelf hardware/power setups.
So you have to be this ultra-bizarre customer, somebody who wants their own servers, but doesn't mind VMs, doesn't need to migrate out of the cloud but wants this instead of whatever hardware they manage themselves now, who will buy a rack at a time, who doesn't need any custom hardware, and is willing to put up with whatever off-the-beaten path difficulties are going to occur because of the custom stuff they've done that's AFAICT is very low value for the customer. Who is this? Even the poster child for needing on prem, the CIA is on AWS now.
I don't get it, it just seems like a bunch of geeks playing with VC money?
(*)
A:
Bryan: "EE side quests" rant; you can't build robust, elastic infrastructure on commodity hardware at scale; "The minimum viable product is really, really big"; Example: monitoring fan power draw, tweaking reference desgins doesn't cut it Example: eliminating redundant AC power supplies
Steve: "Feels like I’m dealing with my divorced parents" post
[@32:24] Q (Chat):
It would be nice to see what this thing is like before having to write a big check
Steve: We are striving to have lab infrastructure available for test drives
[@32:56] Q (Chat):
I want to know about shipping insurance, logistics, who does the install, ...
Bryan: "Next week we'll be joined by the operations team" we want to have an indepth conversation about those topics
[@34:40] Q:
Seems like Oxide is aiming to be the Apple of the enterprise hardware (which isn't too surprising given the background of the people involved - Sun used to be something like that as were other fully-integrated providers, though granted that Sun didn't write Unix from scratch). Almost like coming to a full circle from the days where the hardware and the software was all done in an integrated fashion before Linux turned-up and started to run on your toaster. (*)
A:
Bryan: We find things to emulate in both Apple and Sun, e.g., integrated hard- and software; AS/400
Steve: "It's not hardware and software together for integration sake", it's required to deliver what the customer wants; "You can't control that experience when you only do half the equation"
[@42:38] Q:
I truly and honestly hope you succeed. I know for certain that the market for on-prem will remain large for certain sectors for the foreseeable future. However. The kind of customer who spends this type of money can be conservative. They already have to go with on an unknown vendor, and rely on unknown hardware. Then they end up with a hypervisor virtually no one else in the same market segment uses.
Would you say that KVM or ESXi would be an easier or harder sell here?
Innovation budget can be a useful concept. And I'm afraid it's being stretched a lot. (*)
A:
Bryan: We can deliver more value with our own hypervisor; we've had a lot of experience in that domain from Joyent. There are a lot of reasons that VMware et al. are not popular with their own customers; Intel vs. AMD
Steve: "We think it's super important that we're very transparent with what we're building"
[@56:05] Q:
what is the interface I get when I turn this $$$ computer on? What is the zero to first value when I buy this hardware? (*)
A:
Steve: "You roll the rack in, you have to give it power, and you have give it networking [...] and you are then off on starting the software experience"; Large pool of infrastructure reosources for customers/devs/SREs/... in a day or less; Similar experience to public cloud providers
[@01:02:06] Q:
One of my concerns when buying a complete solution like an iPhone (or an Oxide rack 😉) is how much ownership I actually have over the device. Where does Oxide draw the line on "ownership" of the rack? (*)
A:
Bryan: "You bought the computer, it belongs to you"; it is important to us to enable that ownership
Steve: operator level and developer level; "Anything that we are injected ourselves into is [...] in the interest of making sure that the customer is getting a highly available and efficient system"
[@01:05:03] Q (Chat):
Do you sell support contracts?
A:
Steve: Yes, comes with a support subscription (typically three years, flexible)
[@01:05:37] Q (Chat):
Is the rack dependent on any Oxide cloud services?
A:
Bryan: "Emphatically not" The customer has total agency over data I/O
Adam: Updates will be fetches from a web server
Interlude
[@01:07:49]
This is super cool. I realize a lot of HN folks might not see the point of this, but it literally saves an entire team of people for companies. (*)
[@01:08:00]
Oxide is such an ambitious project, I am such a fan of the aesthetic and design and of course transparency of all of the cool people that work there. (*)
[@01:09:44]
FYI for language pedants like me: It’s “on-premises” or “on-prem”. A “premise” is something assumed to be true. (*)
[@01:11:43] Q (Chat):
When is oxide gonna make a smaller rack?
A:
Bryan: Probably no product for the homelab; "Architecturally, there is nothing here that won't scale down'; ear to the market
Steve: can be scaled down from a power perspective
[@01:15:38] Bryan discusses how to deliver "Oxide value" with a GPU product.
[@01:21:53] Q:
Are you shipping Milan racks and if so what does the moving to Genoa entail for Oxide? (*)
The listed items are for last gen CPUs and 2 gen old nics (100 Gb/s). Also zfs rather than ceph. It's hard to consider under those constraints. Any plans for upgrades beyond single rack deployments and newer gen hardware? And what is the sell case over kube, mesos, or VMware, especially since the hardware and software appear tightly coupled? (*)
A:
Bryan: Keeping a close ear to the market; "You will see a Genoa-based product"
Steve: You can add next-gen sleds to the rack, you have modularity
[@01:26:10] Q:
Do you have any liquid cooling solutions collaborators? (*)
A:
Steve: Collaboration with Motivair; The product is conventionally cooled right now. Looking at what makes sense in future architectures
Bryan: Rack power budget is a challenge
[@01:28:41] Q:
Why do you use Illumos? (*)
A:
Bryan: "We need to be able to control our fate"; dtrace;
Adam: "It was our fastest path to market (both development and support) with our team, expertise, and experience."
[@01:41:20] Q:
Congrats! This seems to be a cool product! How does this work with software updates? Are they subscription-based, or are they for free forever? In case the company goes bankrupt, will the software still work, and is the source code open? (*)
A:
Bryan: "In many ways we viewed the ability to upgrade as the most important feature" The software is open by default, and we're reviewing instances where we're constrained by IP we don't own.
[@01:46:15] Q:
What part of this journey was hardest? E.g., the scope increase when you couldn’t use reference designs? 😉 Dealing with the supply chain? Raising money for this kind of startup? (*)
A:
Bryan: "when we got funding" When you have a blank sheet of paper, where do you start?
Steve: SVB collapse and associated banking crisis
In addition, there was a live chat during the show where members of the Oxide team answered additional questions and in greater depth.
- PRs needed!
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!