-
Notifications
You must be signed in to change notification settings - Fork 340
Hackathon FAQ
Why has Seagate developed an object storage system?
The goal of CORTX is to create a fully open-source mass-capacity object storage system that maximizes efficiency, storage capacity, and value by designing a system that can take full advantage of under-utilized advancements in storage hardware. CORTX has been designed to include features such as cooperative re-builds and multi-level erasure in order to increase data durability while reducing storage capacity overhead. In other words, CORTX is designed to make every petabyte go further and thus to do more – and better – with less.
CORTX is intended to be flexible, adaptable, and deployable across a wide variety of different setups and purposes. While CORTX has been designed by Seagate’s storage hardware experts, the benefits of CORTX are not limited to Seagate-designed hardware. CORTX aims to be fully processor-agnostic and cross-compatible with most storage types- including for scale-out.
Learn more about CORTX's core features and design goals here.
More integrations mean that CORTX can be tested under a larger variety of conditions and for different uses, which will help the project realize its goals to be flexible and agnostic. Additionally, as CORTX expands its ecosystem, more people and organizations will be able to use CORTX to maximize the efficiency of their data storage. The more efficient data storage is, the easier to is to realize the benefits of the data revolution.
A great integration will demonstrate moving, storing, and/or activating data at scale, and include all the requested submission elements, including a pull request on Github, easy to follow documentation/instructions, and a concept and demo video. The wider an integration reaches – the more useful it is across a variety of data workloads and domains- the better! Unsure what to build or looking for inspiration? Here’s a non-exhaustive list of integrations we would love to see:
| Elasticsearch | Search & Analytics |
| pyTorch | Machine Learning |
| Apache Kafka | Stream Processing |
| Apache Spark | Spark Analytics |
| IPFS | P2P File Share |
| Presto | Query Engine |
| Greenplum | Analytics |
| Restic | Backup |
| VXG | VSaaS |
| dataobi | Data Migration |
| Grafana | Dashboard |
| H2O.ai | Machine Learning |
| Traefik | Reverse Proxy / Load Balancer |
| Alexa | Voice |
| IOR | HPC Benchmark |
| Humio | Log management |
| Video Loft | VsaaS |
| Tensorflow | Machine Learning |
| Prometheus | Monitoring |
| FIO | Benchmark |
Yes, it is possible to deploy CORTX using Kubernetes! Please refer to our K8s repository for more information and deployment instructions.
Yes! CORTX is S3 compatible, so you can program to the standard S3 interface. There are many Python libraries available for S3, including boto3, which is among the most popular.
Yes! There are two ways to do so. You can use the standard S3 protocol and connect using a Go S3 client such as this one, or you can use a Go binding that connects to our lower-level Motr interface. See the question below for more details.
Yes, CORTX is S3 compatible! Currently CORTX is using the RADOS Gateway S3 layer. More details can be found in our RGW repository.
I’m interested in building an integration directly with Motr, but I’m finding the Motr interface a bit challenging. What can I do?
We’re thrilled you’re interested in integrating directly with Motr! It’s a bit more challenging but potentially more rewarding as it contains a more powerful set of functionalities including some key-value functions which we believe are key to unlocking and activating the rich potential of data.
Some resources that might be useful to help understand the Motr interface:
- The Motr Developer guide
- The Motr API specification
- An alternative higher-level interface to Motr can be found in our Maestro repo
- A collection of example Motr client applications
- An example Motr application using the key-value functions
- A slightly simpler version of the Motr interface written in Go
- Motr includes a publishing-subscribe mechanism called FDMI which can be used to create integrations which perform work when specified events happen (such as a new object being created matching a particular prefix).
- We’re excited that you want to run CORTX on bare metal. Currently, you can run Motr on bare metal, or you can run CORTX via Kubernetes on either a single node or across a cluster. You can also run CORTX via Minikube or on AWS.
- If you do decide to stand up you own instance of CORTX, whether natively or via the provided VM image, please let us know if you run into any trouble. CORTX is an ever-evolving project, meant to work across as many systems as possible, so we are very interested in hearing about and learning from your experience.
How can I create a ‘CORTX cluster’ (multiple servers or multiple VMs) to provide at-scale integrations?
Thanks for your interest! For this challenge, please note that showing how to integrate CORTX with complementary technologies can be just as clearly demonstrated using a single node CORTX as it can be with a multi-node CORTX. We are currently working on improving our documentation and support, including for more ambitious projects like running CORTX at scale, so we do not recommend attempting to set up a CORTX cluster for this challenge. However, if you’d like to build at-scale integrations, you can do so by using Motr, the underlying object store. It can be run at scale using these instructions.
First, you’ll need to set up a fresh install. The VM needs to be shut down following this procedure- shutting it down without following the procedure is a common cause of this problem. If you ran into this problem despite following the shutdown procedure, please let us know, or if this happens more than once. If you can capture any error messages you see, please provide them as it may help us diagnose the problem and help others avoid it in the future.
Sorry to hear that. Please try to reboot and your system should be up and running again (if not, you may need to start over with a fresh install). If you encounter any problems after a reboot, let us know if you see any error messages, and be sure to let us know about the the state of the workload during or just prior to the crash, e.g.: were there a lot of deletes, or was it a quiescent system, or was there a mix of read and writes of large objects (etc)? If you crashed during a large set of deletes, that might be a workload to try to avoid in the future. You also want to ensure you are following the prerequisite environment for running a VM which is at least 4 CPU cores, at least 8 GB of RAM, and at least 120 GB of local storage. You can read more about this here.
My CORTX virtual machine isn’t connecting to the internet. / All my NICs are disconnected from the internet. How can I fix this?
The first step is to enter sudo nmtui
Your next step is to click on "Edit a connection" -> Select on all connections -> Navigate to "Automatically Connect" -> Press the space bar to select this -> Select OK and Quit until you get back to your terminal.
Lastly, you will need to reboot the VM. You can do this by entering: sudo reboot now
At the terminal, enter su - root followed by your password. Make sure to use CentOS 7.7.1908 to reduce errors.
First, check the “Resources” tab of Devpost. If what you’re looking for is not there, please reference our GitHub repositories. You can access several guides from the top-level README, including an API guide, installation guide, configuration guide, and more. You can also talk to us directly in our Slack! You should receive an invitation to it, but if you’d like a direct link, you can join it by clicking here.