In the beginning was the database. At least that is what the history of modern software looks like from a certain point of view. Databases allowed us to take data that was too big to handle in individual files and build amazing things.
“Big Data” (does anyone still use that term?) starts where databases end. Billions of rows? Trillions of rows? You need something else. In this case “something else” rapidly turns into a proliferation of … stuff. Column store? Pipeline? Streaming data? NoSql? Data warehouse? Data lake? Data mesh? Data fabric? Once data got too big for our one-stop-shop databases life got complicated. Today most organizations live with multiple, partial solutions that attempt to solve slices of this problem with limited or no insight across the entire landscape.
Before commercial databases organizations needed to build their own solutions. This is where we find ourselves again. What would a solution look like?
Scalable: Loading and querying data should be constant time as data grows. Costs should grow no worse than linearly.
Fast: Queries should return in human-friendly time (seconds). Loading and processing of data should happen within the work day.
Fungible: Mistakes will be made. Systems need to be easy to create and destroy. This is necessary for replication, staging and deployment.
Straightforward: The tools should expose and manage the data and not introduce a lot of new complexity. If you are debugging performance issues in query plans something has gone wrong.
This is minusonedb. Our platform automates operations allowing infrastructure to be deployed, rescaled and terminated. Data is stored in organized, compressed files. We use search to index data on elastic, fungible servers. Search is fast and scales horizontally. Almost all tradeoffs between space and time have been made in advance. Features that are frequently needed are built into the platform. We provide some common connectors to load your data and will build new ones as needed. We take you from events or files to analysis.
The development of computers, software and the Internet over the last several decades has unlocked a revolution in data collection and our ability to process that data. People all over the world now have unprecendented access to data to understand phenomena from medicine to astrophysics to music discovery. Our team are not experts in any of those fields; what we do is build great tools for processing data. It is our hope that minusonedb will enable smart people to make the world better for all of us. We look forward to working with you!