Operos

Today we are excited to announce the first preview release of Operos. What is Operos? Well it’s an operating system, a Linux distribution for building distributed computer systems. This might not add any clarity so let’s expand that to explain exactly what it does: Operos is software that you install in about 15 minutes and that turns one computer server into what we call a controller. Other servers are connected to the controller via a Ethernet network interface. When they are configured to network boot and powered on, they become part of one complete warehouse scale computer.

The term warehouse scale computing sounds intimidating - you might not have a warehouse full of computers so why might you need this? This term was coined by Google and prescribes a method of operating computing systems at scale. These methods give Google and the other hyperscalers serious competitive advantages:

  • Very high utilization of their computing systems. They can compute more with fewer machines. For every dollar they spend, they get more features, more new products/services and better insight into their business from data analytics.

  • Higher operator productivity. With a very high ratio of servers to admins, the hyperscalers need far fewer people to run their computing systems.

  • Higher developer productivity. Larger teams can get new and updated code into an environment serving customers faster.

Our goal in founding Pax Automa and in making Operos is to make those advantages scale invariant and ubiquitous: to level the playing field as much as possible.

Secret sauce

So what are the precepts of warehouse scale computing that deliver these advantages ?

  • Homogeneous commodity infrastructure. Machines are cattle not pets. Each cluster is composed of machines generally sharing similar hardware. As much as possible, the machines are used to handle the entire workload - including functions traditionally done by specialized appliances, such as storage and load balancing.
  • Workload co-location. If machines are not treated specially, workloads should be free to move among them. They should be packed Tetris-like onto the machines to enable the greatest utilization.
  • High level of automation. Remove humans from the critical path of most computing operations (i.e. machine provisioning, deployment, configuration, process supervision, placement/scheduling decisions, communication topology): humans make policy, not decisions.
  • Focus on observability. It’s one thing to get code running at scale quickly; it’s just as important to understand how it is behaving.
  • Fault tolerance. Hardware will fail, software will leak memory, bad patches will be deployed, backhoes will find fibre cables with laser like precision. A computing platform must be able to handle as much of this as possible, quicker than humans would be able to diagnose and react.

What makes this possible isn’t the hardware, as that is mostly a commodity anyone can purchase. Their advantage derives from the rest of their computing platforms: the software that runs their systems. The hyperscalers have hundreds or thousands of people to build their computing platforms and thats a good thing as these platforms are complicated systems.

The problem

All of this can be quite disheartning if you are outside the gates. Fortunately, the open source world has produced analogues, sometimes dozens of different choices for most of the components that comprise the hyperscaler platforms. We are, in fact, enjoying something of a Cambrian explosion in systems tooling. While this is fantastic for those of us who are interested in that, the reality is that this really hasn’t improved the situation much for the industry.

Why? As a builder of one of these computing platforms you have to integrate dozens of software components, choosing from competing offerings with differing levels of maturity, compatibility, and performance.

Most platform teams at smaller organizations are perpetually pressed for time. It is inevitable that some really desirable functionality is left out to get the system up, or alternatively isn’t tuned or configured properly. This means the system does not deliver many of the advantages above. Worse, building this system distracts the from difficult task of overhauling development processes and software architectures that would allow an organization to benefit from the properties of a hyperscaler-grade infrastructure.

The fact that integration is the biggest challenge points to the root cause of a current industry crisis: computing platforms are built in an artisanal bespoke fashion. This means that employees must learn the quirks and idiosyncrasies of each computing system. We are stuck perpetually reinventing oval wheels. For application vendors it also makes development expensive and time consuming as each application must be ported to deal with the wildly different systems, creating situations ripe for vendor lock-in.

Our solution

In this problem we saw an opportunity to create Operos. What we believe we have done with it is choose some of the best available software with open APIs, write our own to fill in the gaps and make sure that it all works well together. Most importantly, we packaged it so that distributed computing is accessible to everyone.

  • We focus on deploying Docker containers containing standard x86-64 bit Linux binaries.

  • We use the open source container orchestration system Kubernetes as the API to the system. This delivers the ideal: a declarative software deployment system. Kubernetes takes the users intention and policies and makes the scaling and placement decisions. It also enforces resource limits and supervises processes, handling failures automatically.

  • We provide software defined networking in the form of Calico so that all inter-pod communication is handled automatically. Users declare policies about communication topology and the system makes it happen.

  • We provide persistent storage volumes that can follow containers around the cluster as nodes or hardware fails. This is provided by the scalable block storage system Ceph.

All of that is integrated into the Operos Preview. We have also built components to automatically inventory nodes, give them their identity, boot a stateless copy of Operos over the network using PXE, and configure all the storage found in each system so that it is available both as scratch space and as persistent storage for containers running on the cluster.

We have much more planned, which we will get into in subsequent blog posts. But please give the preview a try, join the Operos Slack to discuss, and if you are interested in piloting at a larger scale send us an email at info@paxautoma.com.

Pax Automa