Stirring the Pot with my First Principles

Today I thought I’d cause a little controversy by comparing my First Principles (things I know to be true) related to building high performance, scalable systems (software, compute, storage and networking) with current conventional wisdom.

Among most architects today, current conventional wisdom states that your architecture must follow a microservice software pattern, use containers like Docker, capture data in an event streaming platform like Kafka, persist data in a NoSQL database, be managed by something like Kubernetes or Mesosphere and run on Linux if you want to have a high performance, scalable system.

So what is something I know to be true from the dot com era?

Back then I was fortunate enough to be part of a team that built an energy trading platform that allowed multiple counterparties to buy and sell financial instruments (think NASDAQ). This platform was built on 32-bit Windows Server 2000 with some kind of Intel Pentium CPU. Classic Active Server Pages would send and receive XML fragments over HTTPS to and from brokers from every major energy company + NYMEX and the Intercontinental Exchange (ICE) following a RESTful API pattern. Incoming data was queued in MSMQ and free-threaded objects pooled inside Microsoft Transaction Server (MTS) moved that data in and out of 32-bit SQL Server 2000. The databases were clustered and Windows Server got 1 GB of RAM whereas SQL Server got most of the remaining 3 GB of RAM. The Internet Information Servers running ASP 3.0 used the built-in network load balancing service (NLB). When we needed to update the software, we just took the servers out of the cluster one by one to make the update. No biggie.

So what’s the takeaway from all this? As someone who has played a prominent role in the Internet of Things space over the years, I struggle to find modern systems that have to deal with the transactional and analytical load that I witnessed with the system I helped build 20 years ago. In spite of all the promise and talk, I’ve yet to see any IoT system that deals with a fraction of the load the world’s financial systems handle effortlessly with their antiquated architectures and relational databases.

Why were we able to do so much more with so much less back then?

Remember folks, we’re just flowing current through a gate to establish a high or low voltage at a particular point in the circuit. The farther away you abstract yourself from this, the more resources your system will require and the slower the system gets. Don’t be impressed by architectural diagrams with hundreds of lines, boxes, arrows and triangles going every which way. Complexity kills. New programming languages, frameworks, and architectural patterns come along all the time. Use you best judgement and fall back to your own first principles before deciding to jump on the next bandwagon.

Improve Employee Productivity by Delivering Web Scale to your Mobile Workforce

Book Cover

Bring #web scale to backend business systems via adapters, replication, sharding, queuing & caching of #data to support a growing #mobile workforce.

Many of the backend systems running global business over the last several decades have something in common besides not being able to communicate with mobile devices. They’re not designed to support the scale needed to empower today’s mobile workforces. Actually, most of these packages and associated databases were designed for departmental or workgroup use in a world where everyone had a single PC on their desk. The IT infrastructures of most companies are wholly unprepared to support mobility.

Whether you’re wrapping your systems in REST APIs or using a mobile middleware package, you have the ability to boost the scale of those old systems using several clever techniques. For starters, the servers used to create a new mobile API tier should be designed to cache frequently-used data so repeated calls to the backend servers won’t be needed. This could range from a simple file cache, to using staging tables in a database. To scale-out this mobile API tier, data could be replicated across multiple nodes resulting in a sharding architecture that further spreads the load generated by mobile devices making requests. To further de-couple the system, data uploads from mobile apps could be dropped in queues and processed by background processes. All these things will have the effect of boosting both the performance and scalability of your existing backend systems.

Improve User Productivity and reduce expenses by increasing the number of employees and customers served by existing line of business systems. What is your organization doing to beef up its systems to serve more users and devices?

Learn how to digitally transform your company in my newest book, “Mobile Strategies for Business: 50 Actionable Insights to Digitally Transform your Business.”

Book Cover

Click to purchase a copy of my book today and start transforming your business!