Author: Warwick Sweeney
While it is generally true that people find change difficult, I suspect it is less true for those of us that are technically minded. At least when it comes to technical change, for technology and the processes we use for delivering technology solutions are constantly evolving. So we shouldn’t be too put out when something challenges the status quo. Or should we?
The business benefits of a Service Oriented Architecture (SOA) for software solutions are well known and rarely questioned. Increased business process agility and flexibility, more scalable systems, reduced development and implementation costs. Who wouldn’t want one? It is also true that there are technical challenges involved with SOA and for as long as we’ve endeavoured to use SOA techniques for the delivery of software systems there have been a number of concepts that have been considered essential for doing SOA well. Things like:
- Getting service granularity right
- Well defined and managed service contracts
- A comprehensive Canonical Message (Information) Model (CMM)
- Establishment of appropriate governance mechanisms
- etc etc
It would be great if all of the things required to do SOA well were simple, but the reality is that most of them are not easy. Not easy to establish and not easy to maintain. For example consider the development of a complete CMM for even a medium sized organisation. To model every information element in a consistent way is a significant exercise. To maintain that model over time is perhaps more of an impost. To maintain service contracts that cater for the needs of all service consumers is not simple either, and as an industry we’ve generally done a poor job of managing and publishing those in a way that is useful and quick to adopt for multiple consumers. Trying to get the level of service/SOA governance right tends also to be more of an art than a science. Too much governance and nothing gets done. Too little and we end up compromising the very principles we think are necessary to make SOA work.
It seems to be the case that the very things required to do SOA well end up becoming a bit of a rod for the back. They result in substantial and real maintenance cost impacts and consequently we run the risk of making SOA adoption too difficult and too costly. Ultimately these challenges mean that the benefits of SOA that relate to flexibility, agility and reduced cost may never be realised.
So how about Microservices architecture?
It isn’t surprising then that something should challenge the long held view of the best way to do SOA. The concept of a Microservices architecture is a relative new-comer to software development, but is challenging/changing the way applications are designed and built. It seems logical that Microservices as an approach should also impact application integration and bring to service based integration the practices that in other areas of software development we’ve accepted as necessary for achieving good outcomes.
There are a number of really good descriptions of Microservices architecture including work by Martin Fowler (http://martinfowler.com/articles/microservices.html) and Chris Richardson (http://microservices.io/articles/index.html), but it is worth considering how Microservices as an approach might challenge some of the long held views on how SOA should be done well. Consider the following key Microservices concepts.
Effective componentisation by services. This is a concept we’re all pretty comfortable with and the traditional SOA principles of service granularity apply. Coarse grained is good and the main goal is to ensure the service interface is simple and efficient. Lowering the cost of API calls is important, it’s hard to argue with that.
Focus on the business need and context. Delivering technology solutions that solve a real business need is what we are all about. In some instances we’ve probably lost sight of what that really means. There is no better way to deliver to the business need than including the business in the processes from start to finish. That is about delivery using, in agile speak, a cross-functional team that incorporates all of the skills required for effective delivery including the business, rather than a pure technical team. It’s also about building business context into every aspect of the delivery thinking and process including design. Again hard to argue with.
Ownership of the outcome – Some Microservices definitions use the concept “products not projects”. The concept is effectively used to describe a different way of thinking about delivery and the final outcome, to drive product based thinking about delivery. What does that mean? That the team responsible for delivery owns the outcome even beyond completion of development. That has broad implications for team structure, operations approach, technology platform and toolsets, but it is true that giving the team ownership through into operations (as opposed to handing over at the end of the “project”) will drive a better outcome. That is true for integration as much as it is for application development.
Simplified integration layer – As integration platforms/products have become all singing and dancing, in terms of the technologies and approaches they support, a tendency to overcomplicate the integration layer has followed. The Microservices approach dictates that the integration layer is logic free and uses very simple integration mechanisms and lightweight messaging, for example REST/HTTP. Logic must be self-contained in the endpoints and not in the integration layer and there are implications for the granularity of services (again). Hopefully not too many arguments here either.
There is probably not too much that is controversial in any of those, but the next two will probably raise the blood pressure of a few.
Decentralised governance – We’re used to having centralised design and governance for SOA. It’s important for making it work right? Microservices flips that on its head and asks the question, who knows best what they want? The customer of course and that means the customer should be involved in services governance from design to operations. From a design perspective that means the customer or the consumer defines the service contract, or is at least engaged in the definition of the contract. What better way to ensure business context is considered and business needs met. There are implications though that need to be considered to ensure this works well and we don’t compromise the benefits of SOA. Implications for the service design (they must be simple and coarse grained) and the design approach. We also need to consider how we go about design. What tools and processes do we need to facilitate the design process and ensure collaboration at a level that drives the right outcome? By necessity decentralised governance demands simple designs of simple services using simple tooling. Simple?
Decentralised data – This one may challenge traditional SOA thinking most. Rather than aiming for information Nirvana (the complete CMM), Microservices takes quite a different view. Delegate ownership of the data to each service. That’s right. Service specific data and databases. Crazy. How can you ever expect to get re-use and efficiency with that model? Doesn’t that just push us back to the dreaded point-to-point integration? Perhaps, but not if we do it well and manage it properly. I suspect in many SOA implementations the CMM gets built incrementally. That might touch a nerve but the reality is it’s the only efficient way to do it. It’s near impossible to try and model everything up front, in spite of the best of intentions we quite often have no idea of the message structure until we nail the service implementation down anyway. So it’s not really a stretch to suggest that we delegate data ownership to the service, model it accordingly and if we manage it properly still realise the benefits that come from an effective service layer. The next challenge with this model is how we ensure consistency is maintained across the service specific data sets, simply. In other words how do we allow a business “transaction” to span multiple services? This becomes more of a challenge if data ownership is delegated to the extent of allowing different services to use different database types, heaven forbid even No-SQL. How do we maintain information consistency across multiple datasets in different database types? For a start it will need to be transaction-less. There are a few options for building non-transactional patterns for data consistency into our service implementations, the trick is to meet the keep it simple principle. Event driven data synchronisation is one approach that can be used, ensuring all services manage their data and publish update events for other services to consume.
Infrastructure automation – the final piece of the Microservices puzzle is ensuring provisioning and management of infrastructure and the associated deployment processes is as automated as possible. We’re all familiar with the concepts of continuous build and integration, so again not much to argue with on this one in the integration space. It is a given that Platform as a Service (PaaS) and Infrastructure as a Service (IaaS) offerings have assisted with achieving some level of automation in infrastructure provisioning. Incorporating smart use of containerised computing will also assist with service autonomy and simplifying deployment and provisioning.
So is it really a step too far to think Microservices could change the thinking about how best to do SOA? Or is it simply accepting what we know and practice as the most effective way to drive successful software development. I’d like to think the latter. Experience dictates that the only way to deliver successful technology outcomes is to keep it simple, as simple as possible. Why should it be any different in the integration space?