Mutually verifying dualism
This post is part of my collaborative research with Shinsei Bank on highly-evolvable enterprise architectures. It is licensed under the Creative Commons Attribution-ShareAlike 3.0 license. I am indebted to Jay Dvivedi and his team at Shinsei Bank for sharing with me the ideas developed here. All errors are my own.
For more than a year, Jay and I have been carrying on a dialogue about his methodology for developing enterprise software. This methodology appears to depart in many ways, sometimes radically, from traditional approaches. As a first step toward characterizing this methodology, I’m going to begin by writing a series of blog posts about the guiding principles that Jay has described in our conversations.
These principles are not necessarily mutually exclusive or collectively exhaustive, but they capture the essence of the methodology. At this point, I don’t fully understand how these principles fit together, and Jay has encouraged me to focus on thoroughly understanding each of principle in isolation before attempting to assemble the principles. Consequently, these blog entries may seem fragmentary or disconnected. As the research progresses, I hope to integrate and synthesize these principles into a coherent set of theoretically grounded design rules. Let’s dive into a principle.
Mutually verifying dualism: Model all operations as pairs of reciprocal actions between two agents in autonomous, reciprocal roles.
In this context, the term autonomous means that neither agent can dictate the behavior of the other; the two agents belong to separate control hierarchies and maintain their own records. Transactions provide a simple example, since they naturally lend themselves to dualism. Without dualism, we might model a transfer of a security from Alfred to Bernard as a unitary transfer operation recorded in a centralized transfer log. Such centralization may be convenient from a design perspective, but it has several drawbacks. First, an agent that controls the transfer functionality can perform transfers without consulting Alfred, Bernard, or any other user. Second, fraudulent or erroneous transactions may be impossible to detect, because the centralized transfer log is the sole source of transfer information and cannot be cross-checked. Third, transfer records are lumped together in a single log, so retrieving transfers performed by a particular user requires extracting them from a large database that may contain billions of records for millions of users. (The magic of modern databases makes this extraction of needles from a haystack possible and even straightforward, but it seems like a lot of infrastructure for an essentially simple task. As I’ll discuss in a future post, Shinsei’s philosophy is not to drop the needles into the haystack in the first place.)
Alternatively, following the principle of mutually verifying dualism, we can break the transfer into a pair of reciprocal operations between Alfred, in the role of provider, and Bernard, in the role of recipient. The transfer occurs only if both Alfred and Bernard cooperate, and the transfer can be verified by comparing Alfred’s record of the transfer with Bernard’s and ensuring that the records match.
Mutually verifying dualism requires that all operations be modeled in this way, entailing a shift in the way we conceptualize many operations. For example, consider Carl logging in to a savings account online. This operation would traditionally be modeled by the system as a unitary login event and recorded in a centralized log. To apply the mutually verifying dualism principle, we break the login operation into a pair of reciprocal operations between Carl and Carl’s Savings Account (assuming, for the moment, that he is logging in to his own account). Carl, modeled in the system as a user with its own identity and records, requests access to Carl’s Savings Account and records the result. Inversely, Carl’s Savings Account, similarly modeled as an independent entity with its own identity and records, grants (or denies) access to Carl and records the result. Using this approach, fraudulent or erroneous logins can be detected (sometimes) by reconciling the user and account records.
Shinsei visualizes its network of mutually verifying operations as a geodesic sphere. Image by Theon via Wikimedia Commons, used under Creative Commons license.
In principle, mutually verifying dualism resembles double-entry bookkeeping, which helps detect errors by breaking unitary financial flows into dual credit/debit operations. As in double-entry bookkeeping, mutually verifying dualism breaks down if a saboteur takes control of the systems on both sides of the interaction (e.g., the user system and the savings account system), or symmetric errors occur in both systems. The probability of these relatively unlikely events can be further reduced by designing a network of mutually verifying dualisms. The imagery used by Shinsei to describe the approach is a geodesic sphere, where the location of every vertice can be verified from multiple, independent perspectives.
Perhaps the most important implication of mutually verifying dualism is that records of past events can be reconstructed if any system component breaks irreparably.
Mutually verifying dualism creates some complications for system design. Since interacting agents must belong to separate control hierarchies, centralized designs are infeasible a priori. Duality implies redundancy, so changes need to be propagated to all concerned agents. Modularity probably precludes the possibility of unified or universal data models.
Hi David,
Mutually verifying dualism has to be put into context. Not all systems support this mechanism while posting entry from one to the other. for example if you take posting from core product processors like core banking /lending when they post entries into the payment gateways like zengin, swift etc. this mechanism is not supported. for control purpose there are alternative mechanisms available.
other examples are – in case we connect internet front end to core product processors, or CRM to core product processor etc. these can be done directly through messaging service or through an intermediary layer of a EAI using Jboss or Web sphere. All of them have control mechanisms built in for Straight through processing.
Secondly part is the audit trail. THis is part of all systems that are built and deployed or used. For example even if you make a change in the DB through a script all events are logged. This is true for other roles are well like a teller, or call centre agent or customer etc. If the customer calls the call centre to do a transaction it is logged in the name of the agent who preformed it and can be reconciled with the IVR logs to ensure activity was done based on call of some customer.
I am sorry but i disagree that past event can be constructed by having mutually verifying dualism. Log is the most important element which is used to reconstruct past events. for example if system goes down and transaction on an account is half done then first log is applied to trace back or forward as the case may be to bring the system / application back to the normal state before it can be used.
MA, thanks for your interesting perspectives and specific examples. Logging is definitely essential for reconstructing past events, and I may not have been clear enough about that in my post. The idea is that both participants in the interaction maintain their own logs from their own perspectives. As I understand it, these logs also serve as audit trails, so the audit trail is distributed and redundant. Either of participant’s log suffices to reconstruct past interactions.
David
Hi,
Good article, I think the mutually verifying dualism was always existed, however it was never visualized as Mr, Jay Dvivedi has taken it into design of system.
Every action that is being triggered, there always a source and there are could be multiple targets. Whenever a change in triggered in geodesic sphere, it need to settle down, so in that case, source, message, signal and sometime targets are part of change packet, which is broadcasted or distributed to have the change taken place. Keeping the track of packet at each level can help to trace it, apply mutual verifying dualism and reconstruct the past, if possible.
Unfortunately as technology evolved differently, we have been most of the time dealing with the limitation of technology, storing data and doing the number crunching used to be real issues we focused in recent decades.
Luckily we invented database some decades back based on the relational algebra and consider that is panacea of all issues, in database we have source and target at same place, so we never focused on the change packet but the agent to apply that change.
We have been so much focused on storing data that we never focused on the objects that are involved into. Later we had network that allows us some interaction between the systems. However having a communication between the systems was so fanciful thing that we all just remained focused on the protocol and their reconciliation.
I think Object orientation was the way that provides some relief from this local maximum of 4GL languages and databases and allows us to think in terms of objects; however that is still in infancy and need to evolve and widely adopted. Web services as considered today still a matter to techie to discuss not a person who designs business process.
Mutually verifying dualism to me more of a vision of higher level that you achieve and then realized how the whole world is connected and Say Wow. I had some discussion with Mr, Jay, so I can appreciate it, however in order to convert into real practice I see some problems need to be solved
1. First issue is volume of transactions and ability to trace them individually, still most of the system involves human intervention which creates this bottleneck.
2. Poor abstraction between the systems, in terms of data, object, protocol etc.
3. Humans, every system can keep a log, but humans can’t, we use agent to interact with the systems and there are very few touch point available between agents and humans, like user id and password is more than enough to agent (web browser) to act on our behalf to any site.
4. Humans again, since we can’t handle volume, we tend to aggregate things, to summarize it, which is like putting needle in the haystack. We do like that.
Regards,
Zahoor
What type of transactions require human intervention and which of these transactions are high volume?
Secondly in my experience transactions are always maintained at granular level. Could you provide details of where have you seen this phenominun.
Zahoor, thanks for your interesting perspective. I think it’s true that technological challenges have often distracted attention away from higher-level architectural questions.
David
Well, most of transactions are initiated by the humans, and where there is gap between automation for the execution of the transaction, we find humans do the the job to get the things done.
Unfortunately such gaps has become the people’s role/designations in business operations norms. which is challenging aspects to deal with.
I always see transactions a complete end to end operations, you see always find human at least on one side.
Zahoor
Hi Zahoor, let take a simple example if i transfer money from my saving account to pay off my credit card oustanding balance usig internet banking – then there is no human intervention for completion of the transaction and accounting for the transaction. so it is important to know if dualism is being talked of then for what.
It is important to provide context to the concept and then substantiate the assertion with emperical data. This will help in understanding the concept and also provide evidence that it actually works.