This post is part of my collaborative research with Shinsei Bank on highly-evolvable enterprise architectures. It is licensed under the Creative Commons Attribution-ShareAlike 3.0 license. I am indebted to Jay Dvivedi and his team at Shinsei Bank for sharing with me the ideas developed here. All errors are my own.
The systems in a large bank handle huge volumes of transactions collectively worth many millions or billions of dollars, so errors or sabotage can cause serious damage in short order. For this reason, Jay often compares running bank systems to flying jumbo jets. To make matters worse, the jets must be maintained and modified in flight. Jay’s approach, which he demonstrated to great effect when he successfully migrated Shinsei from mainframes to inexpensive servers, focuses on building up new systems one component at a time in parallel with existing systems and achieving functional parity. Once parity has been reached, the old systems can be shut down.
Parity is relatively straightforward in principle. As new components—assembly line segments—become operational, they are integrated into the line in parallel with the existing line segments being superseded. Parity is reached when line behavior does not change regardless of whether work flows along the existing line or flows from the existing line onto the new segment and then back onto the existing line.
In practice, achieving parity quickly on a high volume line turns out to be challenging. Jay’s agile development methodology means that many new line segments are being developed, deployed and iterated simultaneously, with new versions being deployed on cycles of only a few days or weeks. At the same time, thousands or millions of transactions are being processed on the line. Achieving parity requires addressing many sources of output disparities, and these disparities may occur only for transactions with certain characteristics. In this context, it becomes extremely difficult to keep track of where disparities are appearing, what factors are driving the disparities, and which disparities are in the process of being resolved.
To handle this challenge, Jay emphasizes the importance of automated reconciliation tools. In reconciliation, transaction streams are compared and disparities are detected. Then, analysts find patterns in these disparities, classify them into categories that share a common cause, and characterize the disparity categories for resolution by the development team. At the same time, the analysts create rules in the reconciliation engine that can detect and correct for disparities of each type.
Using these rules, the reconciliation engine distinguishes between the full universe of disparities, the known disparities that have already been characterized and are in the process of being resolved, and residual disparities that have not yet been analyzed. By filtering out known disparities, the reconciliation engine helps analysts focus on disparities that require attention and facilitates the detection and deciphering of patterns that might otherwise be overlooked or misidentified, especially in cases when multiple disparities overlap.
To the extent that reconciliation tools reduce the time and effort required to achieve parity, they facilitate rapid and safe deployment of new systems.
David, can these principle be universalized and implemented by different organisations or are these too specific and applicable only in shinsei where you are doing the study.
MA, I hope that the principles I’m developing will have applicability beyond Shinsei.
David this is a bit strange because, You said these are not implemented in Shinsei. So my assumption is that there is a fundamental flaw which prevent these principles from being implemented in Shinsei. Either that flaw has to be addressed in these principles to make it complete else they can not be implemented else where.
Also could you let us know the objective of your research and area you intend to cover in this as most things seem incoherent.
David I would also like to point out that some statements you are making with regards to time, speed, mainframes etc at Shinsei are not factually correct.
One reason I can think of is that your study group is very very small and growing smaller, so it seems you should also describe as part of these blogs the source or size of the group you are interviewing.
Hi David /MA
The Parity and Reconciliation is implemented and is a continual proccess improvement strategy across the entire banks enterprise systems including Subsidiaries.My opinion is that ,this is oneof the biggest sucess formual. Many Legacy Modernization ,SOX related audits tracebalita and complete enterprise data city transparency in terms of data, process, Knowledge base of systems is tested .Especially this parity and reconcilation which is reconcilation adn intelligence built analysis tool to arrive at patterns,processing etc for a legacy systems to new systems and right from source of originatio of transactions to its contribution to Banks ledger system can be traced,visualized and very transparent .
Rgds,
LV
Dear David,
I went through lots of your blogs. Can you give me a single example of Parity and reconciliation where it has contributed to shut-down of mainframe within shinsei bank itself.
And please don’t give me retail banking and it associated systems launch in Shinsei bank b’coz as I understand bank didn’t had retail banking beforehand. and believe me lot of 3rd party vendor can support bank in launching retail banking in 3~6 months time frame for bank of a size of Shinsei bank.
And in Shinsei bank case, I don’t understand how its dysfunctional IT set-up without any underline methodology and process is better than traditional/start-up enterprise.
I don’t think I need to explain you how dangerous ad-hoc processes untested, failed methodology can be. That too being implemented by incompetent vendors.
Hi Ramesh,
In couple of previous blogs David wrote that the objective of the study was not to find if things are implemented or not but to document what is possible. But this contradicts with his own statements where he chooses to quote time, speed, method etc as if they are integral part of shinsei.
Fundamentally i have been reading his blogs and every day just seem to grow more sceptical about what he writes as he continues to contradict him self in the various blogs. Also what i found is that when i provided with facts David decided not to allow those blogs to be published. So
Lastly i have been trying to figure out what is the purpose of these blogs and research but have not yet got a clear answer form David. Instead every time some topic is contented he leaves the topic and goes to some other. For example no conclusion of the Jsox topic since August last year, same way you will find no conclusion for this topic as well.
Dear David,
How different interoperation is from integration. To make sure the existing systems wont fail, if anything is processed on the new line segment, should it update the existing line’s database as if it were processed on the existing line? In the same manner, if a transaction is being processed on the existing line, should it also update that transaction on the new line’s databases?
Thanks
Yes, there should be interoperation from old to new and from new to old.