In an earlier post, I posited that Contexts serve as elementary subsystems in Shinsei’s architecture. What does this claim entail?

If Contexts are to be effective as elementary subsystems, then it must be possible to describe and modify the behavior of the system without examining their internal mechanics.  At least three conditions must be satisfied in order to achieve this goal.

1. The normal behavior of the Context is a simple, stable, and well-defined function of its input¹.
2. Errors can be detected, contained, and repaired without inspecting or modifying the contents of any given Context.
3. Desired changes in system behavior can be made by reconfiguring or replacing Contexts, without modifying their internal mechanics.

The first condition requires that a Context be a highly specialized machine, a sort of “one trick pony”.  This renders the behavior of the Context more predictable and less sensitive to its input.  For example, using a mechanical analogy, a drilling machine may drill holes of different depths or sizes, or it may wear out or break, but it will never accidently start welding.  The narrower the range of activity modes possessed by a component, the more predictable its behavior becomes.  The Context also becomes easier to implement, since developers can optimize for a single task.  In this respect, Contexts resemble the standard libraries included in many programming languages that provide simple, stable, well-defined functions for performing basic tasks such as getting the local time, sorting a list, or writing to a file.

The second condition–that errors can be detected, contained, and repaired at the system level–depends on both component characteristics and system architecture².  To detect errors without examining the internal mechanics of the Contexts, the system must be able to verify the output of each Context. Since errors are as likely (or perhaps more likely) to result from incorrect logic or malicious input as from random perturbations, simply running duplicate components in parallel and comparing the output is unlikely to yield satisfactory results. In an earlier post, I describe mutually verifying dualism as a verification technique. To contain errors, thereby ensuring that a single badly behaved component has limited impact on overall system behavior, output must be held and verified before it becomes the input of another Context.  Finally, repair can be enabled by designing Contexts to be reversible, so that an erroneous action or action sequence can be undone.  All outputs should be stored in their respective contexts so that the corresponding actions can be reversed subsequently even if reversal of downstream Contexts fails.

To allow for changes in system behavior without modifying the internal mechanics of Contexts requires only that the system architecture permit replacement and reordering of Contexts.  For an example of such an architecture, let us return to the programming language analogy and consider the case of software compilers.  Compilers allow reordering of function calls and replacement of one function call with another.  Equipped with appropriate function libraries, programmers can exert nuanced control over program behavior without ever altering the content of the functions that they call.

From the preceding discussion, it becomes clear that our goal, in a manner of speaking, is to develop a “programming language” for enterprise software that includes a “standard library” of functions (Contexts) and a “compiler” that lets designers configure and reconfigure sequences of “function calls”.  The limits of the analogy should be clear, however, both from the characteristics of Contexts described elsewhere and from the error detection, containment, and recovery mechanisms described above.

In conclusion, it seems worthwhile to highlight why traditional software design does not satisfy these requirements.  The most important reason is probably the use of centralized databases, the core component of most applications and enterprise systems (note that Contexts store their own data, so Jay’s architecture has no central database).  The database provides a data storage and retrieval module with a well-defined interface and several desirable properties.  Yet the database can by no means be considered an elementary subsystem: the design of its tables, and sometimes even its indices, are directly linked to almost all aspects of system-level behavior.  Although the interface is well-defined, it is by no means simple; indeed, it consists of an entire language with potentially unlimited complexity.  Errors can be reversed midway through a transaction, but they are often difficult to detect or repair after a transaction has completed.  Significant changes in system-level behavior almost always require modifications to the structure of the database and corresponding modifications to the internal mechanics of many other components.  Indeed, even seemingly trivial adjustments such as changing the representation of years from two digits to four can become herculean challenges.

¹ In computer science terms, this function defines the interface of the Context and serves to hide the implementation-specific details of the module (see Parnas 1972).

² The seminal work on this problem is, I think, von Neumann’s 1956 paper “Probabilistic Logics and the Synthesis of Reliable Organisms from Unreliable Components“.  Fortunately, the problem faced here is somewhat simpler: while von Neumann was seeking to build organisms (systems) that guarantee a correct output with a certain probability, I am concerned only with detecting and containing errors, on the assumption that the errors can be corrected subsequently with the aid of additional investigation.  Thus it is sufficient to detect and warn of inconsistencies, which is a far easier task than attempting to resolve inconsistencies automatically based on (potentially incorrect) statistical assumptions about the distribution of errors.

This blog approaches enterprise software architecture from a relatively theoretical perspective, so the implications for C-level strategic management may not be immediately obvious.  I’d like to take a step back and explain why the concepts that I’m trying to develop are of real and pressing importance for large organizations.

Even in the most stable industries, the business environment changes constantly.  Technology, customer needs, macro-economic trends, strategy and tactics of existing and emerging competitors, regulations, shareholder and union demands–all change constantly.  Change may be gradual or violent, predictable or unexpected, but it never stops. To sustain performance in a changing environment, businesses must adapt.

In practice, adaptation almost always requires modification of business processes. For example, incorporating new media into advertising strategies necessitates changing the processes that plan, implement, and evaluate marketing campaigns. Similarly,  to comply with new regulations, processes may need to incorporate different decision rules or generate new reports. It follows that the ability to modify business processes efficiently and rapidly is an important driver of organizational performance and a potential source of competitive advantage¹.

Imagine, for a moment, that you, the CEO of a major enterprise, are approached by a mysterious man in a blue suit who offers to sell you a remarkable robot.  This robot, he says, will perform much of the repetitive, time-consuming, routine work that absorbs the time and energy of your managers and staff, freeing them up to work on more interesting problems such as acquiring competitors, developing new products, or expanding into new markets. Although this robot is very, very expensive and takes several years to train, the man promises that the robot works so efficiently that it will save you a huge sum of money and pay for itself within a few years.  Although somewhat skeptical, you agree to purchase the robot.

Once the robot is finally ready for use, about a year behind schedule, you flip the switch.  The robot is a wonder to behold, continually dashing off in all directions to monitor inventories and account balances, manage employee salaries and vacation allowances, keep track of customer orders, and prepare financial reports.  The robot has some problems, such as calculating taxes incorrectly for a few states and occasionally losing orders, but the man in the blue suit assures you that the problems can be fixed in a few months–at some cost, of course. On the whole, you are satisfied: training the robot took longer than you expected (and the man in the blue suit charged a princely fee), and even now the robot misbehaves from time to time, but it does work very efficiently.

After about a year, your firm decides to introduce a new service.  Instead of simply selling widgets outright, you will also lease them, enabling you to optimize maintenance over the widget life cycle.  Of course, you must adjust your accounting processes to handle the leased widgets, and create new processes for coordinating the maintenance of leased widgets.  Aha!, you think, another task for our marvelous robot.  You call the robot into your office and ask it to take on these new tasks.  The robot stares at you blankly.  It doesn’t seem to understand.  So you consult with the kindly man in the blue suit.  Of course the robot can do this, he tells you, but we will have to retrain it.  And we will have to make sure that the new training doesn’t interfere with the robot’s current work.  This will take a year or two, he says, and it will be very, very expensive.

Couldn’t you just buy another robot, you ask, that would handle these new tasks? That would be very difficult, the man says, because you will end up with two separate accounting statements and two separate maintenance schedules that will conflict with each other.  That may be so, you say, but can’t the robots be trained to talk with each other and solve these problems by themselves?  Yes, says the man, with a twinkle in his eye, but it will take several years and it will be very, very expensive.

In the end, you do as the man in the blue suit suggests.  And as the years go by, you find with increasing frequency that you face unpleasant choices between passing up business opportunities, or paying the man in the blue suit astronomical sums of money to retrain your robot to help you to capture the new opportunities (after seemingly interminable delays).  You find it increasingly difficult to compete with smaller, younger companies with newer robots.  You begin to wonder if you should have bought the robot in the first place.

The robot, of course, is the traditional enterprise software system.  Installed at great expense, it works (perhaps) for a while.  Over time, it becomes increasingly inflexible and increasingly costly to maintain.  It promises a Faustian bargain: improved efficiency in the short term, at the expense of adaptability in the long term.  The cost and time required to modify business processes rise higher and higher, to the point where they may nullify the benefits of otherwise profitable adaptations.  Organizations may find that stagnation and gradual decline are, unattractive as they may be, nevertheless preferable to costly, risky, and time-consuming system modifications.  Moreover, organizations become increasingly dependent on the firms that maintain the systems.  Since many organizations struggle to achieve even short term efficiency improvements, traditional enterprise systems are dubious bargains indeed.

Baobabs

Jay uses the metaphor of a tree to describe this process.  The enterprise system takes root in the organization’s processes and extends ever more deeply into them.  Over time, the organization becomes so tightly bound up by these roots that it loses the freedom to move.  Enterprise systems are like the baobabs described by the Little Prince:

A baobab is something you will never, never be able to get rid of if you attend to it too late.  It spreads over the entire planet.  It bores clear through it with its roots.  And if the planet is to small, and the baobabs are too many, they split it in pieces… (Antoine de Saint Exupéry, 1971, 22)

Pulling up the baobabs

Though the preceding exposition may be a bit whimsical, the threat posed by inflexible enterprise software is not.  Fortunately, just as the Little Prince found that he could solve his baobab problem by uprooting the baobabs as soon as they became distinguishable from rosebushes, the solution in the case of enterprise software may be similarly simple, if not nearly so obvious.

Judging from Jay’s experience “pulling up the baobabs” (i.e., getting rid of inflexible enterprise systems) and replacing them with more congenial species, the problem can be solved by architecting systems according to a relatively small set of design rules. These rules make systems far more modular, and hence smaller and more manageable–more like rosebushes than baobabs.  Some of these rules, such as defining system components in terms of their inputs and outputs, are relatively familiar.  Others, such as extreme decomposition, mutually-verifying dualism, component-level interoperation, context-awareness, and reversibility, are less familiar or, in some cases, entirely novel.

Applying these design rules appears to yield systems that are less costly, less risky, and less time-consuming to build, easier to manage and maintain, and far more malleable. Moreover, the systems can be characterized in terms of machines and assembly lines that build data objects–that is, in terms accessible to CEOs and line of business managers–while banishing arcane software engineering jargon to the domain of implementors where it belongs.

Although I am optimistic about the potential of these design rules (otherwise, I would not bother to research them), much work remains to be done in order to articulate them precisely and evaluate them accurately.  Thus, readers will probably encounter logical gaps, questionable assertions, and as yet unexplored byways.  Please do not hesitate to chime in with constructive ideas, suggestions, or criticisms.

¹For a theoretical treatment of this argument, see the literature on dynamic capabilities (e.g., Winter 2003, Teece, Pisano & Shuen 1998).

Contexts are the elementary building blocks in Jay’s system architecture.  I’ll define Contexts precisely below, but let me begin with a passage from The Sciences of the Artificial that provides a frame for the discussion.

By a hierarchic system, or hierarchy, I mean a system that is composed of interrelated subsystems, each of the latter being in turn hierarchic in structure until we reach some lowest level of elementary subsystem.  In most systems in nature it is somewhat arbitrary as to where we leave off the partitioning and what subsystems we take as elementary.  Physics makes much use of the concept of “elementary particle,” although particles have a disconcerting tendency not to remain elementary very long.  Only a couple of generations ago the atoms themselves were elementary particles; today to the nuclear physicist they are complex systems.  For certain purposes of astronomy whole stars, or even galaxies, can be regarded as elementary subsystems.  In one kind of biological research a cell may be treated as an elementary subsystem; in another, a protein molecule; in still another, an amino acid residue.

Just why a scientist has a right to treat as elementary a subsystem that is in fact exceedingly complex is one of the questions we shall take up.  For the moment we shall accept the fact that scientists do this all the time and that, if they are careful scientists, they usually get away with it. (Simon, 1996, 184-5)

For Jay, the Context is the elementary subsystem.  Like an atom, the Context is in fact a complex system; however, designed properly, the internal structure of the Context is invisible beyond its boundary.  Thus, system architects can treat the Context as an elementary particle that behaves according to relatively simple rules.

What is a Context?

A Context is a logical space designed to facilitate the performance of a small, well-defined set of actions by people acting in a small, well-defined set of roles.  Metaphorically, Contexts are rooms in a house: each room is designed to accommodate certain actions such as cooking, bathing, sleeping, or dining. Contexts exist to provide environments for action.  Although Contexts bear some resemblance to functions or objects in software programs, they behave according to substantially different design rules (see below).

Defining the Context as the elemental subsystem enables us, by extension, to define the elemental operation: a person, in a role, enters a Context, performs an action, and leaves the Context.  All system behavior can be decomposed into these elemental operations, I’ll label them Interacts for convenience, where a person in a role enters, interacts with, and leaves a Context.  The tasks performed by individual Interacts are very simple, but Interacts can be daisy-chained together to yield sophisticated behavior.

Design rules for Contexts

Creating Contexts that can be treated as elementary subsystems requires adhering to a set of design rules.  Below, I describe some of the design rules that have surfaced in my conversations with Jay.  These rules may not all be strictly necessary, and they are probably not sufficient; refining these design rules will likely be an essential part of developing a highly-evolvable enterprise software architecture based on Jay’s development methodology.

1. Don’t misuse the context. Only allow those actions to occur in a Context that it was designed to handle; do not cook in the toilet or live in the warehouse, even if it is possible to do so.  Similarly, maintain the integrity of roles: allow a person to perform only those actions appropriate to his or her role.  The repairman should not cook; guests should not open desk drawers in the study.
2. Physically separate contexts. Locate Contexts on different machines.  Never share a databases among multiple contexts.
3. Only Interacts connect a Context to the rest of the system. Data enter and leave a context only through Interacts, carried in or out by a person in a role.
4. There is no central database. Every Context maintains its own database or databases as necessary.
5. Each Context permits only a limited set of simple, closely related actions. Contexts should be like a European or Japanese house where the toilet, bath, and washbasin are in separate rooms, rather than like a US house where all three are merged into a single room.  If a Context must handle multiple modes of operation or multiple patterns of action, it should be decomposed into multiple Contexts.
6. Avoid building new Contexts. If a required behavior does not appear to fit in any existing Contexts, decompose it further and look for sub-behaviors that fit existing Contexts. Build new Contexts only after thorough decomposition and careful consideration.
7. Only bring those items–those data–into the Context that are required to perform the task at hand.
8. Control entry to the Context. Ensure that only appropriate people, in appropriate roles, with appropriate baggage (data) and appropriate intentions can enter.
9. Log every Interact from the perspective of the person and the Context. The person logs that he or she performed the action in the Context, while the Context logs that the action was performed in the Context by the person.  This creates mutually verifying dualism.

Why bother?

The purpose of establishing the Context as an elementary subsystem is to simplify the task of system design and modification.  As Simon points out, “The fact that many complex systems have a nearly decomposable [i.e., modular], hierarchic structure is a major facilitating factor enabling us to understand, describe, and even “see” such systems and their parts.” (1996, 207) Establishing the Context as an elementary subsystem in enterprise software is a technique for rendering enterprise software visible, analyzable, and comprehensible.

Bounding and restricting the Context vastly simplifies the work of implementors, enabling them to focus on handling a small family of simple, essentially similar actions.  The Context can be specialized to these actions, thereby reducing errors and  increasing efficiency.

Contexts hide the complexity associated with data and problem representations, databases, programming languages, and development methodologies, enabling system architects to focus on higher-level problems.  In discussions with Jay, he almost never mentions hardware, software, or network technologies, since he can generally solve design problems without considering the internal structures of his Contexts and Interacts.

Since myriad organizational processes are assembled from a relatively small library of simple actions combined in different ways, systems that support these processes exhibit similar redundancy.  Thus, Contexts designed to handle very simple actions can be reused widely, decreasing the cost and time required to develop new systems.

Finally, it is possible that Contexts, by explicitly associating people and roles with all actions, may help clarify accountability as organizational action develops into an increasingly complex mixture of human and computer decision-making.

Concluding thoughts

In essence, Contexts and Interacts are artificial constructs intended to allow high-level system design problems to be solved independently of low-level implementation problems.  The extent to which the constructs achieve this goal depends on the effectiveness of the design rules governing the constructs’ behavior.  Positing Contexts and Interacts as the elementary subsystems in Jay’s development methodology establishes a theoretical structure for further inquiry, but neither guarantee their fitness for this purpose nor implies the impossibility of other, perhaps more effective elementary subsystem constructs.

On several occasions, I’ve been asked how this approach differs from service-oriented architectures.  I’ll explore this question in a subsequent post.

Jay recently introduced me to Pivotal Tracker, a “lightweight, free, agile project management tool”.  It looks like a promising step toward computer-orchestrated knowledge work.  To explain what I mean, let’s start by thinking through the relationship between structured work, unstructured work, and computers.

Using computers to orchestrate highly structured work is relatively straightforward, because structure translates relatively directly into software algorithms¹.  Much knowledge work, and especially sophisticated knowledge work at the core of modern economies such as research, design,  product development, software development, strategic analysis, financial modeling, and general management, is relatively unstructured.  Can computers support such general knowledge work?

One way to leverage computers in unstructured work is to decompose the work and factor out structured subproblems that can be delegated to computer systems.  Done effectively, this enables the structured subproblems to be solved more rapidly, reliably, and inexpensively.  In cases where performance on structured and unstructured subproblems complement each other, computerization of unstructured subproblems may lead to qualitative improvements in overall problem-solving performance.  In other words, computerization may result not only in efficiency gains but also in qualitatively better output.

For example, computer-aided design software and spreadsheets make possible more sophisticated building designs and financial models by efficiently solving critical structured subproblems.  Factoring the (relatively) unstructured task of designing a building or modeling the growth of a new business into unstructured, creative subproblems (sculpting the contours of the building, selecting the parameters in the model) and structured, algorithmic subproblems (mathematical calculations, visualizing data, storing and retrieving work in progress) enables architects and business analysts to focus their attention on creative tasks while computers handle routine processing.

If the complementarity between structured subproblems and unstructured subproblems be sufficiently strong, factoring out and computerizing structured subproblems will increase human employment.  If architects can deliver better designs at lower cost, demand for architects will rise.  If business analysts can deliver deeper insight faster, demand for business analysts will rise.  The degree of complementarity depends to some extent on inherent characteristics of the problem domain, but problem factoring and computer system design influence the degree of complementarity as well².  Thus, advances in computer-orchestrated work may have significant implications for firm performance and economic growth.

Seen from this perspective, the Pivotal Tracker is an intriguing technology.  Its design is premised on the agile programming technique of structuring software development as a series of short (one to four week) iterations.  Development work is further decomposed into a large number of small, modular “stories” which (as far I understand the methodology) describe bits of functionality that deliver incremental value to the customer.  During each iteration, the development team implements a number of stories.

Although originally intended for managing software development, Pivotal Labs, the company behind Pivotal Tracker, proposes using the tool for managing just about any kind of project.  From the FAQ:

A project can be anything that you or your team works on that delivers some value, and that is large enough to benefit from being broken down into small, concrete pieces. For example, a project may be to develop software for an e-commerce web site, build a bridge, create an advertising campaign, etc.

Pivotal Tracker screen shot. The active stories for the current iteration are shown on the left, and the backlog is on the right.

The reason Pivotal Tracker (PT) represents a step forward in the computerization of knowledge work is that the tool goes beyond simply tracking progress on a collection of tasks.  To begin with, PT enables quantitative planning and analysis by asking users to rate the complexity of each story on a point scale.  Several scales are available, including a three point scale. Constrained scales enforce discipline in problem decomposition: for example, using a three point scale, stories cannot be rated accurately if their complexity exceeds three times the complexity of the simplest (one point) stories.

PT uses these complexity ratings to measure the rate of progress in terms of points completed per iteration (termed velocity) and estimate the time remaining until project completion.  According to Pivotal, estimates of future progress based on historical velocity prove relatively accurate.  PT orchestrates the work by maintaining a queue of active stories to be completed in the current iteration and a prioritized backlog of stories for completion in future iterations.  After an iteration ends, PT moves stories from the backlog to the active queue.  PT manages the active queue to keep the project moving forward at a constant velocity (complexity points per iteration), helping the team stay on schedule and avoiding last-minute dashes.  All of this occurs transparently, without burdening the team members.

PT also handles simple work flow for each story.  Team members take ownership of stories by clicking a start button on the story, and then deliver them to the requester for approval when finished.  This clearly delineates accountability and enforces separation of worker and approver.

Technologies for computer orchestration of knowledge work are still relatively primitive, but Pivotal Tracker seems to represent a significant step forward.

¹ Work is structured to the extent that it can be executed predictably using specialized routines.  More here.  For a rigorous study of the tasks amenable to computerization, see Autor, Levy & Murnane 2003.

² Regarding the importance of how problems are factored, see von Hippel, 1990.  On the implications of computer system design, see Autor, Levy & Murnane, 2002 and Zuboff, 1989

グーグルは去年発表した次世代コラボレーション・サービスの「Wave」の開発を中止にし、失敗だと認めました。これについて社長のエリック・シュミット（Eric Schmidt）は下記のように話していました。

We try things … we celebrate our failures. This is a company where it’s absolutely okay to try something that’s very hard, have it not be successful, and take the learning from that. (TechCrunch)

小生なりに日本語に訳しますとこういうことです。

我々は試してみるんです。我々は失敗を称賛します。この会社では、とても難しいことを試して、成功せずに終わって、そしてそこからの学びを得ることが全く大丈夫なんです。

日本企業で「we celebrate our failures」と言える会社はほとんどないでしょう。日本の伝統的な経営はむしろ失敗に対して極めて厳しいと思います。通常の業務を効率よくこなすことにおいてはよいのかも知れませんが、イノベーションを壊してしまいます。

In my research on computer-assisted organizing, I set out to understand how computers alter the fabric of organizations.  Here’s how I framed the problem in my dissertation:

In the computer age, complex information processing tasks are divided between humans and computers.  Though designed and developed by humans, computers are autonomous agents that function as independent decision-makers.  The dynamics of electronic information processing influence the dynamics of organizing and organizations in ways that cannot be understood in purely  human terms. (Brunner, 2009)

My dissertation focused primarily on two aspects of this transformation: how computers drive further specialization in information processing work, and how computer-assisted work increases business scalability.  A third aspect of the transformation had been on my mind ever since my days as a management consultant: it seems that computers and people are trading places within organizations.

In the past, humans created organizational structure through their patterns of interaction, while computers were plugged in to this structure to perform specific tasks. Increasingly, these roles are reversed: computers create organizational structure, while humans plug in to the computer system to perform specific tasks.  Shinsei Bank’s mortgage loan operations provide an elegant example of the phenomenon. Rather than human credit approvers managing the loan application process from beginning to end and using computers to perform calculations or look up policies, a loan application system manages the process, calling on human appraisers, data entry clerks, analysts, or supervisors to provide input as necessary.

In Jay’s words, the computers orchestrate the work.  The Oxford English Dictionary defines orchestrate as follows:

To combine harmoniously, like instruments in an orchestra; to arrange or direct (now often surreptitiously) to produce a desired effect.

The word seems apt. In computer-orchestrated work, computers arrange and direct business processes in order to “combine harmoniously” the work of individuals.  Much like an assembly line, computer-orchestrated work enables individuals to focus on simple, well-defined tasks, while computers handle the coordination and integration of these fragmentary outputs. As bureaucracy¹ eliminated the reliance of organizations on specific individuals by defining roles, so computer-orchestrated work enables organizations to survive without the patterns of human interaction that define and sustain the structure of traditional organizations.

Computer-orchestrated work may greatly increase organizational performance. By lowering to nearly zero the marginal cost of coordination and integration, computer-orchestrated work makes possible greater specialization, which accelerates learning and increase efficiency. Moreover, computer-orchestrated work lowers the costs of monitoring and metering, potentially reducing agency costs. Computer-orchestrated work is easier to analyze and modify, which facilitates innovation and increases the returns to highly-skilled human labor (c.f. Zuboff, 1989). Although the design challenges are significant, computer-orchestrated work may be an essential tool for creating more intelligent organizations.

¹In the Weberian sense, as a highly effective organizing technology.

How to deal with the extreme complexity of enterprise software? The traditional approach relies on a set of abstractions related to data models and databases, interfaces, processes, and state machines.  These conceptual tools are rooted in the theory of computer science. Jay takes a different approach: he attempts to mimic physical systems that solve analogous problems in the real world.  Much as modern operating systems mimic the file and folder model that people use to organize paper documents, Jay’s software architectures mimic zoos, houses, railways and cities.

Enterprise software is filled with virtual things that come into existence, experience a variety of transformations, and finally disappear or solidify into permanent log records.  These things may be users, customers, transactions, and so forth.  To Jay, these things are the animals in a virtual zoo.  They have each have their own life cycles and their own needs.  Cages must be used to separate different kinds of animals in order to care for them and prevent them from interfering with each other.  Processes specific to each kind of animal must be implemented for rearing, feeding, healing, disposing. The first step in designing a system is to identify the animals that will inhabit the system, separate them, and cage them.

The house metaphor complements and overlaps the zoo metaphor.  Just as people live in houses in the physical world, representations of users and customers live in virtual houses. Users store their virtual belongings–records, certifications, etc.–in their houses, which they access using virtual keys.  Houses have different rooms which are used for different tasks, and each room has equipment appropriate for the tasks to be performed there.  Users, as represented in the system, must be aware of their context so that they perform the appropriate tasks in the appropriate places. Jay emphatically forbids “cooking in the toilet.” Users must also be aware of their roles: a guest in the house behaves differently from a plumber, and a plumber differently from the owner of the house.

When users are needed outside their houses to perform tasks, they travel on virtual trains to stations where operations are performed.  They are aware of their destination from the outset, so they take with them only those belongings required to complete the task. After completing the task, they return to their houses.  All of this happens transparently to the actual user: the architecture of the system does not dictate the structure of the user interface.

Together, the houses, trains, and stations make up a virtual city that models the work of the bank.  This metaphor seems rather distant from the mechanics of an enterprise software application, at least compared to the familiar desktop metaphor, and I’m still putting together the pieces–so please consider this post a tentative first step toward articulating the idea.  I’ll revisit the topic and add more detail in future posts. In any case, the key takeaway seems to be that the intricate and highly modular division of labor in the real world may be a useful metaphor to guide the design of modular systems in virtual worlds.

In an earlier post, I attempted to identify distinguishing characteristics of enterprise software. Jay pointed out to me another: the coevolution of enterprise software with the business of its host companies. Companies must adapt to continuously changing environmental conditions.  Since enterprise software reaches deep into the operations of the business, the software must change, too.  To achieve high performance, changes in the business and changes in the software must fit with each other; through this mechanism, each influences the evolution of the other.

This suggests a possible reason why standardization of enterprise software has proved so difficult. Companies often respond to competition by seeking to develop differentiated capabilities. If enterprise software and organizational capabilities coevolve, then companies may experience constant pressure to customize their enterprise software systems.

For more than a year, Jay and I have been carrying on a dialogue about his methodology for developing enterprise software. This methodology appears to depart in many ways, sometimes radically, from traditional approaches.  As a first step toward characterizing this methodology, I’m going to begin by writing a series of blog posts about the guiding principles that Jay has described in our conversations.

These principles are not necessarily mutually exclusive or collectively exhaustive, but they capture the essence of the methodology.  At this point, I don’t fully understand how these principles fit together, and Jay has encouraged me to focus on thoroughly understanding each of principle in isolation before attempting to assemble the principles. Consequently, these blog entries may seem fragmentary or disconnected. As the research progresses, I hope to integrate and synthesize these principles into a coherent set of theoretically grounded design rules. Let’s dive into a principle.

Mutually verifying dualism: Model all operations as pairs of reciprocal actions between two agents in autonomous, reciprocal roles.

In this context, the term autonomous means that neither agent can dictate the behavior of the other; the two agents belong to separate control hierarchies and maintain their own records.  Transactions provide a simple example, since they naturally lend themselves to dualism.  Without dualism, we might model a transfer of a security from Alfred to Bernard as a unitary transfer operation recorded in a centralized transfer log. Such centralization may be convenient from a design perspective, but it has several drawbacks.  First, an agent that controls the transfer functionality can perform transfers without consulting Alfred, Bernard, or any other user.  Second, fraudulent or erroneous transactions may be impossible to detect, because the centralized transfer log is the sole source of transfer information and cannot be cross-checked. Third, transfer records are lumped together in a single log, so retrieving transfers performed by a particular user requires extracting them from a large database that may contain billions of records for millions of users. (The magic of modern databases makes this extraction of needles from a haystack possible and even straightforward, but it seems like a lot of infrastructure for an essentially simple task.  As I’ll discuss in a future post, Shinsei’s philosophy is not to drop the needles into the haystack in the first place.)

Alternatively, following the principle of mutually verifying dualism, we can break the transfer into a pair of reciprocal operations between Alfred, in the role of provider, and Bernard, in the role of recipient.  The transfer occurs only if both Alfred and Bernard cooperate, and the transfer can be verified by comparing Alfred’s record of the transfer with Bernard’s and ensuring that the records match.

Mutually verifying dualism requires that all operations be modeled in this way, entailing a shift in the way we conceptualize many operations.  For example, consider Carl logging in to a savings account online.  This operation would traditionally be modeled by the system as a unitary login event and recorded in a centralized log.  To apply the mutually verifying dualism principle, we break the login operation into a pair of reciprocal operations between Carl and Carl’s Savings Account (assuming, for the moment, that he is logging in to his own account). Carl, modeled in the system as a user with its own identity and records, requests access to Carl’s Savings Account and records the result.  Inversely, Carl’s Savings Account, similarly modeled as an independent entity with its own identity and records, grants (or denies) access to Carl and records the result. Using this approach, fraudulent or erroneous logins can be detected (sometimes) by reconciling the user and account records.

Shinsei visualizes its network of mutually verifying operations as a geodesic sphere. Image by Theon via Wikimedia Commons, used under Creative Commons license.

In principle, mutually verifying dualism resembles double-entry bookkeeping, which helps detect errors by breaking unitary financial flows into dual credit/debit operations.  As in double-entry bookkeeping, mutually verifying dualism breaks down if a saboteur takes control of the systems on both sides of the interaction (e.g., the user system and the savings account system), or symmetric errors occur in both systems.  The probability of these relatively unlikely events can be further reduced by designing a network of mutually verifying dualisms. The imagery used by Shinsei to describe the approach is a geodesic sphere, where the location of every vertice can be verified from multiple, independent perspectives.

Perhaps the most important implication of mutually verifying dualism is that records of past events can be reconstructed if any system component breaks irreparably.

Mutually verifying dualism creates some complications for system design.  Since interacting agents must belong to separate control hierarchies, centralized designs are infeasible a priori. Duality implies redundancy, so changes need to be propagated to all concerned agents. Modularity probably precludes the possibility of unified or universal data models.