Tag Archives: Context

Context as constraints in search space

This post is part of my collaborative research with Shinsei Bank on highly-evolvable enterprise architectures. It is licensed under the Creative Commons Attribution-ShareAlike 3.0 license. I am indebted to Jay Dvivedi and his team at Shinsei Bank for sharing with me the ideas developed here. All errors are my own.

It’s hard to have a conversation with Jay without the issue of “context” coming up–often in a sharp rebuke along the lines of “the problem with him/her/them/it/you is that the context is missing”. Clearly context plays a critical role in Jay’s design philosophy, but I’ve had considerable difficulty understanding what this important, easily-overlooked ingredient actually is. In the last iteration of my theory, I equated context with elemental subsystems where tasks are performed (in this iteration, I’ve relabeled these elemental subsystems workstations and defined them slightly differently). I’m not sure the context-as-elemental-subsystem formulation is entirely misguided, but I now think it’s only halfway correct (at best) and possibly misleading. So, in this post, I take another stab at articulating the meaning of context with respect to enterprise software architecture.

To describe the concept of context, Jay often uses the example of a computer booting up and recognizing its configuration. The operating system detects the processor, memory, storage devices, network interfaces, and other peripherals that define its physical structure. Thus the operating system becomes aware of its context; that is, the environment within which the system operates.

During my last visit to Shinsei, I had a discussion with Pieter Franken that helped clarify the role of context in enterprise systems. Pieter walked me through the architecture of the funds transfer system. The behavior of this system, he explained, must be decided within a sequence of environmental constraints. First, there are the rules and regulations that define the space of allowable transfers. Then, there are the restrictions on the sender, which depend on the characteristics of the sender and the bank’s relationship with him or her. Next, there are restriction on the recipient, similarly contingent on the recipient’s identity and relationship to the bank. Finally, there are restrictions associated with a given transfer depending on characteristics of the transaction. These constraints, Pieter explained to me, represent the context within which an actual transfer of funds takes place (or not).

Thus the funds transfer process must “boot up” much like a computer, becoming aware of its context by recognizing first the regulatory environment within which it operates, then the sender who seeks to invoke the process, then the proposed recipient, and then the characteristics of the transaction. Only once the process has constructed an orderly model of its environment is it prepared to carry out the work.

Defined in this way, context represents a set of constraints in the action space within which a system operates. In the case of an operating system, awareness that the computer is not connected to a wired network invalidates a large number of possible actions, such as sending or receiving data over the wired network interface. Similarly, in the case of the funds transfer system, awareness that that regulations forbid transfers to a particular country rules out actions associated with performing such transfers.

Restricting the action space has several possible benefits. First, the system can figure out what to do more easily, because it searches for an appropriate action within a smaller and simpler space. Second, errors and rework may be avoided, since the system is less likely to choose invalid actions if these actions are placed off limits a priori rather than detected post hoc by filters applied a posteriori. Third, simplifying the action space reduces the number of potential security vulnerabilities.

It seems, then, that awareness of context may be understood as the possession of a well-structured model of how environmental conditions constrain or otherwise influence the space of actions available to the system.

Workstations

This is the first of what I intend to be a series of short posts focusing on a few important aspects of the information factory perspective that I’m starting to develop. In the previous iteration of this work, I defined Contexts as elementary subsystems where tasks are performed. In this iteration, in keeping with the information assembly line metaphor, I’ve decided to replace Contexts with workstations. The basic idea doesn’t change: a workstation is an elementary subsystem where a worker, in a role, performs a task. I’d like to add a few nuances, however.

First, at least for the time being, I’m going to rule out nesting of workstations. Workstations can be daisy-chained, but not nested. A hierarchical structure similar to nesting can be achieved by grouping workstations into modular sequences, but these groupings remain nothing more or less than sequences of workstations. Conceptually, workstations divide the system into two hierarchical levels: the organization level (concerned with the configuration of workstation sequences) and the task level (concerned with the performance of tasks within specific workstations). This conceptual divide resembles, I think, the structure of service-oriented architectures, in which the system level (integration of services) is conceptually distinct from the service level (design and implementation of specific services).

The purpose of the workstation is simply to provide a highly structured and controlled environment for performing tasks, thereby decoupling the management of task sequences (organization level) from the execution details of specific tasks (task level). Workstations are thus somewhat analogous to web servers: they can “serve” any kind of task without knowing anything about the nature of its content. Each workstation is provisioned with only those tools (programs, data, and personnel) required to perform the task to which it is dedicated. The communication protocol for a workstation is a pallet interface, by which the workstation receives work-in-progress and then ships it out to the next workstation. Pallets may also carry tools and workers to the workstation in order to provision it.

An implementation of the workstation construct requires an interface for pallets to enter and leave the workstation, hooks for loading and unloading tools and workers delivered to the station on pallets, and perhaps some very basic security features (more sophisticated security tools can be carried to the workstation on pallets and installed as needed).

Information assembly lines

In my previous post, I explained my (admittedly somewhat arbitrary) transition from version zero to version one of my architectural theory for enterprise software. The design metaphor for version one of the theory is the high-volume manufacturing facility where assembly lines churn out large quantities of physical products. Design metaphors from version zero of the theory (the zoo, the house, the city, and the railway) will probably appear at some point, but I’m not yet exactly sure how they fit.

Jay often describes business processes at Shinsei as computer-orchestrated information assembly lines. These lines are composed of a series of virtual workstations (locations along the line where work is performed), and transactions move along the line from one workstation to the next on virtual pallets. At each workstation, humans or robots (software agents) perform simple, repetitive tasks. This description suggests that the salient features of the information factory¹ include linear organization, workstations, pallets, and finely-grained division of labor.

How does this architecture differ from traditional approaches? Here are a few tentative observations.

No central database. All information associated with a transaction is carried along the line on a pallet. Information on a pallet is the only input and the only output for each workstation, and the workstation has no state information except for log records that capture the work performed. In essence, there is a small database for each transaction that is carried along the line on a pallet. In keeping with the house metaphor, information on the pallet is stored hierarchically. (More thoughts about databases here.)

Separation of work-in-progress and completed work. Just like an assembly line in a factory, work-in-progress exists in temporary storage along the line and then leaves the line when completed.

In order to make the system robust, Jay adheres to the following design rules.

Information travels in its context. Since workstations have no state, the only ways to ensure that appropriate actions are taken at each workstation are to either (a) have separate lines for transactions requiring different handling or (b) have each pallet carry all context required to determine the appropriate actions to take at each workstation. The first approach is not robust, because errors will occur if pallets are misrouted or lines are reconfigured incorrectly, and these errors may be difficult to detect. Thus, all pallets carry information embedded in sufficient context to figure out what actions should be taken (and not taken).

All workstations are reversible. In order to repair problems easily, pallets can be backed up when problems are detected and re-processed. This requires that all workstations log enough information to undo any actions that they perform; that is, they must be able to reproduce their input given their output. These logs are the only state information maintained by the workstations.

Physical separation. In order to constrain interdependencies between workstations and facilitate verification, monitoring, isolation, and interposition of other workstations, workstations are physically separated from each other. More on this idea here.

The following diagram depicts the structure of an information assembly line. The line performs six tasks, labeled a through f. The red arrows indicate logical interdependencies. The output of a workstation is fully determined by the output of the preceding workstation, so the dependency structure resembles that of a Markov chain. Information about a transaction in progress travels along the line, and completed transactions are archived for audit or analysis in a database at the end of the line. Line behavior can be monitored by testing the output of one or more workstations.

Information assembly line

By contrast, here is a representation of a system designed according to the traditional centralized database architecture. The system has modules that operate on the database to perform the same six tasks. Although the logical interdependency structure is the same in theory, the shared database means that every module depends on every other module: if one module accidentally overwrites the database, the behavior of every other module will be affected. Moreover, all transactions are interdependent through the database as well. It’s difficult to verify that the system is functioning properly, since database operations by all six modules are interleaved.

Traditional system architecture with centralized database

Clearly, the information assembly line architecture requires more infrastructure than the traditional database approach: at a minimum, we need tools for constructing pallets and moving them between workstations, as well as a framework for building and provisioning workstations. In addition, we also need to engineer the flow of information so that the output can be computed using a linear sequence of stateless workstations. There are at least two reasons why this extra effort may be justified. At this stage, these are just vague hypotheses; in future posts, I’ll try to sharpen them and provide theoretical support in the form of more careful and precise analysis.

First, the linear structure facilitates error detection and recovery. Since each workstation performs a simple task on a single transaction and has no internal state, detecting an error is much simpler than in the traditional architecture. The sparse interdependency matrix limits the propagation of errors, and reversibility facilitates recovery. For critical operations, it is relatively easy to prevent errors by using parallel tracks and checking that the output matches (more on reliable systems from unreliable components here).

Second, the architecture facilitates modification and reconfiguration. In the traditional architecture, modifying a component requires determining which other components depend on it and how, analyzing the likely effects of the proposed modification, and integrating the new component into the system. If the number of components is large, this may be extremely difficult. By contrast, in the information assembly line, the interdependency matrix is relatively sparse, even if we include all downstream dependencies. Perhaps more importantly, the modified component can easily be tested in parallel with the original component (see the figure below). Thus, the change cost for the system should be much lower.

Parallel operation in an information assembly line

¹A search for the term “information factories” reveals that others have been thinking along similar lines. In their paper “Enterprise Computing Systems as Information Factories” (2006), Chandy, Tian and Zimmerman propose a similar perspective. Although they focus on decision-making about IT investments, their concept of “stream applications” has some commonalities with the assembly-line-style organization proposed here.

Contexts as elementary subsystems

Contexts are the elementary building blocks in Jay’s system architecture. I’ll define Contexts precisely below, but let me begin with a passage from The Sciences of the Artificial that provides a frame for the discussion.

By a hierarchic system, or hierarchy, I mean a system that is composed of interrelated subsystems, each of the latter being in turn hierarchic in structure until we reach some lowest level of elementary subsystem. In most systems in nature it is somewhat arbitrary as to where we leave off the partitioning and what subsystems we take as elementary. Physics makes much use of the concept of “elementary particle,” although particles have a disconcerting tendency not to remain elementary very long. Only a couple of generations ago the atoms themselves were elementary particles; today to the nuclear physicist they are complex systems. For certain purposes of astronomy whole stars, or even galaxies, can be regarded as elementary subsystems. In one kind of biological research a cell may be treated as an elementary subsystem; in another, a protein molecule; in still another, an amino acid residue.

Just why a scientist has a right to treat as elementary a subsystem that is in fact exceedingly complex is one of the questions we shall take up. For the moment we shall accept the fact that scientists do this all the time and that, if they are careful scientists, they usually get away with it. (Simon, 1996, 184-5)

For Jay, the Context is the elementary subsystem. Like an atom, the Context is in fact a complex system; however, designed properly, the internal structure of the Context is invisible beyond its boundary. Thus, system architects can treat the Context as an elementary particle that behaves according to relatively simple rules.

What is a Context?

A Context is a logical space designed to facilitate the performance of a small, well-defined set of actions by people acting in a small, well-defined set of roles. Metaphorically, Contexts are rooms in a house: each room is designed to accommodate certain actions such as cooking, bathing, sleeping, or dining. Contexts exist to provide environments for action. Although Contexts bear some resemblance to functions or objects in software programs, they behave according to substantially different design rules (see below).

Defining the Context as the elemental subsystem enables us, by extension, to define the elemental operation: a person, in a role, enters a Context, performs an action, and leaves the Context. All system behavior can be decomposed into these elemental operations, I’ll label them Interacts for convenience, where a person in a role enters, interacts with, and leaves a Context. The tasks performed by individual Interacts are very simple, but Interacts can be daisy-chained together to yield sophisticated behavior.

Design rules for Contexts

Creating Contexts that can be treated as elementary subsystems requires adhering to a set of design rules. Below, I describe some of the design rules that have surfaced in my conversations with Jay. These rules may not all be strictly necessary, and they are probably not sufficient; refining these design rules will likely be an essential part of developing a highly-evolvable enterprise software architecture based on Jay’s development methodology.

Don’t misuse the context. Only allow those actions to occur in a Context that it was designed to handle; do not cook in the toilet or live in the warehouse, even if it is possible to do so. Similarly, maintain the integrity of roles: allow a person to perform only those actions appropriate to his or her role. The repairman should not cook; guests should not open desk drawers in the study.
Physically separate contexts. Locate Contexts on different machines. Never share a databases among multiple contexts.
Only Interacts connect a Context to the rest of the system. Data enter and leave a context only through Interacts, carried in or out by a person in a role.
There is no central database. Every Context maintains its own database or databases as necessary.
Each Context permits only a limited set of simple, closely related actions. Contexts should be like a European or Japanese house where the toilet, bath, and washbasin are in separate rooms, rather than like a US house where all three are merged into a single room. If a Context must handle multiple modes of operation or multiple patterns of action, it should be decomposed into multiple Contexts.
Avoid building new Contexts. If a required behavior does not appear to fit in any existing Contexts, decompose it further and look for sub-behaviors that fit existing Contexts. Build new Contexts only after thorough decomposition and careful consideration.
Only bring those items–those data–into the Context that are required to perform the task at hand.
Control entry to the Context. Ensure that only appropriate people, in appropriate roles, with appropriate baggage (data) and appropriate intentions can enter.
Log every Interact from the perspective of the person and the Context. The person logs that he or she performed the action in the Context, while the Context logs that the action was performed in the Context by the person. This creates mutually verifying dualism.

Why bother?

The purpose of establishing the Context as an elementary subsystem is to simplify the task of system design and modification. As Simon points out, “The fact that many complex systems have a nearly decomposable [i.e., modular], hierarchic structure is a major facilitating factor enabling us to understand, describe, and even “see” such systems and their parts.” (1996, 207) Establishing the Context as an elementary subsystem in enterprise software is a technique for rendering enterprise software visible, analyzable, and comprehensible.

Bounding and restricting the Context vastly simplifies the work of implementors, enabling them to focus on handling a small family of simple, essentially similar actions. The Context can be specialized to these actions, thereby reducing errors and increasing efficiency.

Contexts hide the complexity associated with data and problem representations, databases, programming languages, and development methodologies, enabling system architects to focus on higher-level problems. In discussions with Jay, he almost never mentions hardware, software, or network technologies, since he can generally solve design problems without considering the internal structures of his Contexts and Interacts.

Since myriad organizational processes are assembled from a relatively small library of simple actions combined in different ways, systems that support these processes exhibit similar redundancy. Thus, Contexts designed to handle very simple actions can be reused widely, decreasing the cost and time required to develop new systems.

Finally, it is possible that Contexts, by explicitly associating people and roles with all actions, may help clarify accountability as organizational action develops into an increasingly complex mixture of human and computer decision-making.

Concluding thoughts

In essence, Contexts and Interacts are artificial constructs intended to allow high-level system design problems to be solved independently of low-level implementation problems. The extent to which the constructs achieve this goal depends on the effectiveness of the design rules governing the constructs’ behavior. Positing Contexts and Interacts as the elementary subsystems in Jay’s development methodology establishes a theoretical structure for further inquiry, but neither guarantee their fitness for this purpose nor implies the impossibility of other, perhaps more effective elementary subsystem constructs.

On several occasions, I’ve been asked how this approach differs from service-oriented architectures. I’ll explore this question in a subsequent post.