erp4it: Fundamentals of integration metadata I: Delving into the concept of "application"

Note: This is the first installment in my Fundamentals of Integration Metadata series. It is a foundation piece, delving into the concept of system which is a key structuring mechanism in understanding the IS environment.

Entire series:

Part 1: Delving into the concept of "application"

Part 2: Tracing the integration spiderweb

Part 3: Software deployment

Part 4: An integration metamodel

Overheard in the halls of Fortune 500 IT shops every day...

"PLV is down!"
"Quadrex isn't talking to the IC hub!"
"We're doing a capacity upgrade on the NST databases."
"The X-time batch was late two hours today."

What is meant by such gibberish?

Such conversations are the hallmark of the modern enterprise IT shop. The acronyms and abbreviations are managerial shorthand for large collections of IT components, usually grouped together in service of particular business purposes. These abstractions are essential to managing the complexity of a large IT environment, simplifying otherwise impossibly verbose conversations into pithy exchanges hooking together the technical and business worlds.

What are these things generally called? The words "system" and/or "application" are most often used. Some differentiate; one common pattern is that a "system" contains "applications." Another reading is that the "application" is the software, while the "system" includes the hardware, databases, people, and so forth. (This is a sound representation, but becomes a little more problematic in environments that share servers or databases.)

Others (including myself) simply equate system (sometimes qualified as “software system”) with application, arguing that to attempt the distinction is hopeless. For the purpose of this article, the two will be used interchangeably, and will refer generally to the software world. Systems run on servers and are dependent on databases.

By this approach, an application contains components. That is, the logical concept of system is a grouping of physical executable components. Now, if one sits down at a console in the data center, you don’t see a “system.” The Quadrex application mentioned above might be a collection of components:

qdx.exe
qdclt.dll
qdyr23.dll

and so forth, demonstrating the key logical/physical nature of the problem. How does a person know that the logical system concept Quadrex is embodied in those cryptic executables?

A mapping is necessary; a crossreference something like this:

Quadrex---qdx.exe
Quadrex---qdclt.dll
Quadrex---qdyr23.dll

A simplified information model might be:

Figure 1. System/Component

(This article is meant to be inclusive, so I’ll be doing both UML and IE notations. If anyone has any critique of how I equate things, I’m all ears.)

Simple in concept, but challenging in practice. Few IT shops have such a comprehensive cross-referencing; instead, the dependency of Quadrex on a given server or database simply is tribal knowledge. This can lead to all kinds of problems:

- licensing compliance
- correct version management
- understanding the business dependencies on a server

and so forth. (The general process solution to this problem is configuration management; which has much literature only a Google search away. However, this article is going in a different direction.)

The question of re-use also arises here. A shared executable might be used by many systems. However, if one is going to have any accountability, one must distinguish between uses and owns. This article is about ownership.

System/subsystem

As above, one common pattern is the decomposition of system into subsystem. This is of course not a new concept in software engineering; logical collections of components are generally seen as infinitely composable (that is, they can contain each other with no particular limit, like a series of Russian dolls).

The information model for this is the well-known recursive hierarchy:

Figure 2. Recursive systems

These models above are how to describe a tree structure:

Figure 3. Recursive tree

Software is sometimes seen as conforming to this kind of decomposition:

Figure 4. Recursive systems example

Note that libraries can call libraries, without any real limit to the depth of the tree. (A library might also be called by two different modules or other libraries, but that takes us into the many to many problem which I want to defer for now; again, this is about ownership, not just using.)

This is all fine when one is designing systems and needs to precisely document their interdependencies. However, limits of human communication and comprehension require a simplified approach in the running and management of systems.

“Application” in the IT management sense is usually much more limited; we are not talking about layers of a technological onion, but rather concepts used to structure a consensus reality among cooperating parties. Because of this, an indefinite number of layers is not recommended; for management purposes, a strictly-enforced three-layer hierarchy (e.g. system family, system, subsystem) can work quite nicely.

Figure 5. Explicit three-level

Thus, one may see logical groupings of systems, or systems containing subsystems/applications, but it’s not as fine-grained as the design-time view of executables, libraries, components, namespaces, and so forth.

Within this hierarchy, however, the concept of “system” should predominate: this is ideally where the support and financial aspects of running the IT operations are tied, with system group and subsystem as ancillary concepts. For example, as a business rule one might want to say that support organization is tied to system:

Figure 6. Three-level with support tied to system

This works fine if the hierarchy is established. But if one turns to the recursive approach

Figure 7. Recursion with support

one immediately encounters problems. Since “support org” can now be tied to any level of the hierarchy, one can get ambiguous representations:

Figure 8. Ambiguity in assigning support at different levels

Understanding the problems here is key. If System 1 contains Subsystem 1, why do they have different support organizations? What is meant by this? Is it contradictory? The answers may vary by shop, but the general issue is problematic; similar scenarios could be posed in relating systems to servers, projects, incidents, and many other things.

For a metadata repository or CMDB, it’s therefore recommended that the different levels in the system hierarchy be explicitly laid out, and indefinite depth not allowed. The specific relationships between objects should be strongly constrained by their level in the hierarchy. Note that this is an inherent problem with the indefinitely-decomposable concept of CI popular in many tools.

This is not a new recommendation in business data modeling; Reingruber and Gregory in their excellent, rigorous Data Modeling Handbook explicitly call for the resolution of recursive relationships.

(Note to any mathematicians/computer scientists reading this: I smell a general problem in graph theory here, one involving transitivity and containment relationships. If it has a name I would be deeply appreciative, because I’m running into it all over the metadata space.)

Some technologists might object to this: Why limit recursion? The technology handles it just fine! My response: I'm not talking about the technology, but about business processes and shared consensus of running enterprise IT - requirements that should inform the technology. In my experience it's much harder to build a shared consensus around hierarchies that have arbitrary depth. Reporting is very painful, in terms of simply designing and presenting intelligible information. The executives who consume reporting crave consistency, and if some things have 5 levels where others have 3 it's just one more headache of ambiguity.

Standard approaches: UML 1.x and CWM

The above analysis is strongly informed by the requirements of ITIL and the general ERP for IT concept. I’ve been looking at various standard approaches to representing this core concept; unfortunately, most of them run into the above problem.

Starting with UML 1.5: Here is the OMG conception of “subsystem.”

Figure 9. UML 1.x concepts

Because subsystem inherits from both Package and Classifier, it is recursively decomposable. This is entirely appropriate for the software world UML covers, but for IT management purposes runs into the above problems. The “isInstantiable” flag is significant, as it “States whether a Subsystem is instantiable or not. If false, the Subsystem represents a unique part of the physical system; otherwise, there may be several system parts with the same definition.” This foreshadows what I’ll call the “type/deployment/instance” problem, to be covered in a subsequent article.

A bottom-up representation is also possible using Component, which in UML “represents a modular, deployable, and replaceable part of a system that encapsulates implementation and exposes a set of interfaces.” Components are also decomposable.

Clearly, UML provides some semantics that might be leveraged here, but the indefinite recursion makes everything somewhat squishy for management purposes. The trouble in asserting a “standard” is that a useful standard for IT Service Management purposes would need to define the allowable levels in the recursion hierarchy, taking some firm positions such as (for example) ITSM linkages to other domains such as incident and problem are always applied to the top node of a UML subsystem hierarchy.

The Common Warehouse Metamodel has overlapping semantics:

Figure 10. Common Warehouse Metamodel Software Deployment (click to enlarge)

Based on UML, this representation makes some of the key IT management semantics a little more visible. While UML Subsystems can certainly contain Components, the relationship is explicit (not derived) here, as are the deployment semantics.

UML 2: A step backwards?

A very significant development in UML 2 is that Subsystem is no longer its own class, but rather a stereotype on Component, defined as

“A unit of hierarchical decomposition for large systems. A subsystem is commonly instantiated indirectly. Definitions of subsystems vary widely among domains and methods, and it is expected that domain and method profiles will specialize this construct. A subsystem may be defined to have specification and realization elements.”

This conflation of Component and Subsystem tends to conceal the crucial logical/physical relationship between System and Component; to handle this, a new attribute isIndirectlyInstantiated has been created:

“If false, the component is instantiated as an addressable object. If true, the Component is defined at design-time, but at runtime (or execution-time) an object specified by the Component does not exist, that is, the component is instantiated indirectly, through the instantiation of its realizing classifiers or parts.”

This gets back to the problem of indefinite recursion. By segregating logical from physical, one at least has two clearly distinguished levels in the system hierarchy, and if anything this distinction should be strengthened, not obscured. Logical Systems as managed consensus constructs will have quite different relationships and attributes than physical components. And again, a standard representation suitable (for example) for data interchange between ITIL supporting tools is still required. Further profile work is definitely needed for IT management purposes.

Further overlapping OMG work is to be found in the Enterprise Distributed Object Computing (EDOC) metamodel, but I’ll leave that be for now, as I don’t see any tools vendors rushing to support that work.

DMTF

The Distributed Management Task Force also addresses these issues. Here is a relevant subset of the Application schema:

Figure 11. DMTF Application schema (click to enlarge)

Again, we can see the logical/physical distinction in the relationship between CIM_InstalledProduct (which I equate with Application) and the CIM_SoftwareElements it contains. There is also great richness in the hardware layer and in the relationship of application to that hardware.

However, it’s necessary to realize that everything in the above diagram inherits from Managed Element, which again is an infinitely decomposable concept (actually, with several different hierarchies to choose from!):

Figure 12. DMTF Managed Element

So, the issue of an explicit hierarchy with a specific number of levels, or a privileged system level for management, is still unresolved.

In many ways, the DMTF work appears to have been created to support ITSM; it is unfortunate that it is only a graphical UML, inconsistent with the rigorously specified OMG standards for well-formed UML. This is reportedly going to be fixed in the next year.

Conclusion

This is an area ripe for further standardization, possibly in the same way that WS-I has standardized some common successful approaches for using the W3C’s voluminous Web Services specs. The trouble is not lack of standards, the trouble is too many overlapping standards.

As above, this is a necessary first chapter in understanding integration metadata. This concept of system will be crucial in understanding and managing the complexity of system integration. The next installment starts to examine integration semantics.

Entire series:

Part 1: Delving into the concept of IT "system"

Part 2: Tracing the integration spiderweb

Part 3: Software deployment

Part 4: An integration metamodel