SOA Implementation and Management

Service-oriented architecture (SOA) is a comprehensive concept that has far-reaching consequences for the design of an IT application landscape. While it is comparatively easy for software vendors to establish SOA, for example as a fundamental cross-cutting concern for a new product generation, enterprises have a harder time justifying and controlling the transition of an entire application landscape to a SOA. IT users and IT vendors believe that "SOA is good" and that one has the choice of "either implementing SOA or perishing". However, if you want to use such arguments to convince the IT committee of a fortune 1000 enterprise to invest a two- to three-digit million budget in a SOA strategy, you will fail, and often rightly so. This article shows where SOA initiatives frequently get bogged down and how SOA can be anchored in an IT governance system to make SOA a success. SOA governance helps to achieve this goal. This article, which treats SOA governance as an integral part of IT governance and architecture governance, will show where SOA governance does not differ significantly from other current standard practices of IT enterprise architecture and where SOA requires special treatment in architecture management.

1. Introduction and Overview

Service orientation today is often portrayed as an imperative, without which an IT organization can practically no longer exist. At least one book title aptly summarizes this hype: "Service Orient or Be Doomed!" [Bloomberg+2006]. In addition, SOA governance tools are offered by various vendors, creating the impression that SOA can be implemented and governed simply by purchasing these tools. In this way, we are led to believe that an IT organization without SOA will not survive much longer. In daily practice, however, you will also observe the following, for example:

Although the SOA hype was created in part by Gartner, the following graph shows that Gartner thinks the hype on SOA technologies has not yet reached its peak. By way of comparison: after passing the "peak of inflated expectations" and descending into the "trough of disillusionment", organizations will climb the "slope of enlightenment" toward the "plateau of productivity" in the hype cycle. The path to this goal is long, however, and generally is estimated to take 10 years.
You will find large-scale IT organizations that claim they have been practicing SOA since the late 1990s. The credibility of such claims will be discussed in the course of this article.
In practice it can also be observed that many SOA initiatives fail and are abandoned quietly; some never develop any degree of effectiveness. This article will discuss why that can happen.
And you can observe that there certainly are organizations that score success with SOA, often without having had an explicit SOA strategy from the start.

Figure 1: Position of many SOA technologies on the hype curve. See also the citation from Gartner in the box below.

Without already giving a precise definition of SOA governance at this point, here is a concise description of the concept:

SOA governance means creating conditions under which the SOA in the enterprise can grow.

Of course, this has to benefit the enterprise, because IT governance deals with controlling IT so that it creates a positive effect on the bottom line for the enterprise in which it is embedded, precisely with IT as an end in itself. At least 3 general ways of looking at SOA can be extracted from the above:

Technical vendors who offer products related to SOA: They never tire of claiming that a business only has to use their SOA products and their customers will automatically have the flexibility be able to implement new business processes.
End user IT organizations: They can see SOA as a vehicle for attempting once again to implement good architecture, as they previously attempted to implement it by means of component hype or OO hype. But they can also see SOA as one among many tools that they can use to generate business value.
Business units: The business units are indifferent as to how IT supports their business goals, as long as IT supports the goals. Whether the means of achieving that is called SOA or something else is irrelevant.

It is relatively easy for vendors of large software packages to justify why they have to switch their software to service orientation, because it is then easy to integrate the software in business processes and the customer will require that additionally purchased software be "service oriented". This article will not deal in more depth with the perspective of these vendors.

The case from the point of view of the end user organizations is less clear. There is usually no direct and trivial effect on the success of these enterprises if the software landscape is service-oriented. This article explores the question of what has to happen in a company to permanently establish the issue of SOA, and to control it to the advantage of the enterprise.

Gartner on SOA

Service-Oriented Architectures (SOA) will also be a long-term issue for enterprises, according to Gartner. Event-driven architectures and model-driven architectures are mentioned as types of technologies for SOAs. Both have yet to pass the "peak of inflated expectations" on the hype curve and will need five to ten years before one can speak of widespread productive use. The analysts understand an event-driven architecture to be a software landscape in which the program function is encapsulated and can be flexibly linked to business objects or application components. It is started by a message trigger and can be triggered by other application components, adapters or software agents. There are already examples for early EDA implementations, such as the financial or telecom verticals and supply chain logistics.

Source: Computerwoche: Gartner Assesses Maturity of New IT Technologies, URL: http://www.computerwoche.de/produkte_technik/579955/index.html visited on 12/05/2006

Section 2 will describe the widespread technology-driven adoption paths for SOA. The purpose of this is not to show it as being exemplary, but rather to show what it can achieve. If you are lucky, it can help you to arrive at your goal.

The next Section will describe business-driven adoption paths for SOA. A key word here is IT alignment – alignment of the IT function to business strategy, and business value. Hence Section 4 will give an outline of IT governance is. We will demonstrate how IT governance can ensure that the IT function is aligned to business strategy, and business value. This allows us to see SOA governance as a derivative of IT governance. You will see that SOA governance has very little to do with special tools, but instead with the alignment of IT and therefore of the SOA to the business goals.

If an enterprise has intact and goal-oriented IT governance and IT architecture governance, then SOA governance is practically an automatic consequence of these.

The summary will show what an IT function can do to progress from IT alignment to pro-active IT management, which not only takes advantage of opportunities, but also develops them systematically. In this process, SOA is only one of many building blocks for achieving the goal.

2. Technology-driven Adoption Paths for SOA

SOA as IT initiative

Quite often, SOA starts as an initiative of IT for the sake of IT. When good IT people look at SOA, they often notice that SOA can be seen as "old wine in a new bottle" and that it can be used to finally implement a good software design at the enterprise level. Concepts such as modularization, interfaces, encapsulation or the information hiding principle are not new to the field of software design. But the budgets for their consistent implementation were not and are not always available.

Such considerations frequently are the basis for deriving a technology-driven adoption path for SOA. Those responsible in IT then search selectively for suitable projects on the business side in order to obtain financing for their SOA agenda. This approach will first be described in the following. It does not have to fail and can even be successful. However, it does contain inherent risks, which can be avoided with the procedure described in Section 3. You may be wondering at this point why this approach is even being described here. It is being described so that you will recognize it in case it is pursued in your enterprise.

The problem of the first SOA budget

So you are convinced that the implementation of a SOA would be a very desirable program for your IT organization. You have also read case studies and know from these that the final stage of a company-wide SOA program, in which all existing systems make their services available for example via an Enterprise Service Bus (ESB), is an expensive undertaking, which can consume two- to three-digit million sums for a larger bank or insurance company. You also know that such budgets could perhaps still be obtained in the 1990s, but that you will no longer obtain such a budget today, in the 21st century, without calculating a business case.

Therefore, you have to think about where to get the means for your entry to SOA. Consider, for example, the following approaches:

Pilot project in IT department: If the business side shows no spontaneous interest in SOA, you can conduct a SOA pilot in the IT division, which is financed by IT itself. Such a pilot project will cost you the software and hardware for a suitable service infrastructure, which you can purchase for a larger 5-digit euro sum, and a few person years. However, the decisive question in this approach, as with most others to be asked, is: what happens once the pilot project has been successfully implemented. In the best case, it will have been shown that your enterprise can handle SOA technically and that this could result in savings potentials in IT. But you still do not have a business sponsor, whom you would need for financing the really large modifications for an integrated SOA. The case is therefore followed seamlessly by the next idea.
Financing of the SOA through savings in the IT department: EAI was already sold with the promise of being able to save costs in IT. If one then considers, however, what share of an IT budget can be saved with a SOA, optimistic estimates are limited to 5-10% ¹ of the IT budget. For an IT budget of a large enterprise totaling approximately 200 million euros, you could invest two to three times the amount saved, i.e. between 20 million (5% times two) to 60 million euros (10% times three) in a SOA-compliant transformation of the IT function. This approach will only seldom pass the decision-making boards, because savings of 5-10% of the IT budget today are not considered significant enough to justify risky investments of 20 – 60 million euros. In addition, this dimension is usually not even sufficient for the complete SOA-compliant transformation of an application landscape with construction costs of one billion or more. The dimension is sufficient, however, to provoke questions from the business side on what could be achieved more feasibly with such amounts.
Powerful sponsor: Another possibility is to find a senior manager, who simply orders SOA and declares himself responsible for the investment means in the board of directors. Most of the big SOA success stories that you will find in the current literature are based on this procedure and were initiated in the late 1990s. The business value was seldom calculated ex-ante. Business arguments were used to start a SOA program. In the case of a Swiss bank, after the SOA was established, ex-post calculations showed that the SOA actually brought about effects. However, not necessarily those that were originally intended. If such a program survives long enough, these are the cases in which the entire software landscape of a company is converted.
Pilot project with real business value: The fourth variation is oriented more toward business. The IT is convinced that SOA is positive and desirable and looks specifically for a problem among their customers that can preferably be solved with SOA. Such projects are then given such a high budget that the share for the SOA pilot project in the overall project budget shrinks by comparison. The pilot project is then successfully implemented. The only problem then is frequently the next project approval. A complete conversion of the entire application landscape to SOA is hard to achieve this way.

Frequent adoption paths driven by technology

The SOA adoption path via a SOA Competence Center (SCC) can frequently be observed in all of the above cases. It breaks down as follows:

Pilot project: First, a pilot project is implemented. The source of the budget is irrelevant (business or IT).
SOA Competence Center: After successful completion of the pilot project, the knowledge gained is concentrated in an SCC and made available to the entire enterprise for further projects.

If additional pilot projects come about, then additional SOA projects are implemented. It can be observed quite frequently, however, that no further large-scale SOA projects follow, so that the original goal of a comprehensive SOA initiative is never achieved.

Why is it dangerous to motivate a SOA initiative by technology alone?

That brings us to the central dangers of all SOA adoption paths that are driven only by the IT department, and which are implicitly rooted in the belief that the IT department would like to use SOA to acquire a comprehensive software landscape with "nice technical properties". Instead, it can be observed that the SOA issue gradually loses momentum or becomes dormant for a few years after the introduction of a SOA competence center.

This has dire consequences if SOA was driven by the IT department and the rhetoric of business was used only pretentiously. The arguments for SOA were often formulated only using "pro forma business language". Reference is made to abstract "flexibilization" instead of naming a specific field in which money can be earned with services – for example the full automation of simple business processes according to the 80/20 rule, which for example is a part of the "factory approach" ² in the financial industry. The IT department is then surprised that it gets bogged down on the way to a comprehensive SOA, when it actually is following its own agenda and not really pursuing business value.

The remainder of this article will therefore deal with the question of how you can organize SOA governance to create as much SOA as the business needs. It makes sense to give up the idea that this always has to be the comprehensive, maximum solution.

3. Business-driven Adoption

In their book about business models in 2010 [Kagermann+2006] Kagermann and Österle describe e.g. the following building blocks for innovative business models, which enterprises can use to increase competitiveness and profitability: integration of value chains, customer self service, reduction of transaction costs, integration of enterprises in so-called eco systems, fully unattended processing and automated business processes for standard cases, outsourcing of sub-processes (BPO), use of Internet trade platforms, eBay-like business models, reduction of complexity in the IT application landscape.

This list is intentionally neither complete nor sorted: You can now ask yourself, what all these components of business models have in common. First of all, that they do not contain the word SOA. And then, that all of them are considerably easier to implement with the help of a SOA. Therefore, this Section will discuss how to implement a SOA that supports just such projects.

The question of how you can implement a SOA automatically leads to the question of how you can effectively align the IT function with the business requirements. If this causes the SOA to be recognized as a suitable means of implementation, that will also automatically result in the necessity and creation of a SOA governance. But not vice versa.

Examples for use of SOA

Before entering into a theoretical explanation of how IT governance and SOA governance control the use of a SOA, two examples with locally limited effectiveness and more extensive effectiveness will be presented in order to arrive at a classification of business initiatives, which frequently are also SOA initiatives.

Example: Process optimization: Integration of a broker platform

The integration of a broker platform is presented here as an example for a SOA pilot project from the insurance industry. One way that insurance companies offer their products is through insurance brokers. Since brokers work together with more than one insurance company, it makes sense to offer the services required by insurance brokers on an Internet platform, so that the brokers can communicate via the platform with more than one insurance company. From the perspective of a single insurance company, this results in the following graphical representation.

Figure 2: Integration of an insurance company in a broker platform. The graphic shows the integration from the perspective of an insurance company that provides data to the broker platform. There are several insurance companies, all of which provide data and receive inquiries. The platform is used by many brokers.

The insurance company supplies the broker platform with information about contracts, tariffs, terms and conditions. The insurance applications and claims are transmitted electronically and paperless by the brokers to the respective insurance company.

It is apparent that process optimization is taking place here. And it is also quite apparent that EAI or SOA technologies are being used here beyond the borders of a single enterprise ³. Process optimization is achieved by speeding up processes and avoiding paper communication. Therefore, you can calculate the profits from such an initiative using a conventional profitability analysis and consider whether the use of SOA technology – limited to the field of the respective process optimization – will pay for itself. Initiatives of this type can be implemented by mid-level managers from business and IT, because they are locally limited and calculable.

Example: Comprehensive decomposition

The following graphic shows an example for a large-scale change program, in which a SOA could be quite useful:

Figure 3: Deconstruction of an insurance company into process blocks. The Enterprise Service Bus links various process blocks with each other.

The entire process landscape, e.g. of an insurance company, is deconstructed into process blocks, with radical intervention in some of the well-worn processes. In the entry processing, simple cases are clarified immediately and closed. This process can be carried out automatically if an automated process is used in certain cases instead of an employee, who processes a scanned document.

In addition, the decomposition into process blocks also means that single blocks, such as entry processing or the claims process, could be completely outsourced or centralized and combined as a shared service in a group of companies.

Such models bring about large-scale changes in an enterprise and often provoke resistance. Change processes on this scale therefore have to be decided and implemented by top management.

This type of change is often referred to as "deconstruction and reconstruction" [Bieberstein+05] ⁴. Similar to the acquisition of enterprises by financial investors, the enterprise is deconstructed into blocks and then it is decided how the services of each block can be rendered most economically. It is relatively clear that exactly one comprehensive SOA supports such an approach. If SOA is to be profitable throughout the enterprise, then only with such models, since it was already shown above that a comprehensive SOA will never be worthwhile only through savings in the IT budget. In order to finance a comprehensive SOA, the business models must first be "re-invented" so radically that this could have effects in the three-digit million or even billion range in a large enterprise.

Classification of SOA initiatives

Now that we have seen examples of SOA initiatives on all three scales, the following graphic again summarizes this classification.

Figure 4: Classification of SOA initiatives

Optimizations of the IT, which are often linked to the technology-driven adoption paths described in Section 2, have limited scope. They are listed in Figure 4 under micro governance. They frequently run the risk of fizzling out. Optimizations of business processes, such as the integration of a broker platform described above, lead to additional SOA islands, but not to a SOA that covers the entire application landscape. These are referred to here as operational layers. The cases in which a comprehensive SOA can be financed are cases in which entire enterprises and business models are totally remodeled and optimized by means of deconstruction and reconstruction. In the implementation of such initiatives, we will use the term macro governance. The optimization of IT and, for example, the administration of the IT assets and technical services will be referred to as micro governance. The term governance was already used in the illustration above, but not explained. This omission will be corrected in the following Section.

4. SOA Governance

The term "governance" appears in the title of this article and was touched upon in the preceding Section. Therefore, it should be properly introduced:

gov•ern•ance /vnns; NAmE vrn/ noun [U] (technical) the activity of governing a country or controlling a company or an organization; the way in which a country is governed or a company or institution is controlled.

Source: Oxford Advanced Learner's Dictionary, http://www.oup.com/elt/catalogue/teachersites/oald7/?cc=global (site visited on 12/18/2006)

Literally, IT governance means "governing" or "controlling" the IT function of an enterprise. In the following, it will therefore be shown that SOA governance, i.e. controlling of the SOA activities for the operational and technical layers, is practically inevitable if you implement effective IT governance. Therefore, let us come back to IT governance. Various definitions of IT governance exist. The definition from the ITGI ⁵ will be used here.

Definition: IT Governance (ITGI)

IT governance is the responsibility of the board of directors and executive management. It is an integral part of enterprise governance and consists of the leadership and organizational structures and processes that ensure that the organization’s IT sustains and extends the organization’s strategies and objectives.

Source: IT Governance Portal, http://www.itgi.org/overview.htm (site visited 12/28/2005). See also ITGI: Board Briefing on IT Governance, IT Governance Institute, 1998

From the point of view of IT management, one of the prerequisites for effective IT governance is management of the control systems depicted in the following illustration.

Figure 5: IT control systems of the meta group [Meta02]

You can now ask yourself, what the above has to do with SOA. Deploying a proper IT governance system means aligning IT with business. If the business side has the above-mentioned strategic demand for projects and plans that can best be implemented with a SOA, then SOA will happen automatically, so to speak. In any case it then does not have to be "forced", but instead can be introduced as a suitable means for implementing business goals in such a manner that it is created in the background – while the business goals remain in the foreground.

IT strategy and SOA

If the IT strategy should therefore support the business strategy and if the business strategy requires the kinds of projects listed in Section 3, it will be easy to anchor a SOA strategy in an IT strategy. It will be shown here that an integration strategy is always a part of an IT strategy. In any case we need a brief explanation of what a strategy or IT strategy is.

Definition: Strategy

A strategy takes a vision or objective and bounds the options for attaining it.

Source: [Gartner03a]

A corporate strategy develops paths of action based on a goal or a vision of the enterprise that are supposed to achieve the business goal. In this process, it is important not to develop a trivial strategy ⁶. Actions and measures for which there are no alternatives are not real strategies.

The following illustration shows fields for which a CIO has to have answers when an IT strategy is developed. The illustration is to be understood as a checklist. It is not necessary to have information in each field. But it is necessary to at least determine whether it is useful to fill in the individual fields.

Figure 6: Strategy matrix of the Gartner Group (Source: [Gartner03b]). The matrix shows sample questions that can be asked in order to develop a strategy. The rows contain typical dimensions of a corporate strategy. The columns are fields for which there should be information in an IT strategy.

You can and must work through these fields regardless of your business strategy ⁷. The matrix is actually a checklist. Comments on the integration strategy (alias SOA strategy) are an integral part of any proper IT strategy. SOA is included in every well-managed IT department as a background activity under the title Integration Strategy and Integration Architecture.

The IT account managers and the IT enterprise architects have an intermediary function. They have to talk about business models to the customers of IT, the business side, in the language of business and recognize the resulting opportunities, for example to advance IT itself in the direction of a SOA.

Architecture governance and SOA

The actual goal of this Section is to show how you can develop a "good" SOA governance. If architecture governance is regarded as the implementation of the binding architectures defined in the enterprise, and if a SOA is simply one of many technologies that exist in the enterprise, then SOA governance is nothing other than architecture governance, although specialized in the issue of SOA. As with architecture governance, it is necessary to implement standardized platforms. You have to prevent the occurrence of technological and procedural islands and you have to manage a portfolio of services as one would otherwise manage a portfolio of technologies, projects or applications.

SOA governance can be divided into several layers, as already described in Figure 4.

The strategic layer: This layer ensures that IT is integrated in the major issues of the enterprise, is familiar with the challenges to be met and tackles them together with the business side. IT can ensure that the SOA part of this is implemented. Close cooperation with top management is necessary for the strategic layer.
The operational layer: Bundles the activities connected with optimizing business processes, but not the reorganization of the entire enterprise. Optimizations at the operational layer are carried out in cooperation between department heads in business and IT. The issues to be processed there are discussed in more detail in the next section.
The technical layer: Provides what many tool manufacturers understand SOA governance to be, i.e. ensures technical integrity. The main concepts here are: repository, deployment, operation or versioning of services.

The strategic layer of SOA governance therefore does not differ from other architecture governance issues. Some authors reduce SOA governance almost to the technical layer and parts of the operational layer. Consider, for example, the following definition:

Definition: SOA Governance

Process of enforcing organizational policies and standards and tracking the life cycle of each service within a SOA deployment.

Source: http://www.bitpipe.com/tlist/SOA-Governance.html (site visited on 12/6/2006)

This form of SOA governance, which is more technical in nature, has no considerable effect on whether budgets exist for SOA and whether the SOA deployment can be further expanded, because the SOA is in agreement with the business value. Even the operational layer does not enable the creation of a SOA that covers all applications. In this context it is interesting that most of the SOA literature only treats the technical layer, or in terms of business value, only goes as far as the operational layer. The strategic layer is seldom treated intensively, although purely on grounds of the budget dimensions this can be the only key to a truly comprehensive SOA.

Special issues of operational and technical SOA governance

Nevertheless, the operational and technical aspects should not be totally ignored here. Once the strategic issues have been clarified, it is comparatively easier to deploy governance mechanisms at the operational and technical layers precisely to ensure that SOA can grow in an enterprise to its advantage. By way of example, various authors list the following decision-making fields [Holley+06, Malinverno2006a, Malinverno2006b, Mitra05, Windley+06], for which decisions have to be made:

Which organizational unit manages the SOA: Similar to the procedure for a project portfolio, a central unit is needed that oversees all applications for new services and can prioritize and manage them. This also includes monitoring of service level agreements to supervise the use of the SOA in operation.
What services should be implemented?: In order to decide which services to implement, it is first necessary to determine which services already exist. To this end, one has a repository of services and lists of service candidates. The first thing to check is whether the candidates are suitable. This also includes determining whether a service can have the potential for re-use and, for example, whether it is shared by several organizational units or used exclusively by one unit.
Which services will be implemented first?: Analogous to projects in project portfolios, the implementation of services also has to be prioritized.
Who will finance the service?: Also analogous to projects, financing of the implementation and later maintenance of each service must be clear.
Who will be the owner of the service?: Responsibilities must be defined for the maintenance and development of each service. The operational responsibility must be clarified.
What non-functional requirements does a service have to fulfill?: Agreements on maintaining non-functional properties, such as availability and performance, must be concluded for services.

It is fairly apparent that one could replace the word "service" with "project" or "software system". The services of a SOA are operating components like many others that are subject to analog administration (governance). Once procedures have been established for clarifying all of the above issues, tools can be procured for support in processing the issues. As so often, however, the tools (as the expression of HOW) should come after the decisions of business strategies, operational procedures and budgets (as the expression of WHAT).

5. Summary

This article has shown that initiatives for the implementation of service-oriented architectures can get bogged down if they are only technically motivated. The primary goal, therefore, should never be to introduce a SOA for the sake of technical progress.

Instead, as the responsible person in an IT function, you have to talk to the business side about business initiatives and be perceived as an equal partner in the creation and implementation of innovative business models. Only then is a service-oriented architecture that covers all applications possible as part of a strategic business initiative. However, the focus should still be on the strategic business initiative.

SOA can be the result. In many modern business models, you will automatically go in the direction of service orientation. The key is not in the SOA, however, but rather in the implementation of the business initiatives.

If you want service orientation, you need to concentrate less on SOA and more on business models. There is a very good probability that you will then obtain a service-oriented architecture.

SOA governance is nothing other than a sub-area of the normal IT and architecture governance. If you have IT governance processes that work, it is relatively easy to integrate the technical aspects of SOA governance in them as well. However, even a good technical SOA implementation and SOA governance will not replace a proper IT governance.