Training - An Introduction to Enterprise Integration

What is EI?

Enterprise Integration (EI) is a business computing term for the plans, methods, and tools aimed at modernizing, consolidating, and co-coordinating the computer applications in an enterprise.

EI Levels

Data Level

Integrates and synchronizes data stores to help them support the sharing of information among applications in a more efficient manner. Typically, this approach entails extracting information from one database and updating other database(s), while simultaneously processing information between them.

Application / interface level

This links applications at their individual interface levels. This often requires implementation of an adapter between the application interfaces and the message broker. message brokers are typically delivered with ‘out-of-the-box’ adapters for popular packaged applications such as SAP. Application-level integration is a form of EI that integrates different applications on the functional level. A function (which can be viewed as a business logic) in one application causes the execution of another function in a different application.

Process level

Process level integration takes application level integration a step further, allowing a business model, which defines the flow of business activities, to control inter-application processing links. This often leads to modeling and implementation of business process flow (or business process management), beyond the logic embedded within the existing applications. Conceptually, this is a layer above the application integration layer that deals with business processes rather than data and interfaces.

Middleware Types

Data Middleware

Data Middleware includes data gateways, data replication, as well as extract, transformation, and load (ETL) tools. They integrate applications by giving direct access to their databases, or through file transfers.

Message oriented Middleware (MOM)

Message oriented Middleware (MOM) facilitates the exchange of messages between applications allowing them to operate independent of each other, and yet remain loosely coupled. MOM provides an API that abstracts the application developer from having to deal with the intricacies of lower-level communication chores including guaranteed message delivery, automatic error detection, and retry amongst others.

Application servers

Application servers include Web-application servers for monitoring purposes. Object-request brokers offer a centralized, object-oriented integration framework or platform that provides built-in support for environment management and transaction services. This type of Middleware either partially or completely insulates the application program from the operating system. Application servers are used in a more synchronous environment where the primary focus is development of new applications rather than integration of existing applications.

Message brokers

Message brokers are high-level Middleware that provide a set of functions that is specifically related to connecting application programs or Middleware products that differ from each other. The message broker provides additional functionality including a variety of adapters, publish or subscribe communication techniques, intelligent message transformation including syntactic conversion and semantic transformation, content-based routing, process flow automation, and business process modeling.

Data Integration Features

• Data movement: This focuses on the transfer of data (directly) from one database to another.
• Data transformation: This processes and transforms data to fit or match the target database's format.
• Data services: This has the ability to transfer data through service.
• Data synchronization: This has the capability of enabling the same data to be made shareable amongst different systems.
• Changed data capture: This has the capability of loading differential change.
• Data quality: This has the ability of performing data standardization, constraint-checking, data validation, and rule-based cleansing.

Data Integration Advantages

• It is relatively inexpensive.
• It represents a proven approach that has been used successfully multiple times.
• The technology to implement this approach has been available for quite some time.
• Appropriate expertise is more readily available than other types of Data Integration.
• Since Data Integration does not involve any traditional kind of programming like Cobol or C++, it can produce results faster than them.
• In some scenarios, the only practical integration option is the DBMS interface.
• Data Integration is useful in situations where there is a need to wrap existing data sources to allow integration through encapsulated interfaces, like a data abstraction layer for example.
• It does not require any knowledge of the application code of Legacy Applications.

Data Integration Disadvantages

• Data Integration does not solve the problem of integration of business events, since a business event in one application does not trigger an execution of business logic in another. In that sense, it can be viewed as more of a stopgap arrangement.
• Data Integration does not scale well for OLTP type of application integration.
• Data Integration can mask important issues with the enterprise information systems since it accesses data without invoking the business logic in the application that ‘owns’ the data.
• In many cases, Data Integration may require a detailed knowledge of data semantics as well.
• Data Integration has been known to result in lower levels of integrity when business rules and data-owning (and validated) applications are bypassed.

Key Points to Real time Data Integration

• Business processes generally requires real time data processing.
• A data replication mechanism should be in place to copy information in more than one application.
• In real time Data Integration, business logic is applied to the data.
• The push mode of Data Integration is used during real time Data Integration.
• Data throughput is relatively small in the case of real time Data Integration.
• A two phase commit can be supported by real time Data Integration.
• Real time Data Integration is a costly option when compared to batch mode Data Integration (to be covered in the next section).

Key Points to Batch mode Data Integration

• Business models commonly include bulk processing of activities.
• Data transfer can be scheduled in a predefined time for it.
• Generally, business logic is not applied to the data in the case of a batch mode of Data Integration.
• This is a pull mode of Data Integration.
• Data throughput is comparatively higher in the case of a batch mode of Data Integration.
• Legacy applications can be supported very easily on it.
• This is a cheaper option when compared to real time Data Integration (covered in the previous section).

Data Integration approach

• Data consolidation: This is a technique that integrates data from multiple source systems and integrates them into a single persistent data store. This approach is generally adopted in data warehousing.
• Data federation: This provides a single virtual view of one or more source databases. Enterprise Information Integration (EII) is an example of a technology that supports a federated approach to Data Integration.
• Data propagation: This copies data from one location to another. This event-driven approach is usually operated online.

Data Integration Styles

• Extract, Transform and Load (ETL)

  • It works by extracting data in bulk from a source system, then bulk transforming it, and then finally bulk loading it into the target system.
  • When it is used, the collection of data level records is medium at the most.

• Enterprise Information Integration (EII)

  • Providing real time 'read and write' access
  • Transforming data for business analysis and data interchange
  • Managing data placement for performance, currency, and availability

• Enterprise Data Replication (EDR)

  • Same instance of data is replicated across multiple data repositories
  • Employs effective methods for disaster recovery and data mirroring
  • Ensures an exceptional availability of data as there are multiple repositories of similar data repository
  • Uses an extremely effective technique for load balancing

• Master Data Management (MDM)

  • Central repository of master data
  • Single source of core business entities
  • Reference data to transactional applications
  • Useful for cross selling and up selling of products
  • Core business entities can be modified at a single place
  • Can be used to implement better CRM

WebSphere MQ Features

• Provides cross platform support
• Has the same messaging API on all platforms
• Guarantees a one-time delivery
• Employs a two-phase commit
• Enjoys a wide EI industry support
• Has the capability of triggering an application to process a message
• Capable of grouping messages
• Capable of performing load balancing

Message Broker Features

• Facilitates interaction among applications
• Acts as a shareable component that sits between information sources, or message publishers
• Sometimes, it also includes basic message-oriented Middleware and database Middleware
• Supports functionality of a variety of adapters
• Facilitates intelligent message transformation, and content-based routing
• Supports process flow automation and business process modeling

Message Oriented Middleware Features

• Can combine many input messages into a single message (many to one)
• Can split one input message into multiple messages (one to many)
• Employs a complex conditional logic, especially across repeating records
• Can hold or maintain state and variables, both within as well as across messages
• Processes communications synchronously
• Can perform data related activities beyond transformation and substitutions such as data de-duplication and data cleansing

WebSphere Message Broker Features

• Distributes information irrespective of its type
• Provides a mechanism for connecting, routing, and transforming business data from a variety of transports without a need to change the underlying applications generating the data
• Is built upon the WebSphere MQ
• Reduces the number of point-to-point interconnections
• Routes information in real time based on topic and content
• Validates and transforms in-flight messages
• Routes messages based on business rules which can be reevaluated
• Improves business agility by dynamically reconfiguring the distribution of information
• Provided access control in order to securely deliver personalized information

What is Service Oriented Integration (SOI)?

• SOI is an architectural style which requires a service provider, a service requestor, and a service description.
• SOI consists of a set of architectural principles, patterns, and criteria which address characteristics such as modularity, encapsulation, loose coupling, separation of concerns, reuse, and composability.
• SOI is in many ways a programming model, complete with standards, tools, and technologies such as Web Services.

The relationship between SOI and SOA

• SOI integrates both SOA and non-SOA applications: It exposes existing business functions as services using the service adaptor pattern.
• SOI integrates non-SOA applications as if they were services: It allows coexistence of architectures during transition using the service proxy pattern.
• SOI does not require an enterprise-wide initiative: Individual projects, regions, or LOBs can adopt SOI and show business benefits. The wider the adoption in the enterprise, the greater the business benefit. 4.Incrementally adopt SOA after SOI adoption without a significant penalty: Loose coupling of SOI components makes integration less brittle.

SOI Advantages

• It lowers the cost to add new interfaces and modify existing interfaces
• It easily, as well as incrementally expands integration services by managing to grow capabilities over time
• It works with single as well as multi-vendor integration environments
• It supports both synchronous as well as asynchronous interfaces
• It starts with the software an organization probably owns already, along with their existing integration architecture
• It begins return on investment with a single project itself and subsequently increases with every additional project
• SOI helps lower maintenance costs

Web Service Features

• Web Services is an implementation technology that is standards based, offering the promise of interoperability.
• Web Services can be used to implement multiple integration styles.
• Many of today's Web Service integration technologies use the point-to-point integration style, which has a number of disadvantages.
• Nowadays, quite a few hub-and-spoke Middleware have adapters that have Web Service interfaces.
• Web Services is often selected as an implementation technology for Service Oriented Architectures (though SOAs can be implemented using other technologies like XML as well).

Enterprise Service Bus(ESB) Features

• The ESB is a specific pattern of connectivity wherein multiple ESB configurations and topologies are possible.
• The topology you implement should be optimized to meet your unique business requirements.
• Your topology can change as your business requirements evolve.

Business Process Management (BPM) Features

• The optimization and automation of business processes allows a new level of flexibility and responsiveness to the changing requirements inherent in an on demand business environment.
• Through monitoring of the automated business processes, both IT performance and business goals can be measured, providing a rich capability for both the business executives and the IT executives to understand, evaluate and improve the organization’s capability to deliver business function that satisfies business requirements.
• A business process is a series of tasks executed in a specific order that an organization follows to achieve a larger business goal.
• WebSphere Process Server is a process engine that provides a hosting environment for business processing. It provides support for the Service Component Architecture (SCA) programming model. 5.WebSphere Process Server includes support for both Web Services Business Process Execution Language (WS-BPEL) based process flows and business state machines. It supports the integration of business rules and for the incorporation of tasks that are carried out by users (human tasks) in a business process.

What is Process Integration?

• An abstraction layer that synchronizes business processes by integrating business applications and human activities
• Integration that takes application level integration a step further by allowing a business model, which defines the flow of business activities, to control inter-application processing links
• A process that allows modeling and implementation of business process flow (in a way similar to business process management) outside the business logic embedded within existing applications

Process Integration Features

• Integrates enterprise with key partners, suppliers, and customers
• Responds swiftly to any customer demand, market opportunity, and external threat
• Automates business processes involving systems and people
• Offers easy exchange of real-time information across multiple participants, platforms, business applications, and organizations
• Makes optimal use of human resources to support business processes and deliver enterprise services
• Manages business process logic in accordance with business design in separation from implementation and integration details
• Increases business agility and flexibility by aligning business models with strategic objectives