XML Europe 2002 logo

Technical Challenges in Developing and Using Web Services for Enterprise Applications

For Web Services Architects, Developers and Users

Abstract

Web Services have captured everyone’s attention in the computer industry during the past year. Not all of the Web Services is hype. Although everyone is trying to take simple usage scenarios (for which web services may not be appropriate or necessary) to explain the concepts, it is important to discuss the technical challenges that face a developer or consumer of web services while developing enterprise class applications. Some of these challenges deal with reliability, failure-management, recovery, administration, tracing and logging of activities, business intelligence information capture, distributed transactions, heterogeneous security, distributed error/exception handling and so on. Many enterprise applications require most of these capabilities for business reasons. Most of the web services implementations and standards do not directly address these requirements. However, there are many existing solutions and technologies that allow developers and consumers of web services to meet many of these needs without having to develop everything from scratch. The biggest challenge that the developers have to deal with comes from the fact that Web Services are distributed by nature, asynchronous and run on top of HTTP connection (which by itself is not very reliable and passes through many security and firewall servers). A critical business application may suffer severely, if the application is not equipped to handle random failures, loss of services, unpredictable round-trips and distributed dependencies. With careful planning and right technical approach, such situations can be managed, even though they cannot be avoided. If one can find a comprehensive model that the operating engines for web services can understand and act upon them, then they may be able to solve their problem.

In this paper, we will first discuss a list of issues that are relevant to web services. Then we will focus on how XML based approaches and modeling techniques make it possible to address these issues both within web services environment as well as outside. We will also discuss how existing infrastructures and technologies can help in developing web services frameworks that can meet all the requirements of an enterprise class application. Some implementation details for application developers will also be discussed especially focusing on Java technology for achieving the desired goals. We will also take a closer look at web services as ad-hoc application integration technique that benefits from the solutions discussed earlier in the presentation.


Table of Contents

1. Web Services: A Fresh Look
2. Developing Application With Web Services
3. Web Services: XML and Standards
4. Developing and Integrating with Web Services Framework
5. Conclusion
Acknowledgements
Biography

1. Web Services: A Fresh Look

Web Services have been described and defined in many different ways by different parts of the Industry. Many of those definitions are accurate and represent the versatile utility of Web Services concept. In this section, lets take a fresh look at the definition of web services and see how all the other definitions point down to the same core.

Web Services are interfaces described with XML, found with XML and invoked with XML. This is as general a definition as we can get to. Now, one thing to note here that we are not calling web services as component models because they are not. They are interfaces to your own areas of expertise which could be developed with any component modelling technique or none what so ever. In other words, web services is an agreement that allows programmatic (pseudo automatic) access to expertise from different providers. This opens up a possibility for developing software applications that can learn, adopt, interact and cooperate with each other without new code, new compilation and deployment. This is what the browsers could do for presentation for a long time. Now applications are getting ready to do the same. Except that this is a bit more complex process and that is why we see different artifacts of XML and definitions of new standards (WSDL, SOAP and UDDI at the moment but this is not the end of it by any means).

A huge assumptions, which everyone in the industry has to note and remember any time they are looking at Web Services, is that they are meant for inter-operability. They must not assume application environment or the operating environment in which the producers as well as the consumers of Web Services are operating. If one slips off of this thought process and get caught with the tidal wave of Web Services hype, one is very likely to apply web services notion to a problem not remotely relevant for Web Services.

Now we can look at all the different definitions of web services. For a software vendor, it means opening up new APIs for allowing their clients to take advantage of their application capabilities without specific APIs and libraries to deal with (as an example assume JDBC described as web services and all database vendors implementing an access point for this kind of access, no more fuss about the drivers and the platforms and different behaviors on different platforms). For internet service providers, this means access to many of their services for a whole new user space (think EBay for your private auctions). For Portal vendors this means standard access to client information for unified visualization with great possibilities of personalization. For application integration companies it means adapters for any and all applications that expose themselves as web services (reduces cost for custom development). For b2b company it means open exchanges that anyone can participate in (provided the business process allows them to enter). These are just some of the meanings and benefits that different part of the industry derive from adopting Web Services model. And that explains the hype. All we have to do is to provide software and services for making it real.

One note before we close this section, XML is the basis of all the agreements that are being made in the world of Web Services. Even though Web Services could have been developed with any mark-up, it is XML which has raised the level of understanding of easy inter-operability in simple terms in many minds throughout the industry. Therefore, I would like to add this as a closing statement, "XML is the foundation of Web Services and the catalyst that has made all the ingredients mix well together for the first time".

2. Developing Application With Web Services

Having looked at what Web Services are supposed to mean, let us now examine what it takes to develop applications with them. As a part of this section we will see how Web Services simplify some application features while add other complexity. We will also examine what those issues are that the application developers or software vendors need to be aware of before really deploying a mission critical application that is developed with Web Services.

Developing applications for Web Services: Developing applications for Web Services access is probably an easier task than developing one with Web Services. When you are developing application that needs to be exposed as web services, the primary considerations must be the granularity of access, security, sessions, transactions and performance. Many of these considerations are the same as what web based applications have been worrying about in the past anyway. Therefore there is virtually no sharp transition for internet application developer while they are trying to expose their functionality as Web Services. The only change that they now need to worry about is compliance with the standards and getting information that they need over SOAP call instead of HTTP. I would like to suggest that these developers is not to compromise the level of functionality that they support today.

Web Services standards are not fully developed and all the real application needs are not satisfied with any vendor's software. Some vendor's have their own way of addressing their issues but one has to ask this question every time they pick a solution: If my goal is inter-operability with Web Services and I buy into vendor specific approach, I am tied down to the same software at both ends. Does it really make sense to use Web Services then and lose other features like performance and language level flexibility?". This is a critical question to ask because there is always an additional cost associated with Web Services and that is multi-level of transformations and mappings. And, if you are not even gaining inter-operability at additional cost then why take the jump into Web Services now. It may be wise to wait until all your requirements are met with standards. Even though others may have different opinion but in my personal opinion, taking a step towards exposing our functionality as Web Services may not even require code changes or new deployments of your own applications. Therefore, it is not a huge jump to take after the fact (after you have developed, deployed and tested or even released your software).

If you are hosting this software, then there is another dimension to consider. You may need to worry about tracking and metering the use of your services (typical web server logs may not help here). You need application server level support for tracking Web Services that you expose from your hosted environment.

Developing Applications With Web Services: Those who are planning to build with Web Services have a number of things to consider simply because they are now placed in the wonderful yet complex world of distributed computing even though all they they did was to call an API (stub for web services) in their program. They need to understand and know way more than that. This is where I would like to add that when you develop with Web Services, you need application administration role more than ever before. This is also necessary to keep the end user away from learning all the geeky details that they don't care about or would need a degree to learn. So you have an application administrator to help you with your application administration.

The role of this administrator may simply be to figure out what happened to the web services that are being used in your applications. They may be down, unreachable, changed, redeployed on a different node or you may have lost the license for using them. The application users would not be able to get the actual information simply by the error they see. On the other hand, now you are dealing with distributed error handling. This means that you don't always know what to do with errors and issues other that just saying that something changed.

You need an architecture to handle many of the issues we have discussed so far. Application infrastructure vendors may provide you with one when you buy their software but you need to know what it supports and how. Web Services being XML based provide this wonderful ability to application developers to make decisions at run-time. For example if your web service start failing or become less responsive, you can automatically switch to an alternate. This will enhance user experience to a great level and you application downtime to a minimum. Since you now know your application users better than any Web Service providers, it is your responsibility to transform any/all the information that you receive for your end user consumption (even if it is an error message). This means you are also dealing with and architecture for message interpretation and translation. Having an architecture will also help you in detecting the failures and alerting your application administrator for immediate action even when you can use an alternate service. The business user would like to know when and why you switched to an alternate and your framework can help you capture that information. This architecture may also help you improve the performance of your applications when they are being used by many users. One can cache the web services responses in their own choice of cache or even a database for certain period of time and serve all the users from cache until the expiration time. Using cache will add other set of management, cache management (however transparent the solution may be) to the application administrator's tasks.

What software vendors are getting ready to provide you is a generalized solution but it is still application architect's responsibility to use what the vendors provide to meet their application's needs. Businesses still have the same requirements and expectations from their business applications. Web Services may offer them flexibility but their actual application implementation is what takes full advantage of Web Services. A bad application architecture can not be helped with Web Services. It is important to note that Web Services do not reduce application developer's responsibilities for a good application design. What it helps in is to make these applications flexible and perhaps (with enough foresight) reduce the cost over the life of their application by reduced maintenance, customization and upgrade costs.

As we discussed in this section, Web Services can present many challenges to the application developers in the initial phases of design and development. Over a period of time, the benefits that Web Services offers in terms of adaptability with business policy changes, ease of application integration and allowing to reduce overlap between the applications, can pay for much more than the initial investment.

3. Web Services: XML and Standards

As we have seen throughout this paper, Web Services main selling point is inter-operability, ease of information integration and for the first time in an easy to understand and platform and language neutral fashion. All these attractive capabilities are possible because Web Services are laid on top of XML as their foundation. As we all understand, XML is just a mark-up language and the lowest common denominator for all of the Web Services and related standards.

As we start to scan the Web Services landscape, we find that there are three basic operations that Web Services need to address from the standards perspective and they are:

  • Description (WSDL)

  • Discovery (UDDI)

  • Invocation (SOAP going to XML-RPC)

There are going to be many auxiliary standards but they are all going to play supportive role. Among the existing standards there is one common standard and that is XML and XML Schema. Even UDDI and use of it is not absolutely required for Web Services but it is important for the future of Web Services and for the vision that there will be a marketplace of services that will be found from such registries. Then we will really see the full power of having a standard that allows all programs to request for specific information.

WSDL is currently under development by W3C and there is no standard but the fact remains that WSDL 1.0 note that IBM and Microsoft published a while ago has become de-facto standard. WSDL as it is defined today is very primitive and useful only for very simplistic scenarios but through it's extensibility mechanism it can allow you to define any interface with it (it is a different issue if that interface will not be inter-operable with those who do not understand those extensions. It provides the basic envelop, none the less.

SOAP 1.1 and now 1.2 has been developed by W3C and it is now moving along with XML-RPC. SOAP is largely defined and almost every vendor has an implementation for this but in the current state, it lacks some important application requirements (being addressed by the working group) namely session, transaction and security.

There are many security considerations that W3C has for Web Services and Web Services architecture group is largely responsible for selecting which standard (XML based of-course) will be useful for Web Services standards to depend upon.

Does not matter what standards are defined for Web Services the basic notion behind all of this still is inter-operability with platform and language neutral access to information. There are perceived standards that many application vendors are working on namely J2EE and .NET. Actually these are implementations of Web Services and do not affect anything in Web Services. These standards are within a particular language space for inter-operability among those who stay with a single language environment (J2EE for Java language). Web Services standards cut across all of them and go well beyond. Web Services provide true independence from language and operating environment while exchanging information and allow applications to move from one operating system to another one with little or no effort (as far as web services invocation/publishing is concerned).

4. Developing and Integrating with Web Services Framework

One important piece of information that all application developers ought to remember is that Web Services is an evolutionary progress in computer industry (perceived as revolution, sometimes due to excessive hype). But, application developers now have a new tool to evolve with and even invent new and very creative ways of using Web Services. We have also discussed some strong motivation for application developers to consider developing an architecture that should support their applications use of Web Services. In this section we will discuss some development considerations (tips) which will allow Web Services users to make their applications more modularize and be able to handles the underlying issues with distributed environment better.

Applications architects must consider complete isolation of their applications from the web services invocation, even at the interface layer. This means that they should plug a framework in the middle. As discussed above, this framework needs to provide with the capabilities that applications and businesses require for the large part. This framework should also address complete management of client application services. This framework should/could capture the following:

  • Independent interfaces and then static/dynamic binding to actual web services

  • Management framework and automated recovery to the extent possible

  • Notification/messaging framework

  • Logging/Auditing framework

  • Complete error management and handling

  • Translation/transformation framework for ease-of-use and and end user convenience

  • Asynchronous/Synchronous transport adaptation without burdening the application

  • Metering and business relationship tracking

  • Caching

  • Performance management and tracking of the application while calling web services

  • Call optimization by completely optimizing out the XML path (provided you know that you are dealing with compatible system)

  • Resource and service sharing among applications and users

  • Personalization in coordination with central user preference management store at web services level (sometimes it may be necessary to provide user specific parameters to the service provider in order to provide effective personalization)

These are just some of the features that are needed behind the scene from web services invocation. The actual Web Service invocation may happen at the end and this is where stub generation facilities provided by different vendors may be useful. Make sure that many of the application server vendors will provide with the facility that is compatible with their deployment platforms. In this situation, it may not be necessary to use web services at all and your framework can hide that away. At a later time when you have to move the invocation to a different service provide, you can simply update the framework with new stubs (if static) or new WSDL (if dynamic environment) and not have any of your application be affected.

Integrating applications together usually means exchanging data from one to another. Currently these activities are being performed with exclusive knowledge about what kind of data is exchanged between applications and then providing specific transformations at appropriate layers. This is application dependent integration. With Web Services, application architects can build such a facility right into their application so it can receive and process messages from any application. The architecture (application framework) discussed above will help them in doing so because the application is primarily isolated from the knowledge how and where the messages are arriving from. Therefore application can process the messages as their own data structures and the framework can do the necessary processing to get application the data they want. This simplifies ad-hoc application integration tasks tremendously and this does not even require custom coding.

In order to keep the discussion short, I would not get into the details of how such architectures can be implemented and how these systems operate. Hopefully the Web Services architect can learn from the brief discussion presented in this section and use it in their own application process.

5. Conclusion

We have discussed many issues that an application developers must consider before delving into Web Services. We have also discussed standards and what different parts of software industries understand from Web Services. We have not discussed in detail the two camps of Web Services development namely J2EE and .NET. Readers can find numerous resources on this on the Internet. In conclusion, I would like to emphasize again that Web Services can succeed only if many companies and developers get together in signing the contract in form of agreed-upon well-defined XML based standards and to keep the implementation issues separate from standard definition. It should also be clear that simply because one has standards, does not mean that they do not need to pay attention to sound application architecture and design. That responsibility still lies on the shoulders of application designers. It is a bigger challenge when they are dealing with Web Services. Web Services have potential of introducing distributed computing in every application, therefore it is also important that many application designers/architects make themselves familiar with the issues that have been at the core of distributed computing for years.

Acknowledgements

I would like to thank Archana Srivastava, Application architect at NerveWire Inc. for assisting me with this paper and reviewing it.

Biography

Director
»Nashua, New Hampshire
»USA

Mr. Srivastava has been working with the computer industry for over nine years. Mr. Srivastava's interests include intelligent software, distributed and real-time software, web services and database systems. Mr. Srivastava received his bachelors degree from Indian Institute of Technology, IIT Kanpur, India and his masters from UofL Louisville, Kentucky, USA. He has been actively involved in research and development of distributed, open and component based software for database and application computing environments for past several years and he is currently representing Oracle Corporation different standards bodies related with XML web services, UDDI and Syndication (ICE).