Ku-Yaw Chang1 and Lih-Shyang Chen*
Department of Electrical Engineering, National Cheng-Kung University, Tainan, Taiwan, ROC
1 E-mail: canseco@mirac.ee.ncku.edu.tw
*Correspondence should be addressed to Dr. Lih-Shyang Chen at Department of
Electrical Engineering, National Cheng-Kung University, 1 University Rd., Tainan, Taiwan,
ROC. Tel:886-2-27377009; E-mail: chens@mail.ncku.edu.tw
The basic idea of hyper-control is to use a multimedia document to control other application systems[Chen 96]. In the original Command Processor pattern, the fixed execution sequence of computation codes and dialog codes hinders an application from supporting the hyper-control mechanism. Based on an extension of this pattern and other related patterns, we have developed a medical image system Discover, which can support the hyper-control mechanism. In this paper, the system architecture of Discover will be described. We will also give a practical example to explain the available ways for a hyper-control document to "hyper-control" an application, and describe our experience in using patterns.
When an application system is getting complicated, it is difficult for an end user to use the application. One way to overcome the system's complexity is to use a multimedia document to guide a user to operate the system. This concept was called the hyper-control[Chen 96], which will be described later. One important issue for an application to support this mechanism is to provide an external control channel in addition to the internal control which is built in the original application. The Command Processor pattern[Busc96] is a good approach to achieve this goal, but it causes problems because of the fixed execution sequence of computation codes and dialog codes inside the body of a command object. This paper describes the design of a medical image application system called Discover that is a hyper-controllable system, and how it required that we extend the Command Processor pattern. In addition, our design also uses the Document-View pattern, the Memento pattern, the Visitor pattern, and the Singleton pattern.
The remainder of this paper is organized as follows: Section 2 describes the hyper-control mechanism and gives an overview of the Command Processor pattern and other related patterns. In section 3, we illustrate the system architecture of Discover, which supports the hyper-control mechanism well. Section 4 gives an example of the cooperation between a hyper-controllable application and its hyper-control document. In section 5, we summarize our experiences in applying pattern-based strategy to our system.
Discover is a distributed interactive visualization system, which has been running in Nation Cheng-Kung University hospital since 1993 [Liu 96]. Shortly after Discover was implemented, we observed that physicians were having trouble understanding and navigating through Discover processes for image analysis and generation. Even with the aid of help documents, physicians still have to go back and forth between working with Discover and reading through its help documents while learning how to use the Discover. During such a process of learning, some misunderstandings and mismatches may occur. These problems go far beyond the ability of general help systems. In other words, there still exits some physical as well as conceptual gaps from the documents to its associated application system, especially when the application is very complicated.
The concept of hyper-control was first proposed in [Chen 96] to fill the gaps mentioned above. Its basic idea is to use a multimedia document (written by non-programmers working with engineers) to control other hardware or software systems. The control commands are installed in anchors, which is similar to that of "hyperlink". The major difference is that when such an anchor is clicked, a command is issued to activate an operation of its associated application system, which is originally a stand-alone application system with its own user input, i.e. internal control, and is currently running simultaneously on the same machine. Such a multimedia document with the ability to control an application is called a hyper-control document. An application system that supports the hyper-control mechanism is called a hyper-controllable application system.
In order to clarify the ideas of hyper-control and to explain how it works, we first give a simple example here. A hyper-controllable medical image application is shown in figure 1(a). It can process two different types of medical image data: gray-level and true-color images.
Fig. 1(a) A hyper-controllable medical image
application
(with a gray-level image on the left side and a true-color image on the right side)
Suppose a physician wants to segment out the spine of the left gray-level image(12 bits/pixel), he/she may need to do the following steps after loading the image data: (without the aid of a hyper-control document)
A physician has to remember each step in the above processing procedure and choose a corresponding command under a correct menu for each operation. Different clinical cases have different processing procedures. As the number of procedures increases, it is virtually impossible for a physician to remember all the detailed steps correctly.
With the aid of the hyper-control mechanism, a physician needs to record each function he/she uses to process the image as a hyper-control document, as shown in figure 1(b), and tries to come up with a procedure that would apply to a similar data set. If necessary, a physician can work with engineers. A physician can apply the same procedure to many data sets to make sure the procedure consistently produces reasonable results. If so, the procedure becomes a diagnostic protocol for the particular case. After the establishment of a hyper-control document, physicians can browse it and apply the same procedure to other similar data sets by clicking on each anchor in the document.
Fig. 1(b) Using a WWW browser(Internet Explorer) to browse a hyper-control document.
In other words, the hyper-control mechanism can help us standardize the processing procedure. Other users simply follow through each instruction in the procedure, click on each anchor, and interact with the associated application as prompted. The hyper-control can also shorten training time, ensure that users operate the system in a consistent way and at a uniform quality level. In some sense, it also provides users with a more friendly user interface(instead of remembering each step in the processing procedure and clicking on the pull-down menus for each operation). When applied to the on-line documentation, the hyper-control mechanism can also fill up the physical as well as conceptual gaps from the documents to its associated application system and increases the effectiveness of the on-line documentation. More generally, the hyper-control mechanism enables us to tailor a general-purpose application system (that is a hyper-controllable application system) into a turn-key system by using a hyper-control document. In other words, the users can completely ignore the original general-purpose user menus and work with the hyper-control document that has been tailored to solve a particulr problem.
Fig. 1(c) The key to support the hyper-control mechanism.
However, a hyper-controllable application does not come for free. If we partition an application according to Document/View pattern as shown in figure 1(c), the key to support the hyper-control mechanism is to provide an alternative user input channel to accept external control from other applications, in addition to the original user input channel, i.e. the internal control. In fact, a hyper-controllable application is a stand-alone one with its own input/output and can still work well even without any hyper-control documents.
In our implementation, hyper-control documents are in the HTML format. By doing so, we can avoid "re-inventing the wheel" and use existent WWW browsing/authoring tools to browse/edit these documents. Besides, it also becomes very easy and natural to integrate hyper-control documents with some other HTML-based help systems, which are the de facto standard format for the majority of on-line documentation as the fast growth of WWW. Therefore, a WWW browser is just a tool to issue commands to control a hyper-controllable application through the external input channel. Theoretically, any other application can control a hyper-control application if it knows how to send commands to the external input channel.
In fact, the basic operations of an interactive application can be divided into those that take arguments and those do not. A more complicated operation can be composed by these basic operations. In the above example, the histogram equalization command does not take any arguments, while the threshold command does. There is only one way to issue commands that take no arguments from a hyper-control document to its associated application. That is to invoke the algorithm of the command directly. However, there are three different ways to issue commands that take arguments: (take the above threshold command as an example)
Without parameters / dialog displayed: a threshold command message without parameters is sent to the application. The Threshold dialog box is displayed and contains two scroll bars initialized at 2000 and 2048 respectively. The initial values are provided by the application itself.
With parameters / dialog displayed: a threshold command message with parameters 4000 and 4095, which is kept on the hyper-control document as shown in figure 1(b), is sent to the application. The Threshold dialog box is still displayed, but its two scroll bars are initialized at 4000 and 4095 respectively according to the parameter values coming along with the command message. Note that in case 1 and 2, a user still has a chance to adjust the parameters through the dialog interaction before applying it.
With parameters / no dialog displayed: a threshold command message with parameters 4000 and 4095, which are kept on the hyper-control document, is sent to the application. No dialog box is displayed. The threshold operation (with parameters 4000 and 4095) is carried out directly.
In other words, a hyper-controllable application should be able to accept not only commands from itself(internal control), but also those issued in different policies from hyper-control documents(external control). Generally speaking, an interactive application system is composed of a dialog component and a computation component[Hart 89], as illustrated in figure 2. In order to achieve the goal mentioned above, we divide the dialog component into three constituents as follows:
When a user clicks a menu item or strikes a keyboard shortcut through Trigger Generator, the flow of the control is the following: (refer to figure 2)
(1) an operation that requires no parameters invokes a Trigger on the Computation Component to carry out the associated computation directly (1-a).
(2) an operation that requires parameters first invokes a Trigger(2-a) to fetch pre-stored parameters from the Parameter Storage. After the parameters are obtained, the Request(2-b) is invoked. As a result, the corresponding dialog box will be displayed with the parameters as the initial values. At this moment, a user can adjust the parameters according to his/her own needs. After the confirmation of parameter setting, a Reply(2-c) with final parameters will be invoked on the Computation Component to complete the whole operation. Of course, the final parameters should be stored back to the Parameter Storage after the computation is done, as stated previously.
Fig. 2 Four interception points for hyper-control.
According to the above analysis, we found that there are totally four interception points in the course of each operation for a hyper-control document to activate an invocation to control the application directly:
(1)Trigger: to simulate the effects of choosing menus or keyboard shortcuts by invoking a corresponding Trigger on Computation Component(h1-a) or Dialog Component(h2-a).
(2)Request: to invoke a Request with parameters on Dialog Component(h2-b). A corresponding dialog box of the application system will be displayed. And its parameters are initialized according to the parameter values coming along with the Request invocation.
(3)Reply: to invoke a Reply with parameters on Computation Component(h2-c). The major difference from a Request invocation is that the computation is carried out immediately without showing any dialog box.
Any invocation coming from one of the above interception points will also activate its following invocations or actions. Therefore, an invocation from a hyper-control document has exactly the same effects as if the invocation were activated by the application itself. More importantly, such an arrangement can meet all the requirements that a hyper-controllable application can accept commands issued in different ways from hyper-control documents.
The basic idea of Command Processor pattern, which builds on the Command pattern in [Gamm95], is to encapsulate service requests into command objects and to separate the request for a service from its execution. The Command Processor Pattern illustrates more specifically how command objects are managed. The structure of Command Processor is shown in Figure 3. The Abstract Command component defines the interface of all command objects. An indispensable procedure of this interface is the one to execute a command object. For each function, a concrete command component is derived from the Abstract Command. A Command component implements the interface of the Abstract Command by using zero or more Supplier components. The Controller represents the interface of the application. It accepts requests and creates the corresponding command objects. The command objects are then transferred to the Command Processor for execution. The Command Processor receives command objects from the Controller and takes responsibility of managing them, including starting their execution. It is also the key component that implements additional services such as the storing of request objects for later undo. The Supplier components provide most of the functionality required to execute concrete commands. When an undo mechanism is required, a supplier usually provides a means to save and store its internal state.
Fig. 3 The structure of Command Processor pattern.
Since the Document-View pattern is a variant of Model-View-Controller pattern (MVC)[Busc96], we will first describe MVC briefly. The MVC divides an interactive application into three components. The model contains the core functionality and data. Views display information to the user. Controllers handle user input. Views and controllers together comprise the user interface. The separation of the model from the view and the controller components allows multiple views of the same model. If the user changes the model via the controller of one view, all other views dependent on this data should reflect the change. Therefore, we need a change-propagation mechanism to ensure the consistency between the user interface and the model.
However, in several GUI platforms, window display and event handling are closely interwoven. The responsibilities of the view and the controller from the MVC are often combined together into a single component. This kind of structure is often called a Document-View architecture. The document component corresponds to the model in MVC. The view component combines the responsibilities of controller and view in MVC.A memento is an object that stores a snapshot of the internal state of another object - the memento's originator [Gamm95]. Only the originator can store and retrieve information from the memento. In other words, the memento is "opaque" to other objects. A caretaker will ask the originator to create a memento and is responsible for the memento's safekeeping. It never operates on or examines the contents of a memento. Later, the caretaker can use the memento to restore the originator's internal state. Thus when it is necessary to record the internal state of an object, we can capture and externalize the object's internal state without violating encapsulation by using a memento. This pattern will be helpful in supporting the undo mechanism in an application.
Visitor pattern allows us to define a new operation without changing the classes of the elements on which it operates [Gamm95]. A visitor is a package of related operations from each class. When an operation is invoked, we can create a visitor and pass it to an element. When an element "accepts" the visitor, it sends a request to the visitor that encodes the element's class. It also includes the element as an argument. The visitor will then execute the operation for that element - the operation that used to be in the class of the element in a general object-oriented programming paradigm.
Sometimes, it is important for some classes to have exactly one instance. A global variable makes an object accessible, but it does not keep you from instantiating multiple objects. A better solution is to make the class itself responsible for keeping track of its sole instance. This is the Singleton pattern, which ensures a class only has one instance and provides a global point of access to it [Gamm95].
Conceptually, the key for a hyper-controllable application to receive external control is to provide four interception points to the outside, i.e. other applications such as the hyper-control document.
Since one of the basic ideas of the Command Processor pattern is to separate the request for a service from its execution to support different modes of user interaction like external control of the application, it seems to be suitable for supporting the hyper-control mechanism. However, when we adopt the Command Processor pattern, the invocations of the elements in the computation component and dialog component are usually interwoven inside the body of a command object. In other words, the execution sequence of the invoked elements in the computation components and dialog components is fixed in a command object. A piece of computation codes is tightly bound with its preceding menu selection or dialog activity. Only after a menu selection or a dialog activity, will the corresponding computation codes be executed. Thus it is virtual impossible for the application to be fully controlled by hyper-control documents through all the interception points shown in figure 2.
In order to overcome above limitations, we separate the computation codes and dialog codes completely by duplicating the whole Command Processor constituent components into two different parts: one for the computation component, the other for the dialog component(user interaction). Each part functions as the original Command Processor pattern and has its own Command Processor, Controller, Supplier and Command. The following two diagrams show the relationships of these two parts of Command Processor pattern.
The Dialog Command Processor, as shown in figure 4, is for dialog activities. When the Dialog Controller receives a Trigger or a Request invocation, it will create a corresponding command object, called a Dialog Command Object. The Trigger and Request invocations stand for the Trigger and Request interception points provided by the dialog component in figure 2. The controller then transfers this new command object to the Dialog Command Processor for execution. The processor activates the execution of the dialog command object, whose main job is to prepare a set of parameters for the following computation. Usually, there are three major steps during the execution process of a dialog command object:
Fig. 4 Dialog Command Processor. All its constituents are inside the rectangle in dash line.
(1)Trigger invocation: to invoke a Trigger to retrieve pre-stored parameters from its supplier, i.e. Parameter Storage, as the initial values in the following dialog activity. This step is skipped if what the Dialog Controller receives is a Request invocation, which comes with parameters.
(2)Request invocation: to invoke a Request with parameters to display the corresponding dialog box. The final parameters will be kept inside the body of the dialog command object.
(3)Reply invocation: to invoke a Reply with final parameters to the Computation Controller to carry out the computation. This invocation is also the only connection between the Dialog Command Processor and the Computation Command Processor.
The Computation Command Processor, as shown in figure 5, is in charge of the core computation. When the Computation Controller receives a Trigger or a Reply invocation, it will create a corresponding command object, called a Computation Command Object. The Trigger and Reply invocations stand for the Trigger and Reply interception points provided by the computation component in figure2. The parameters coming along with Reply invocation will also be forwarded to the computation command object, which will be sent to the Computation Command Processor for execution.
Fig. 5 Computation Command Processor. All its constituents are inside the rectangle in dash line.
By providing two sets of command processors, we can separate the computation codes and dialog codes completely. Their execution sequence is no longer fixed in a command object, but separated into two command objects. With this arrangement, it becomes more flexible for hyper-control documents to control a hyper-controllable application. We will give an example to explain the scenarios in the next section.
Our application domain has at least two types of medical image data. They are gray-level images and true-color images. In terms of the Document-View architecture, they represent two image documents. In other words, a command object has to face different supplier types. When the same operation is applied to different documents(suppliers), it may take different arguments. That is, the interfaces to implement the same function on different suppliers may be different. Therefore, we may need to provide different command objects for the same operation. For example, when a Threshold command is applied to a gray-level image, part of the execution codes may look like this:
// pSupplier: a pointer to a supplier
// nLow, nHigh: two threshold parameters
pSupplier->Threshold(nLow, nHigh);
or like this when applied to a true-color image:
// nRLow, nRHigh: two threshold parameters for Red component of the image
// nGLow, nGHigh: two threshold parameters for Green component of the image
// nBLow, nBHigh: two threshold parameters for Blue component of the image
pSupplier->Threshold(nRLow, nRHigh, nGLow, nGHigh, nBLow, nBHigh);
A better solution to this problem is that the same command object can be applied to different supplier types without any change. In order to achieve this goal, we use Visitor pattern to package related operations from each document in a separate visitor and organize the documents as elements, as illustrated in figure 6. When an operation is executed, a computation command object simply passes a visitor to its associated element - supplier. When an element "accepts" the visitor, it sends a request to the visitor, which will then execute the operation for that element. The execution function of the Threshold command object may look like this(no matter what types its supplier is):
void CCmpCmdThreshold::Do()
{
// 1. get and store the memento
m_pMemento = m_pSupplier->CreateMemento();
// 2. apply the algorithm
m_pSupplier->Accept(&m_VisitorThreshold);
}
In this example, the visitor m_VisitorThreshold is created in the constructor. The class CCmpCmdThreshold provides two different constructors: one for gray-level images and the other for true-color images. The Computation Controller will choose one of them to create this command object according to the arguments it receives.
Fig. 6 Visitor Pattern
In fact, a computation command object also plays the role of caretaker in Memento pattern. In other words, before the computation is carried out, the command object will ask the supplier to create a memento to save its current internal state(See step 1 in Do function of CCmpCmdThreshold). Later, when the undo procedure is invoked, this memento will be delivered to the supplier for restoring its internal state.
void CCmpCmdThreshold::Undo()
{
// to restore the memento
m_pSupplier->SetMemento(m_pMemento);
m_pMemento = NULL;
}
Figure 7 shows the structure after combining the Document-View pattern and the Command Processor pattern. In order to support the undo/redo mechanism, each image document has its own command history, which is maintained by the command processor. As a result, every image document has an instance of the Dialog Command Processor and an instance of the Computation Command Processor. However, because a command object can be applied to different supplier types by using Visitor pattern, its creator, the Dialog Controller or the Computation Controller, is designed to be a global and unique component by using Singleton pattern. Therefore, a command object is created by the controller and is first transferred to the active document, instead of delivering to the processor directly. The active document will then just forward the command object to its processor.
Fig. 7 The structure after combing Document-View pattern and Command Processor pattern.
In figure 7, although both the Dialog Controller and the Computation Controller can be accessed globally, only those objects belonging to the same application, i.e. the same address space, can access them. In order to accept external control from hyper-control documents, two proxy controllers are provided, as shown in figure 8. These proxy controllers are automation objects, which allows other applications to launch and operate on it directly[Adle 95]. Each proxy controller provides exactly the same interfaces as what its corresponding original controller has. When receiving a request, the proxy controller will simply forward the request message to the original controller. In fact, two original controllers cannot tell where the Trigger/Request/Reply invocation comes from: an external control from the proxy controller or an internal control from the application itself.
In implementation, we use VBScript language in the hyper-control document to create and control two proxy controllers, which are scriptable objects.
Fig. 8 Two proxy controllers.
In this section, we will give a practical example to show how a hyper-control document can control an application through the interception points described section 2. A hyper-controllable application and a hyper-control document are shown figure 1(a) and 1(b) respectively. In the hyper-control document, the initial low and high values of the threshold parameters are 4000 and 4095.
(1)When a user clicks the Equalization anchor to apply a histogram equalization operation that takes no arguments, a Trigger is invoked on the Proxy Computation Controller through the automation mechanism(h1-a in figure 2). This invocation is forwarded to the Original Computation Controller, which will then create a computation command object for the histogram equalization operation and execute it. The results are illustrated in figure 9.
Fig. 9 The result after invoking a Trigger (histogram equalization) on the Proxy Computation Controller from the hyper-control document.
As mentioned in section 2, there are three different ways to issue a command that takes parameters, including Trigger, Request, or Reply. When users click the Display button on the hyper-control document, three radio buttons are displayed. Users can choose one of them to issue a command. When the user clicks the Threshold anchor that corresponds to Trigger, Request and Reply selection respectively, the results are shown in figure 10, 11 and 12.
(1)With Trigger selection: a threshold command message without parameters is sent to the application through the Trigger interception point on Dialog Controller(h2-a in figure 2). The Threshold dialog box is displayed. Its low and high values are initialized at 2000 and 2048 according to the values retrieved from the Parameter Storage of the application. Users can adjust the parameter values before applying it.
Fig. 10 The result after invoking a Trigger(threshold) on the Proxy Dialog Controller from the hyper-control document.
(2)With Request selection: a threshold command message with parameters 4000 and 4095 is sent to the application through the Request interception point on Dialog Controller(h2-b in figure 2). The Threshold dialog box is also displayed. But its initial low and high values are initialized at 4000 and 4095 respectively according to the values on the hyper-control document. Users can adjust the parameter setting before applying it.
Fig. 11 The result after invoking a Request(threshold) on the Proxy Dialog Controller with parameters from the hyper-control document.
(3)With Reply selection: a threshold command message with parameters 4000 and 4095 is sent to the application through the Reply interception point on Computation Controller(h2-c in figure 2). No dialog box is displayed. The threshold operation is carried out directly with low value 4000 and high value 4095.
Fig. 12 The result after invoking a Reply(threshold) on the Proxy Computation Controller with parameters from the hyper-control document.
Hyper-control is a powerful and flexible mechanism to standardize the processing procedures. It can make a general-purpose application easy to use in different domains. The basic requirement for an application to support the hyper-control mechanism is to provide an additional input channel to receive external control from other applications. We extend the Command Processor pattern by separating the computation codes and dialog codes into tow kinds of command objects, i.e. the Dialog and Computation command objects, to support the hyper-control mechanism.
Besides, when combined with Document-View pattern, a command object may have different supplier types (image document types). In such a case, the reusability of a command object becomes an important issue. By adopting Visitor pattern for execution and Memento pattern for the undo function, a command object can be applied to different supplier types without any change..
In order to receive requests from hyper-control documents or other applications, two proxy controllers are also provided. This allows hyper-control documents to become part of the application's user interface and to be integrated with an HTML-based help system easily.
Based on above pattern technology, we have developed an interactive medical image application system to support the hyper-control mechanism successfully. During the development process, it was discovered that patterns do provide a very good solution in the design of system architecture to solve our practical problems. Although several patterns work together, they still help us have a clear road map of the complicated control flow.
Copyright 1998, K.Y. Chang & L.S. Chen.
Permission is granted to copy for the PLoP-98 conference.