Prepared for the Proceedings of MARSIM 2000,
International Conference on Marine Simulation and Ship Maneuverability,
Orlando, Florida, May 8-12, 2000
EVALUATING SIMULATORS FOR USE IN
ASSESSMENT OF MARINER PROFICIENCIES1
Dr. Marvin C. McCallum, Battelle Seattle Rearch Center, Seattle, WA Dr. Myriam Witkin Smith, U.S. Coast Guard Rearch & Development Center, Groton, CT2
Abstract
The paper describes a study sponsored by the U.S. Coast Guard to examine the implications of the STCW mandate for asssment of mariner proficiency by practical demonstration before a qualified asssor. Among the products of the study was a structured approach to evaluating simulators in their capability to support mariner asssment. The approach was refined by the trial evaluation of two desktop simulators for the asssment of mariner proficiency in ARPA operation.
Introduction
The asssment of mariner proficiencies by practical demonstration, as mandated by the International Maritime Organization (IMO) in its 1995 Amendments to the Seafarer’s Training, Certification and Watchkeeping (STCW) Code, is a substantial departure from earlier practice. Therefore, methods for developing, implementing, and documenting such asssments need to be established. The United States Coast Guard sponsored a rearch project to examine the implications of the mandate and to ensure that the best practices of current technology were available to the maritime industry. A group consisting of rearchers, faculty from veral maritime academies, and a commercial shipping company developed a model approach to developing asssments that are technically rigorous, and then demonstrated the practicality of the approach by implementing sample asssments on simulators, training vesls, and, finally, commercial vesls. Project efforts are reported elwhere, including in IMO MSC Circular 853.
An Approach to Developing Performance-bad Asssments Our earliest efforts on this project were to design a systematic method for the development of performance-bad asssments of mariner proficiencies. Our project team examined the IMO STCW Code requirements, the methods of Instructional Systems Development (ISD), and the best practices of the maritime industries. Our i
ntention was to distill a method that prerved the ISD objectives of establishing valid and reliable asssment objectives, measures, and standards while streamlining the process to rve the prent needs of the maritime industry. Our five-step method is illustrated in
1 The views expresd in this paper are tho of the authors and are not official U.S. Coast Guard policy.
2 The authors of this paper wish to gratefully acknowledge the participation of a varied and capable team, including especially Ms. Mireille Raby, now of the University of Iowa; Ms. Alice M. Forsythe, of Battelle Seattle Rearch Center; and Captains John M. Nunnenkamp and George R. Sandberg, both of the U.S. Merchant Marine Academy.
Figure 1. Step 1 is to identify asssment objectives , the critical elements of performance that will be assd. In the language of STCW, the are “knowledge, understanding, and proficiencies.” A valid list is achieved by consulting the STCW Code, other relevant regulations, job procedures, task analys, and subject matter experts. At this point in the process, the list of mariner asssment objectives could be customized to include requirements to follow the job procedures of a specific shipping company or to work with the equipment of a specific manufacturer.
With the asssment objectives identified, it is possible to go on to Step 2 and determine asssment methods for each performance element, whether as written or oral questions, exercis in a simulated job tting, or exercis in an actual job tting. Step 3 is to specify asssment conditions , a more detailed planning of how the asssment will be conducted. Step 4, to develop proficiency criteria , is the least intuitive part of the process, but extremely critical to the validity and reliability of asssments. Criteria are bad on performance measures and standards. A “performance measure” is an element of the required activities that can be obrved or measured; a “performance standard” is the acceptable or target level of an individual performance measure that is required for proficiency. An example of an objective might be to asss a mariner’s ability to steer by gyrocompass. The corresponding obrvable measure might be the accuracy with which the mariner is able to maintain the ordered heading. The standard, in this ca, might be to maintain the ordered heading to an accuracy of plus or minus three degrees. To continue this example, if the mariner can achieve that accuracy of steering under specified operational conditions, he/she is to be considered proficient in steering by gyrocompass.
Step 5 is to prepare asssment materials , a final package of forms and documentation that should be adequate to guide the candidate and the asssor in the conduct of the asssment. An overvie
w of the method is given in IMO, 1998; the design of the method is described in McCallum,Forsythe, Nunnenkamp, Sandberg, and Smith, 1999; and guidance for its u is given in McCallum,Forsythe, Barnes, Smith, Macaulay, Sandberg, Murphy, and Jackson, 1999.
Figure 1. Five-Step Method for Developing Mariner Performance-Bad Asssments
As we designed the method, we tested and refined it with a ca study, an asssment of mariner proficiency in Automatic Radar Plotting Aid (ARPA) operation. STCW mandates that this competence be assd on a simulator and provides considerable detail and guidance as to the
asssment objectives. We derived our asssment objectives from an analysis of ARPA proficiencies
identified in the STCW Code, then concentrated our efforts on the details of the asssment conditions and the criteria. We were fortunate to have faculty at the U.S. Merchant Marine Academy (USMMA) as partners in this effort, individuals with extensive experti in the training and asssment of mariner skills and in the u of simulators for that purpo. USMMA has a fully-capable ARPA laboratory with simulators bad on real equipment, so we were able to develop our mariner asssment with little concern for any limitations of the simulator.
A sample excerpt from our ARPA asssment procedure appears in Table 1, to illustrate the method.
The two mariner asssment objectives listed here come directly from the STCW Code, Section B-I/12, Guidance regarding the u of simulators, paragraphs 18 through 35, Training and asssment in the operational u of automatic radar plotting aids (ARPA). Asssment Objective 2.1 requires a demonstration of the mariner’s understanding of the criteria for lection of targets by automatic acquisition. With our USMMA partners, we developed simulator exercis that required the candidate to demonstrate the performance of interest, in this ca, tting up the ARPA with guard zones. Performance measures were also defined that specified the outcome of mariner performance was to be obrved in order to asss performance. For Objective 2.1, the guard zone ttings made by the mariner constituted the performance measure. The complete ARPA asssment, including descriptions of the simulator exercis, is available for review and for adoption in our first report (McCallum, Forsythe, Nunnenkamp, Sandberg, and Smith, 1999).
Table 1. Sample Mariner Asssment Objectives – ARPA U for Situation Asssment
Mariner Asssment Objective STCW Reference
for Objective
Asssment
Condition
Performance Measures
2.1 Understanding criteria for lection of targets by automatic acquisition Section B-I/12,
paragraph 25.1
Exercis A & B
ARPA t-up
instructions for
tting guard zones
2.1.1 Guard zone ttings
2.2 Appreciation of us, benefits, and limitations of ARPA operational warning
Section B-I/12,
paragraph 27.0
Exerci B
ARPA t-up
instructions for
tting safe limit
2.2.1 Safe limit ttings
2.2.2 Safe limit warning
2.2.3 Guard zone warning An Approach to Evaluating Simulators
At the prent time, a broad range of marine simulators are available to support any given mariner asssment. Simulators range from tho designed to provide highly realistic operational environments, controls, and displays to tho that are designed to reprent a facsimile of only some limited portion of the operational situation. Obviously, a simulator that is a full-scale mock-up o
f the operational tting is more likely to have many of the features and functions necessary to conduct a valid mariner asssment. However, with the advances in personal computer (PC) processing capability and the incorporation of PC processing in shipboard equipment, the advantages of the more complex and elaborate simulators are becoming less pronounced. When the advances are considered in conjunction with the affordability of PC-bad simulators, it is evident that the capability of PC-bad simulators to support mariner asssment must be riously considered. Thus, another
major objective of our investigation was to design and test a systematic method to evaluate the capability of a simulator to support a specified t of asssment objectives.
Our method is illustrated in Figure 2. Step 1 is to identify mariner asssment objectives . We assume that the are not developed solely for the purpo of evaluating the simulator but must be developed for the larger purpo of asssing (or training) mariner performance. We designed and tested our general approach to simulator evaluation by continuing our ARPA ca study, using the mariner asssment objectives that we had already developed. Step 2 in Figure 2 is the specification of the simulator evaluation objectives, conditions, and criteria , which establish the basis for determining if specific systems can support the asssment objectives. In specifying evalu
ation objectives, we considered, first, the simulator features needed to demonstrate the mariner performance to be assd. We also considered additional simulator requirements described in STCW in Section A-I/12, Standards governing the u of simulators , Part 1, Performance standards , paragraph 2,General performance standards for simulators ud in asssment of competence . The compri the simulator’s ability to simulate the capabilities of real equipment, to provide behavioral realism, to provide an interface to the mariner, to provide a variety of operating conditions, and to permit an asssor to control, monitor, and record exercis. We consulted IMO standards for the capabilities of real ARPA and navigational radar (IMO, 1979, 1971). For each simulator feature required to support the asssment objectives, we defined the conditions under which it would be evaluated, including the tasks that the mariner would perform and any environmental effects that would influence the performance of the equipment. Criteria are bad on the simulator’s ability to provide the necessary controls or displays for the mariner’s tasks, or the asssor’s needs, or for a simulation of real equipment. More discussion of how to develop the objectives, conditions, and criteria, and our complete ARPA example are available in a project report (Raby, Forsythe, McCallum, and Smith,1999).
Figure 2. Method for Evaluating Simulators
To illustrate Step 2, a summary of two simulator evaluation objectives is prented in Table 2,which c
ontinues the example from the preceding table. Simulator Objective 3.2 requires the simulator to allow the u of automatic acquisition of targets. This capability of the simulator is necessary for Mariner Objective 2.1, the demonstration of an understanding of the criteria for the lection of targets
by automatic acquisition. The necessary traffic and environmental conditions for the simulator
evaluation are tho included in the exercis prepared earlier for the mariner asssment. Specific simulator capabilities that supported the control and display of targets through manual and automatic acquisition were identified and defined in terms of evaluation criteria. The specific criteria ud to evaluate the capabilities of the simulators for Objective 3.2 consisted of four parate control-related criteria (e.g., ability to manually acquire, track, process, and continuously update information for at least 10 targets) and five parate display-related criteria (e.g., ability to simultaneously display information for at least 10 targets in manual mode). For the two example simulator objectives in Table 2, a total of 24 parate simulator evaluation criteria were defined.
Table 2. Sample Simulator Evaluation Objectives – ARPA U for Situation Asssment
Simulator Objective Mariner
Asssment
Objective
Evaluation
Condition
Simulator Control and
Display Evaluation
Criteria
3.2 U of manual and automatic acquisition 2.1 Understanding
criteria for lection
of targets by
automatic
acquisition
Acquire and track
one or more targets
using manual
acquisition and
using automatic
acquisition
Control target acquisition in
manual and automatic
modes (four criteria)
Display targets and indicate
acquisition mode (five
criteria)
3.3 U and limitations of ARPA operational warning 2.2 Appreciation of
us, benefits, and
limitations of ARPA
operational warning
Target violates safe
limit area and
activates warning
Control specific operational
warning modes and
parameters (six criteria)
Display operating mode
and provide visual and/or
audible warnings (nine
criteria)
Step 3 is to develop a simulator evaluation protocol, a package of forms and documentation that will support the conduct of the evaluation of a candidate simulator. The protocol should describe, for each objective, the evaluation conditions needed to ensure comparability across evaluators or across simulators and the evaluation criteria for controls and displays. It should include a means of rating how well the feature satisfied the evaluation criteria. An excerpt from the protocol, illustrating sample evaluation criteria, is prented in Table 3. The table shows a sample of two control-related criteria and two display-related criteria for Objective 3.2 (from Table 2). Evaluators were instructed to rate the extent that each criterion was met (Yes, No, Partial) and provide brief comments regarding their ratings. A discussion of ways to organize a protocol and our complete ARPA protocol
are prented in Raby, et al. (1999). Step 4, to conduct the simulator evaluation, and Step 5, to summarize and analyze findings, are described along with an overview of our ca study in the next ction.
Table 3. Sample Excerpts from Simulator Evaluation Protocol
Simulator Evaluation Objective Simulator Evaluation
Criterion
Reference Rating
Yes / No / Partial
Comments
3.2.Control-1 Ability to acquire, track,
process, and continuously update
information for at least 10 targets
STCW
A.422
ÌÌÌ
3.2.Control-2 Ability to acquire, track,
process, and continuously update
information for at least 20 targets
STCW
A.422
ÌÌÌ
3.2.Display-1.1 Ability to display
information for at least 10 targets
simultaneously
STCW
A.422
ÌÌÌ
3.2 U of manual and automatic acquisition
3.2.Display-1.2 Indication of tracked targets STCW
A.422
ÌÌÌ
Trial Application of Protocol and Analysis of Results
To test the approach, we conducted an evaluation of two commercial off-the-shelf PC-bad ARPA simulators. Becau cost is such an important factor in lecting simulators, we lected simulators that differed considerably in complexity and cost. Simulator X was of moderate complexity and cost;
Simulator Y more cloly approximated real equipment and was more costly. We were fortunate in having the assistance of the manufacturers in both cas, ensuring that each simulator exhibited its best potential. We forwarded to each manufacturer an overview of the evaluation objectives and a detailed description of the exerci scenarios. Subquently, we nt a team of three rearchers to apply the protocol and perform independent ratings. Our ratings for each criterion indicated how well the subject simulator met, partially met, or did not meet that criterion. We later combined the ratings for the rearchers to achieve one connsus rating for each criterion. Even without further analysis, a comprehensive listing of each simulator criterion with its rating rves to identify the strengths and weakness of a simulator and provides detailed information regarding its effective u and potential areas for its improvement by the manufacturer.
We explored the potential u of the ratings further by performing a number of more complex analys. First, we examined each simulator’s capability to support mariner asssment. For each simulator, we calculated summary scores reprenting its capability to support each of six categories of mariner asssment objectives: tting up and maintaining displays, situation asssment, factors affecting performance and accuracy, parallel indexing, application of COLREGS (Collision Regulations), and operational warnings and system tests. Ratings of the simulator features needed t
o support the mariner asssment objectives in Table 1 are included in the summary score for situation asssment. We found that the more costly simulator was generally more capable, supporting more of the mariner asssment objectives, but neither of the PC-bad simulators was fully capable of supporting all of the objectives. We also examined each simulator’s capability to provide the general simulator capabilities taken from STCW, by grouping the individual criteria into the categories of exerci programming, equipment t-up, simulation, and debriefing. Again we found that the more
