VERIFICATION AND VALIDATION OF SCIENTIFIC AND ECONOMIC MODELS R.C. KENNEDY,∗ X. XIANG, G.R. MADEY, and T.F. COSIMANO,
University of Notre Dame, Notre Dame, IN
ABSTRACT
实习鉴定表个人总结As modeling techniques become increasingly popular and effective means for simulating
real-world phenomena, it becomes increasingly important to enhance or verify our
confidence in them. Verification and validation techniques are neither as widely ud nor
as formalized as one would expect when applied to simulation models. In this paper, we
prent our methods and results through two very different ca studies: a scientific
model and an economic model. We show that we were able to successively verify the
simulations and, in turn, identify general guidelines on the best approach to a new
simulation experiment. We also draw conclusions on effective verification and validation
techniques and note their applicability.
Keywords: Verification, validation, simulation, natural organic matter, agent-bad
modeling, Ramy problems
INTRODUCTION
The u of simulations to model and study scientific and economic phenomena has the potential to be informative; however, the data produced by simulations are most valuable when they can be both verified and validated. In simple terms, this means the data produced are credible and indiscernible from real-world data. This achievement proves to be very difficult, as most real-world systems contain far more constraints and details than computers allow us to reasonably model, and this situation is even more difficult for agent-bad simulations and simulations of social and economic phenomena. This leaves most of our simulations as abstractions of real-world phenomena. Their purpos range from helping us to better understand natural phenomena to allowing us to predict the behavior of a system. With the varying purpos of simulations, verification and validation techniques also vary. The problem is that there is no universal verification and validation process that can be applied to all models. The purpo of our work is to explore and apply verification and va
lidation techniques to two very different ca studies. The first ca study focus on a scientific problem: the study of natural organic matter (NOM). It has an agent-bad backbone and was written first in Pascal then transformed into Java with Repast. The cond ca study involves an economic problem: solving Ramy problems in a stochastic monetary economy. It has a more numerical basis and was written first in Matlab and then in C++. We will compare two unrelated simulations that were each written in different programming languages and then compare and verify results. In addition, we will explore some general guidelines to u as an approach to increasing the confidence of a new simulation.
∗Corresponding author address: Ryan C. Kennedy, 384 Fitzpatrick Hall of Engineering, University of Notre Dame, Notre Dame, IN 46556; email: rkenned1@nd.edu.
The organization of this paper is as follows. The next ction outlines what we mean by verification and validation and introduces some general methods. The ction that follows describes various aspects of our first ca study, including background, validation, and implementation. Then the same is done for our cond ca study. A conclusion and some general guidelines are provided in the ction following that. Finally, a discussion of future work and references conclude the paper.
VERIFICATION AND VALIDATION PROCESS
Simply put, model verification is getting the model right. This means that the code generating the phenomenon being modeled correctly matches the abstract model. Model validation is getting the right model, meaning that the correct abstract model was chon and accurately reprents the real-world phenomenon. It is important to note that verification and validation are key parts of the model development process. Moreover, they must be performed in tandem for the best results. Effective verification and validation of a model will increa the confidence in the model, making it more valuable. An adapted version of Sargent’s (1998) and Huang’s (2005) verification and validation process diagram is shown in Figure 1. It has been modified for agent-bad scientific and economic simulations.
While there have been many verification and validation studies performed for general engineering purpos, verification and validation studies for agent-bad and social science simulations are lacking. Some of this can be attributed to agent-bad modeling not being as mature as engineering modeling. The point is that we can adapt what has already been done as well as create new tools to fit the needs of agent-bad modeling.
FIGURE 1 A verification and validation process
for scientific and economic simulations
Balci (1998) outlined 15 general simulation principles, developed primarily for engineering or management science applications. His principles help engineers and rearchers better understand the verification and validation they are performing. This understanding is directly related to model success. A few of his principles that are relevant to scientific and economic modeling are prented next.
1.The outcome of the simulation model verification, validation, and testing
should not be considered as a binary variable where the model outcome is
absolutely correct or incorrect. It is important to realize that models, being
abstractions and not absolute reprentations of phenomena, can never totally
and exactly match a system.
2.Complete simulation model testing is not possible. As we cannot test all
possible inputs and parameters for a system, we must choo the most
appropriate ones.
3.Simulation model verification, validation, and testing must be planned and
documented. Successful planning and documentation are critical and involve
the work of many people throughout the lifetime of the system.
4.Successfully testing each submodel (module) does not imply overall model
credibility. Simply becau the modules work well independently does not
mean they will work cohesively in a system.
市场力When verification and validation of a model are being performed, it is good to begin by identifying the key principles and techniques to be ud for that model. Moreover, planning the verification and validation process, as outlined previously, makes the process more complete and effective. Utilizing Balci’s (1998) principles and techniques is a great starting point; from there, model confidence can be improved with further subjective and quantitative methods. We next outline a general verification and validation process that can be adapted to fit many agent-bad, social, and economic models. A hierarchy of such methods is shown in Figure 2.
Subjective Methods
Subjective methods largely rely on the judgment of domain experts. They are often ud for initial quick-and-dirty validation, but they can also be more formalized. Whatever the purpo, subjective methods typically require less effort than quantitative methods, can detect flaws early in the simulation process, and are often the only applicable verification and validation methods for exploratory simulation studies. We next describe some of the subjective techniques propod by Balci (1998) that may be applicable to economic and agent-bad scientific simulations. His techniques are widely ud in validating the models of manufacturing, engineering, and business process. The following has been adapted from Xiang et al. (2005).
1.Face validation. This preliminary approach to validation involves asking
domain experts whether the model behaves reasonably and is sufficiently
accurate. This is often achieved by evaluating the output or obrving a
visualization, if applicable.
FIGURE 2 Verification and validation
methods
2.Turing test. This technique is performed by giving domain experts model
outputs and real-world outputs and asking them to discriminate them.
3.Internal validity. This involves comparing the results of veral replications of
a simulation, with the only difference being the random ed. Inconsistencies
in the results question the validity of some aspect of the model.
4.Tracing. Here, the behavior of entities in the model is followed to determine if
xp系统安装
the logic of the model is correct.
5.Black-box testing. This technique involves how accurately the model
transforms the input to output in a system.
松脂价格Quantitative Methods
Incorporating quantitative, or statistical, methods into the validation process can significantly increa the credibility of the model. Model validation is conducted by using statistical techniques to compare the model output data with the corresponding system or with
the output data of other models run with the same input data.
The first step to starting quantitative analysis is to determine a t of appropriate output measures that can answer ur questions (Xiang et al. 2005). After a t of output measures has been collected, various statistical techniques can be applied to complete the validation process. Time ries, means, variances, and aggregations of each output measure can be prented as a t of graphs for model development, face validation, and Turing tests. Confidence intervals and hypothesis tests can be ud in the comparison of parameters, distributions, and time ries of output data for each t of experimental conditions. The statistical tests can help model developers determine if the model’s behavior is acceptably accurate.
The cost of the validation process increas exponentially with the confidence range for a model. There is no single validation approach applicable to all computational models. Choosing the appropri
ate statistical test techniques and measures of a system is important when conducting a validation process. It is important to note that there is no correct t of statistical tests to u for every simulation; the best results are achieved when tests are carefully chon according to the model. Some of Balci’s (1998) more quantitative techniques that are relevant to our ca studies are next described.
1.Docking. Docking, or model-to-model comparison or alignment, is ud when
another model that models the same phenomenon exists or can be created.
Docking helps to determine whether two or more models can produce the
same results (Axtell et al. 1996). The main idea is that model confidence is
significantly improved when two or more models produce the same effective
results, particularly if the models were developed independently and with
different techniques. In addition, the output from a model can be validated
against real-world data.
2.Historical data validation. When historical data exist or can be collected,
the data can be ud to build the model, and the remaining data can then
ud to determine if the model behaves as the system does.
貌什么神什么成语
3.Sensitivity analysis/parameter variability. Here, one changes the input values
and the internal parameters of a model to determine the effect on the model
and its output. Ideally, the relationship in the real-world system should be中国人最早的姓氏是什么
mimicked in the model. Sensitive parameters that cau significant changes in
the model’s behavior should be made sufficiently accurate before this model
双曲线函数
is ud.
4.Predictive validation. This technique is ud to compare the model’s
prediction with actual system behavior. The system data may come from an
被动式operational system or specific experiments, such as from a laboratory or from
field experiments.
CASE STUDY I: AN AGENT-BASED SCIENTIFIC MODEL NOM is a heterogeneous mixture of molecules. NOM plays a crucial role in the evolution of soils, transport of pollutants, and carbon cycle (Cabaniss et al. 2005; Xiang et al. 2005). Its evolution is an important rearch area in a number of disciplines. NOM is complex; it is made up of molecules with varying molecular weights, reactivity levels, and functional groups. This
makes it difficult to model. Performing chemical experiments with NOM is difficult and time-consuming becau of its complexity and becau of our limited knowledge of its inner workings. The ability to effectively predict NOM behavior as it evolves over space and time would be very valuable to scientists and an accomplishment in the modeling discipline. Conceptual Model
The NOM conceptual model was bad on the work of a chemist working at the University of New Mexico (Cabaniss et al. 2005). He generated his model from extensive obrvation and experimentation in the laboratory. His basic model outlined the u of the precursor molecules cellulo, lignin, and protein (among others) to be ud in a controlled environment where parameter
s such as light intensity, temperature, and density could be varied.
A more detailed description of our model follows and has been adapted from Xiang et al. (2005). Agents
Our agents are molecules. Each molecule is a reprentation of its underlying elemental formula, meaning the number of C, H, O, N, S, and P atoms prent. This gives ri to a molecular weight for each molecule. Molecules also contain a functional group count, such as the number of alcohol or ester groups prent.
Behavior
In our environment, agents can move around a grid, interacting with other molecules and their environment. Molecules undergo chemical reactions on the basis of specific probabilities. Reactions can result in structural changes in the molecule, such as the addition of functional groups. They can also generate new molecules from predecessor molecules, and the predecessor molecules may leave the system. Twelve types of chemical reactions, including first- and cond-order chemical reactions, are modeled as described in Table 1. The categories of reactions are as follows:
1.First-order reactions with a split. The predecessor molecule A is split into
two successor molecules B and C. Molecule B occupies the position of
molecule A, while one of the empty cells clost to molecule B is filled with
molecule C.
2.First-order reactions without a split. The transformation only changes the
structure of the predecessor molecule A.
3.First-order reactions with the disappearance of a molecule. The predecessor
molecule A disappears from the system.
4.Second-order reactions. Two molecules A and B are combined to form a new
molecule C. Molecule C replaces molecule A, and molecule B is removed
from the system.
