Guidance
This particular page provides information on the rationale behind the RAMI exercise, its functional organisation and data-usage policy. It is intended for current and prospective participants in the RAMI benchmarking activity. If you intend to contribute model results to (current or) future intercomparison exercises, please read on to understand what are your rights and responsibilities in this venture. Your active participation in the exercise and in particular your willful, spontaneous and unsolicited submission of results in a particular RAMI (or RAMI4PILPS) phase will be deemed to imply your knowledge and acceptance of the terms and conditions described on this web page. Otherwise, you can safely skip this page.
The interpretation of remote sensing data generated by Earth Observation satellites hinges on the exploitation of radiation transfer models that describe the interaction of solar radiation with the various geophysical media (atmosphere, vegetation, soil, etc) of interest. It is thus of the utmost importance that these models be reliable and accurate. Since different models have been designed by different authors for different purposes, it is natural to carry out comparisons between these tools. This exercise is of primary interest to the authors because it provides an objective way to evaluate the performance and limits of applicability of these models. The results are also of relevance to the users of these models, in that they can select a particular tool for their specific purpose, according to requirements dictated by the application at hand.
The goal of the experiments described in the following pages is thus to provide a basis for comparing a large variety of computer codes that model the radiation fields typically encountered in remote sensing. The fundamental measurement is a radiance, or equivalently a reflectance, and these models thus simulate the spectral Bidirectional Reflectance Distribution Functions (BRDF) of the observed geophysical media. Of particular interest initially are the radiance distributions of solar radiation reflected from vegetation canopies, as well as the corresponding inverse problem of deriving biogeophysical canopy parameters from remotely sensed radiances.
The approach for such a model intercomparison is to provide benchmark cases and solutions which will be useful in the development and testing of models. The intercomparison exercise can also help identify the regimes of applicability of existing models. The overall benefit to the BRDF and remote sensing communities will be a demonstration of maturity and a better understanding of how these models are to be used in the practical interpretation of remote sensing data.
Scientific communities in several other technical specialty fields have been using such model performance intercomparisons in the past, as a yardstick to guide the development and evolution of complex computational tools. Most recently, for example, a Web site has been established by the climate modeling community for the Intercomparison of 3D Radiation Codes, whose approach was emulated here. Similar exercises have taken place to compare General Circulation Models,land surface parameterization, and other simulation tools.
Historically, the remote sensing community had already performed a 'model cook-off' in the mid 1980's, when the first computer models of surface reflectance were tested for their usefulness in the interpretation of these data (N. Goel and F. Hall, private communication). Internet technologies did not exist in the form we know today, but this 'somewhat pedestrian model intercomparison' was nevertheless very useful for the further evolution of computer models and it even identified some errors in databases published earlier, which could then be corrected. RAMI constitutes a formalization and extension of that effort.
This Web site and the proposed intercomparisons reflect a self-organized activity of the BRDF community. The impetus for Phase 1 of RAMI came from the activities linked to the organization of the IWMMM-2 conference of ENAMORS, but participation in its continuation (Phase 2, 3 and beyond) is open to all interested participants, who can contribute freely to and benefit from these exchanges. Please read our privacy and data-usage policy.
A scientist - not belonging to the RAMI coordination group - qualifies as a full participant to a particular phase of the RAMI benchmarking exercise if:
1.1.1: he/she is the original developer of the model to be tested,
or
1.1.2: he/she is duly authorized, by the original developer, to conduct the experiments on his/her behalf,
or
1.1.3: he/she is using a publicly available model,
and
1.2: he/she submits the model simulation results within the required constraints (deadline, format, etc),
and
1.3: the volume and quality of results submitted justifies full participation in the current benchmarking phase
Each model can participate only once to each exercise of a given phase. In other words, multiple participants cannot simultaneously submit results obtained from the same or similar models. Prospective participants should contact the RAMI coordinators in case of doubt. Participating in one phase of RAMI does not automatically imply or guarantee participation in any subsequent phase. Qualified participants accept full responsibility for the following tasks:
Since the analysis of the simulation results submitted is largely done automatically, files that do not meet the formatting and other constraints set by the RAMI coordinators may be rejected or unused. This may lead to the de facto disqualification of the participant for the particular exercise, or for the entire phase. The RAMI coordinators will ensure that all the necessary information and constraints are publicly available and accessible, and will make best reasonable efforts to provide warnings and error messages to the participants to inform them of possible problems.
The RAMI model benchmarking activity is led by a small group of scientists at the Joint Research Centre (JRC) in Ispra (VA), Italy. Information on the rationale and development of the RAMI project can be found on the relevant section. The JRC group will continue to coordinate the RAMI exercise for the foreseeable future, and offers the following services to the community:
The RAMI coordinators may extend deadlines for submitting results or modify other aspects of the exercise if such action is clearly to the advantage of the modelling community and if all qualified participants are given equal opportunity to exploit these changes. In so doing, the coordination group guarantees that RAMI
The RAMI Advisory Body (RAB) is composed of prominent members of the radiation transfer modelling community, and long-standing participants of the RAMI initiative. The functions of the RAB include:
The RAB is the ultimate authority with regards to the conduct of RAMI; it is ultimately responsible for the scientific credibility and integrity of the benchmarking process. Its decisions are final, definitive, irrevocable and cannot be appealed.
The authors/developers of the models maintain full rights on their models. In fact, participation in the RAMI activity does not assume, imply or require any transfer of ownership, intellectual property, commercial rights or software codes between the participants and the RAMI coordinators.
Qualified participants do deliver to the RAMI coordinators the results of computations (typically, suitably formatted tables of numbers representing reflectances and other properties of the simulated radiation field), obtained with their own models, for the explicit purpose of comparing them with similarly derived results from other models. These results become the property of the RAMI coordination group, but the latter, in turn, uses these results exclusively for the stated purpose. Submitted results are not returned to their originators, but they can be updated or modified as needed and within the limits set by the coordination group.
Qualified participants are expected to actively participate in the scientific debate surrounding the RAMI exercise and to contribute to the writing and editing of publications and presentations resulting from this activity. They are automatically co-authors of such publications and presentations, unless they explicitly decline this privilege. Such works are typically submitted for publication by professional organizations (e.g., the American Geophysical Union), which normally own the copyright on the publication itself.
The RAMI coordinators, with the assent of and in collaboration with the qualified participants, publicize the results of the benchmarking exercises, in scientific publications and presentations, as well as through the RAMI web site. By submitting simulation results to the RAMI coordinators, qualified participants authorize the latter to analyse these results and compare them to other similarly derived models, they also accept that the results of this analysis will be published as described earlier. However, individual model results as submitted by qualified participants and intermediary results obtained during the analysis phase are not published without the explicit consent of the authors, and never disclosed to anybody (including other qualified participants) for any reason. Qualified participants may select or approve the publication of their particular model results, but the recommended procedure is for them to organize the delivery of software codes or results themselves.
For practical reasons, the copyrights on the RAMI web site, including the text, figures, tables and other graphical, textual or programming elements remains with the RAMI coordinators. Within the legal limits allowed by copyright laws, figures, tables, statistics and other materials published on the RAMI web site can be downloaded and used in other works, provided full and explicit reference to the source materials are duly provided. For refereed publications, the full reference to the published paper is sufficient. For information gathered on the web site, the full HTTP address of the appropriate web page or site is required.
Qualified participants in RAMI agree to publish the results of the benchmarking as a group, and to refrain from initiating, running or publishing these results, or those resulting from similar benchmarking activities on their own or as a subgroup, unless full and explicit consent by the RAB, the RAMI coordinators and all participants is formally obtained.
Neither the RAB nor the RAMI coordinators take any responsibility for the scientific value or suitability for any particular purpose of the model results submitted in this context. They do however assume responsibility for the analyses and make every effort to ensure the appropriateness, accuracy and fairness of these benchmarks, and request advice or support from the qualified participants as and when needed. The qualified participants, jointly with the RAB and coordinators assume the scientific responsibility of the contents of the publications (both printed and electronic) pertaining to RAMI. If errors or inaccuracies are found after publication, every effort will be made to correct them, e.g., on the RAMI web site or as appropriate.
By its very nature, RAMI is a dynamic, evolving activity. New experiments and benchmarks are designed and proposed at the start of each new phase, both to exercise the participating models under more stringent conditions or to remain in line with the progressive development, complexity and performance of the models.
By the same token, existing models are continually improved and expanded, obsolete codes are disregarded and new approaches and tools are being developed to address past and current limitations, or to take advantage of new findings or observational opportunities.
As a result, the benchmarks and comparative evaluations issued by the RAMI process must be considered snapshots describing the state of the art at the time of the exercise, not as a final, absolute and definitive judgment on the worthiness and performance of any particular model. This also implies that model developers should document the version and release numbers of their codes, that the rating of a particular model can change in time, and that the users of these benchmarks must understand these issues and make sure they associate the results with the proper model version.
Last but not least, if you have any questions or suggestions about these issues, please contact the RAMI coordinators before initiating your participation in a particular phase of the activity.
A scientist qualifies as a full participant to the RAMI4PILPS benchmarking exercise if:
1.1.1: he/she is the original developer of the model to be tested,
or
1.1.2: he/she is duly authorized, by the original developer, to conduct the experiments on his/her behalf,
or
1.1.3: he/she is using a publicly available model,
or
1.1.4: he/she has obtained the go-ahead from the RAMI4PILPS coordinators,
and
1.2: he/she submits the model simulation results within the required constraints (deadline, format, etc),
and
1.3: the volume and quality of results submitted justifies full participation
Each model can participate only once to each exercise of a given phase. In other words, multiple participants cannot simultaneously submit results obtained from the same or very similar models. Prospective participants should contact the RAMI4PILPS coordinators in case of doubt. Qualified participants accept full responsibility for the following tasks:
Since the analysis of the simulation results submitted is largely done automatically, files that do not meet the formatting and other constraints set by the RAMI4PILPS coordinators may be rejected or unused. This may lead to the de facto disqualification of the participant for the particular exercise, or for the entire phase. The RAMI4PILPS coordinators will ensure that all the necessary information and constraints are publicly available and accessible, and will make best reasonable efforts to provide warnings and error messages to the participants to inform them of possible problems. If you have any questions or suggestions about these issues, please contact the RAMI4PILPS coordinators before initiating your participation in the RAMI4PILPS activity.
