Navigation path

LEft

18-Oct-2017   
Path: FAQs
  Up Print version Decrease text Increase text Home

Frequently Asked Questions:

This page aims at overcoming eventual starting-difficulties that potential RAMI participants may experience. To this effect a series of questions and answers is provided below:

General issues regarding RAMI and its participation:
  1. What is a RAMI phase?
  2. How can I participate with my model(s) in a RAMI phase?
  3. How many of the prescribed scenes do I need to generate?
  4. How many of the prescribed measurements do I need to perform?
  5. What is the privacy and data usage policy of RAMI?
  6. Can I participate with multiple RT models?
  7. What if other RAMI participants use a similar version of a publicly available RT model than myself?
  8. How can I contact the RAMI coordinators?
  9. What is new in RAMI-IV?
Specific issues regarding models, results and the submission process:
  1. What if my model cannot exactly simulate the structure of a proposed test case?
  2. What if my model cannot generate all of the required measurements?
  3. Can I submit files in other formats than ASCII (txt)?
  4. Do I need to adhere to the file naming convention?
  5. Can I provide additional information about my model (including pictures)?
  6. What format do my result files have to be in?
  7. Can I check the format of the output-files of my model before submitting all these results?
  8. How do I submit my results?
  9. What if I discover an error in already submitted results?
  10. Why do I need to pay attention to the reference plane when performing measurements?


General issues regarding RAMI and its participation:

  1. What is a RAMI phase? up

    The expression 'RAMI phase' refers to a time period during which the radiation transfer within an ensemble of predefined test cases (i.e., scenarios where both the illumination conditions and the structural and spectral properties of the illuminated geophysical environment are prescribed) has to be simulated by the qualified RAMI participants (using their models), and the results of these simulations must be submitted for subsequent analysis. A RAMI phase is declared closed when all results have been made public, initially and in a more concise form via presentations on conferences as well as a publication in the refereed scientific literature, and ultimately and more comprehensively by using this web site. Conceptually a 'RAMI phase' can thus be divided into:

    • The launch period, during which the RAMI coordinators decide upon what test cases and measurements to incorporate into the next phase of RAMI. For each of these test cases, the proper documentation will then be placed on the RAMI web site. The final step being the announcement of the launch of a new phase of RAMI via email broadcast to the radiation transfer modeling community.
    • The active submission period, during which the RAMI participants visit the RAMI web site in order to obtain the necessary specification for the test cases to be performed during that phase. This information then leads to the implementation of the test cases within the models of the participants, as well as the subsequent generation of the required measurements and their submission to a location specified on the RAMI web site. The earlier the submission of a participant's results, the greater the chance that obvious mistakes can be identified and he/she be able to re-submit a new set of results prior to the closing date of the active submission period.
    • The results analysis and publication period, during which the RAMI coordinators analyse the submitted results and report files from all participants, and produce a panoply of graphical results and statistics files, some of which will be placed onto the RAMI web site. In difficult or controversial cases, the RAMI coordinators may seek a decision from the RAMI advisory body, which among others, sets and maintains the overall work ethics of the RAMI initiative. Last but not least, a paper with the most important results may also be submitted to a scientific journal.

    As soon as a new phase of RAMI is launched, its Active submission period is indicated at the top of the home page for that phase (i.e., the web page that is obtained when clicking on the RAMI-123 buttons in the left-hand frame). Similarly, as soon as results have been synthesized and made available for inspection on the web, the On-line results publication date will also be shown on that web page.

    Image of RAMI phase scheme

  2. How can I participate with my model(s) in a RAMI phase? up

    Participation in the RAMI initiative is open to everyone subject to the criteria listed in the role, rights and responsibilities of the RAMI qualified participants section of the DISCLAIMER page. Please note, however that only models with traceable publications in the refereed scientific literature may feature in any resulting RAMI publication. Other models may still use RAMI as a testbed to evaluate their performance through comparison with graphical results files where available. Note that our privacy and data-usage policy does not allow the distribution of data submitted by individual RAMI participants. Upon decision to participate in a RAMI phase, it would be courteous to inform the RAMI coordinator of your intent (via the email link at the bottom of every web page within the RAMI site).

    All information regarding the structural and spectral properties of the various geophysical environments (scenes), their illumination conditions and associated measurements can be accessed via the various links available at the main level of that phase (to find this click on the appropriate RAMI* button in the left hand frame). When implementing the structural scenario of a given 3-D experiment, you should aim to stick as close as possible to the information provided. Obviously different models have different RT formulations and also scene representation capabilities. This is why - in the discrete cases - we provided large text files containing the location and orientation of every leaf in the scene. Obviously such information would be of no relevance for some geometric optical models that may need a description of the envelope of the foliage volumes instead. In any case, we hope that you will find sufficient information on the various experiment description pages that allow your model to generate geophysical environments that are quite similar to those proposed for a given phase. If you feel, however, that some information is missing please do not hesitate in sending an email to us. We also suggest that you note any approximations (or better still a short description of the functioning of your model) in the report file that you will submit with your results files later on. Please make also sure that you utilize the proper 'definitions' regarding angular sign conventions, leaf normal distributions, and other RT model technicalities prior to starting your simulations (in particular the reference level to which the RT model simulations refer to). Also read the relevant file naming and formatting conventions that must be adhered to by all participants. Again, in case of doubts or problems please do not hesitate in contacting us.

    Upon completion of your model simulations, you are strongly encouraged to perform an on-line format check, to see if your model has produced correctly named and formatted result-files (ideally you need to check only one sample output file of your model for each of the recommended RAMI measurements for a given phase). To do this online-format-checking just click on the pulsating online-format-checker link located in the top navigation bar and follow the instructions provided. Notice that this link will only be visible during the active submission period of a given RAMI phase (i.e., the period of time for which the submission of simulation results is valid). Once you have checked the correctness of the filenames and formats of your model outputs (in the sense of the RAMI file-naming and formatting specifications), please contact the RAMI coordinator (here) and request a password for the formal submission of the complete set of your results (and report) files to our ftp site. Note that although it may not be possible for all models to perform all experiments and measurements provided we encourage you to submit as many test cases as possible since this will allow better to pinpoint the strengths and weaknesses (if any) of your model(s).

    Participants that do not perform the suggested online-format-check on typical output files of their models risk the eventuality that their submitted results files may be rejected by the RAMI coordinator on the grounds of incorrect file naming and/or formatting. In that case the participant will have to engage in the correction and re-submission of (sometimes many hundreds of data) files if he/she wishes to have his/her model participating. To avoid such last minute stress, you are encouraged to use the online-format-checker to make sure that your results adhere to the RAMI formatting and file naming conventions prior to the formal ftp submission of all your results (and report) files. Participants will be contacted by the RAMI coordinators to acknowledge receipt of their data files.

  3. How many of the prescribed scenes do I need to generate? up

    The simple answer to this question is: all of those that your model is capable of simulating. Within RAMI, the performance of each model is described by comparing its output to those of all (mutually different) models that completed the same experiment i.e., the prescribed set of measurements on a given scene structure. Exhaustive testing of your model in all feasible scene structures is thus strongly encouraged, as this effort will provide you with the best diagnostics of the performance of your model. However, given the various degrees of complexity that exist between different radiation transfer models, RAMI experiments were subdivided into two sets of test cases: structurally homogeneous environments that can be represented by 1-D and 3-D RT models, and structurally heterogeneous environments that can only be represented with 3-D RT models. One-dimensional RT models thus need to perform only those test cases prescribed under homogeneous environments, whereas three-dimensional RT models need to perform both the experiments prescribed under structurally homogeneous and heterogeneous environments. In fact it is strongly recommended that 3-D models perform the proposed set of homogeneous experiments. This will allow to ascertain their mutual consistency prior to the launching of RT simulations in environments with increased structural complexity. Similarly before addressing highly specialized simulations within heterogeneous domains, it may be advantageous to first perform the various baseline scenarios in the heterogeneous category, that is the ''floating spheres'' experiments. In general, RAMI participants should aim at performing all of the prescribed test cases (within the applicability of their model), since the behaviour of any given RT model may depend (sometimes very strongly) on the kind of structural and also spectral conditions that are encountered. In this context particular attention should also be given to the various purist corner scenarios.

  4. How many of the prescribed measurements do I need to perform? up

    Again, the simple answer to this question is: all of those that your model is capable of simulating. As was already mentioned previously, within RAMI, the performance of a model is described with respect to as comprehensive as possible an ensemble of structural and spectral scenarios, and this over multiple types of measurements. Participation in as large as possible a number of measurements is thus a necessity for the comprehensive evaluation of a particular model. Spectral albedo, for example, is of interest since it may give an idea of the absolute accuracy of a model simulation under conservative scattering conditions (purist corner). Similarly the collided, uncollided and multiple collided BRF components can help identify issues in the mathematical formulation within computer codes that otherwise may remain untraceable from total BRF simulations alone. Simulations in the principal plane may point to issues regarding the hotspot formulations. In general, the more simulations are available from any given model the more confidence can be placed in its overall evaluation.

    Please make sure that you utilize the proper 'definitions' regarding angular sign conventions, leaf normal distributions, and other RT model technicalities prior to starting your simulations. Also read the relevant file naming and formatting conventions that must be adhered to by all participants.

  5. What is the privacy and data usage policy of RAMI? up

    All data and information submitted by a RAMI participant belongs to the RAMI coordinators for the specific and exclusive purpose of running this benchmarking exercise. The RAMI coordinators will not distribute results or report files that were submitted by its participants. Further information about our privacy and data-usage policy can be found here.

  6. Can I participate with multiple RT models? up

    The participation with multiple models is generally permissible. This, however, brings up the question as to when a model is in fact a new model, rather than a replacement of an older version of the same model (for example due to enhanced computing powers, etc). If multiple models are at the disposition of a given participant, and if these models can all be traced to some common ancestor model, then the interested participant should decide (for the benefit of the scientific community) which one of these is the most suited one to satisfactorily complete all the prescribed test cases and measurements within RAMI. Failure to do so may prompt the RAMI coordinators to exclude some of the submitted RT models from participation in any subsequent results analysis - in particular if the output of these models are very similar, and their differences can be traced to slight modifications of some common ancestor model that do not impact significantly on a better representation the physics of radiative transfer contained within them. It is thus recommended to participate only with one RT model - unless significant methodological differences exist between the models that are to be submitted by the same participant.

  7. What if other RAMI participants use a similar version of a publicly available RT model than myself? up

    Each model can participate only once to each exercise of a given phase. In other words, multiple participants cannot simultaneously submit results obtained from the same or similar models. Prospective participants should contact the RAMI coordinators in case of doubt. For more details on the role, rights and responsibilities of the RAMI qualified participants, click here.

  8. How can I contact the RAMI coordinators? up

    The RAMI coordinator can be contacted with the following e-mail.

  9. What is new in RAMI-IV? up

    RAMI-IV is a continuation of previous phases of RAMI but due to the availability of the RAMI On-line Model Checker (ROMC) all canopy scenes were newly designed for RAMI-IV. More specifically, in RAMI-IV two different types of plant environments are available: 1) abstract canopies, that are generated only on the basis of disc-shaped scatterers, and 2) actual canopies,that are based on inventories of actual forest and plantation test sites. Within these canopies a series of new measurements apply:

    New measurements applying to all canopy scenes:


    New measurements applying to selected canopy scenes:

    • Isotropic Bihemispherical Reflectance (White Sky Albedo) is a new domain-level measurement. It follows the new flux measurement format and applies to all ACTUAL canopies having an isotropic diffuse illumination component, i.e., not the ABSTRACT canopies nor the ACTUAL plantation canopies.
    • Total canopy absorption (foliage and wood): is the new name for the old domain-level 'Canopy absorption' (fabs) measurement. It follows the new flux measurement format and applies to all ACTUAL canopies.
    • Total transmission at lower boundary level for diffuse illumination only is a new domain-level measurement. It follows the new flux measurement format and applies to all ACTUAL canopies having an isotropic diffuse illumination component, i.e., not the ABSTRACT canopies nor the ACTUAL plantation canopies.
    • Uncollided transmission at lower boundary level for diffuse illumination only is a new domain-level measurement. It follows the new flux measurement format and applies to all ACTUAL canopies having an isotropic diffuse illumination component, i.e., not the ABSTRACT canopies nor the ACTUAL plantation canopies.
    • Collided by canopy only transmission at lower boundary level for diffuse illumination only is a new domain-level measurement. It follows the new flux measurement format and applies to all ACTUAL canopies having an isotropic diffuse illumination component, i.e., not the ABSTRACT canopies nor the ACTUAL plantation canopies.
    • brf in an azimuthal ring (total): This measurement applies to all ACTUAL canopies. The measurement consists of computing the BRF for a zenith angle of 37 degree and all azimuths from 0 - 358 degree in steps of 2 degrees. The format is the same as for standard BRF measurements.
    • Local uncollided transmission at lower boundary for direct illumination from finite-sized sun: These transects of the direct transmission from a finite-sized sun have their own format and apply to all but the ABSTRACT HOM canopies (due to the length of the transects).
    • Thresholded Hemispherical Photography (binary fish-eye image) is a new measurement intended to document the angular distribution of gaps in the overstorey when looking upward from nine selected locations within a given RAMI scene. This measurement has its own format and applies to all ACTUAL canopies.
    • Total waveform LIDAR return signal is a new measurement mimicking the total return signal of a waveform lidar. It has its own format and applies to all but the HOMOGENEOUS ABSTRACT canopies (due to the width of the Lidar footprint).
    • Single collided waveform LIDAR return signal is a new measurement mimicking the single scattered return signal of a waveform lidar. It has its own format and applies to all but the HOMOGENEOUS ABSTRACT canopies (due to the width of the Lidar footprint).

    In addition all ACTUAL CANOPIES scenes have 19 different spectral bands for the prescribed measurements to be performed. ACTUAL CANOPIES with snow as background medium as well as ABSTRACT CANOPIES with anisotropic background conditions use the parametric RPV model to describe the directional BRF variations of the background surface. Finally a series of the ACTUAL CANOPY scenes uses spectral band specific ratios of direct to total incident radiation.


Specific issues regarding regarding models, results and the submission process:

  1. What if my model cannot exactly simulate the structure of a proposed test case? up

    Canopy reflectance models are developed to simulate the reflectance fields of existing types of vegetation architecture. The choice of representations of such vegetation canopies is primarily related to the dimensionality of the RT model, but also to the computational constraints at the time of its conception, as well as the potential field(s) of application. Particularly for the panoply of existing 3-D RT models the scene representation strategies may vary widely. Beyond the actual performance of a particular model it is thus of interest to determine what the actual limits and strengths of these different scene implementations within RT models are. Hence for any RT model participating within RAMI, the goal should be to represent the prescribed geophysical environments to the best of their abilities. This means that within the constraints provided by the canopy formulation of that model, the structural properties of a given scene should be described as faithfully as possible to the original description (provided on the corresponding HTML pages). For some sophisticated models this may mean to use the (optional) ASCII fiels that we provide to reproduce exactly the location and orientation of all individual scatterers in the scene. Other models, may do well by generating their scenes in a manner that is suitable for their own models, provided that structural parameters like canopy height, leaf area index, leaf area density, etc. are respected at least at the level of the scene if not at the level of every individual object. In any case a detailed report file should be included during the submission of the measurement results, that describes to what extent the actual implementation of the test case differed from that described on the RAMI web pages. Remember that nobody is going to `loose face' when participating in an inter-comparison exercise like RAMI. Identifying the limits of a model, or discovering significant differences amidst model results is the best driving force for improvements.

  2. What if my model cannot generate all of the required measurements? up

    Within the RAMI phase a series of measurements have been defined that allow to test a variety of issues regarding the simulation of the radiation transfer in the solar domain and within terrestrial environments. The larger the number of performed test cases and also measurements the more complete the evaluation of any given RT model will be. Participants are strongly encouraged to simulate all prescribed measurements within the capabilities of their model. If some measurements cannot be performed then please indicate the reasons for not doing so in the report files.

  3. Why do I need to pay attention to the reference plane when performing measurements? up

    Within RAMI all radiation transfer simulations are carried out with respect to a reference plane. Only those portions of the incoming and exiting radiation that pass through this reference plane are to be considered in the various RAMI measurements. Unless specified otherwise, the default reference plane within RAMI covers the entire test case area (known as the "scene") and is located at the top-of-the-canopy height, that is, just above the highest structural element in the scene. The spatial extend of the reference plane can be envisaged as the (idealized) boundaries of the IFOV of a perfect sensor looking at a 'flat' surface located at the height level of the reference plane. Changing the height, location and extend of this reference area will obviously affect your simulation results.

  4. Can I submit files in other formats than ASCII (txt)? up

    No. Due to the large number of files that need to be processed, only ASCII files (plain text) are accepted provided that they adhere to the file naming and formatting convention. Similarly, results files should not be compressed when submitting them via ftp. However, prior to sending any result you are strongly encouraged to perform an on-line format check of the results files that your model has produced (ideally you check one sample output file of your model for each of the recommended RAMI measurements). To do this click on the pulsing online-format-checker link located in the top navigation bar and follow the instructions provided there. Notice that this link will only be visible in the top navigation bar of an active phase of RAMI (i.e., one for which the submission period is still valid). Once your output results are conform with the RAMI file naming and formatting specifications, please contact the RAMI coordinator (here) and request a password for the formal submission of the complete set of your results (and report) files to our ftp site.

  5. Do I need to adhere to the file naming convention? up

    Yes. The filenaming convention is mandatory since it allows the automatic processing of the submitted results. During the second phase of RAMI almost 4200 different files were included in the analysis process - a manual verification of individual files is thus not feasible. Filenames that do not adhere to the prescribed file naming convention will be excluded from the subsequent process of analysing and publishing the results. Thus, prior to the sending of any results, you are strongly encouraged to perform an on-line format check, to see if your model has produced correctly named and formatted result-files (ideally you check only one sample output file of your model for each of the recommended RAMI measurements for that phase). To do this online-format-checking just click on the pulsing online-format-checker link located in the top navigation bar and follow the instructions provided there. Notice that this link will only be visible in the top navigation bar of an active phase of RAMI (i.e., a RAMI phase for which the submission period is still valid). Once you have checked the correctness of the filenames and formats of your model outputs (in the sense of the RAMI filenaming and formatting specifications), please contact the RAMI coordinator (here) and request a password for the formal submission of the complete set of your results (and report) files to our ftp site.

  6. Can I provide additional information about my model (including pictures)? up

    Yes. During the submission process you may include a file written in HTML - called <MOD>.php, where <MOD> is the identifier name of your model - containing a lengthy description of the model's underlying mathematical concepts as well as their implementations within your RT model. References to studies published in the scientific literature in which your RT model featured may also be provided. Similarly, GIF images of scene representations and applications may be included in the HTML document (don't forget to send the GIF images as well).

  7. What format do my result files have to be in? up

    All submitted files must be of type ASCII (plain text). Detailed information as to the precise format of the results and report files can be found here. You may also consult the measurements definition pages directly for examples of results files. Adherence to these format conventions is mandatory since the large number of submitted files (almost 4200 during RAMI-2) prevents any manual editing by the RAMI organizers. Incorrectly formatted results files will be excluded from the intercomparison. Please perform the on-line format check, to see if your model has produced correctly named and formatted result-files (ideally you check only one sample output file of your model for each of the recommended RAMI measurements for that phase). To do this online-format-checking just click on the pulsing online-format-checker link located in the top navigation bar and follow the instructions provided there. Notice that this link will only be visible in the top navigation bar of an active phase of RAMI (i.e., a RAMI phase for which the submission period is still valid). Once you have checked the correctness of the filenames and formats of your model outputs (in the sense of the RAMI filenaming and formatting specifications), please contact the RAMI coordinator (here) and request a password for the formal submission of the complete set of your results (and report) files to our ftp site.

  8. Can I check the format of the output-files of my model before submitting all these results? up

    Yes, RAMI provides an on-line interface allowing the participants of an active phase to verify the file-naming and formatting of their model-generated results files. In fact, you are strongly encouraged to perform an on-line format check of the results files that your model has produced (ideally you check only one sample output file of your model for each of the recommended RAMI measurements). To do this click on the pulsing online-format-checker link located in the top navigation bar and follow the instructions provided there. Notice that this link will only be visible in the top navigation bar of an active phase of RAMI (i.e., one for which the submission period is still valid). Once your output results are conform with the RAMI file naming and formatting specifications, please contact the RAMI coordinator (here) and request a password for the formal submission of the complete set of your results (and report) files to our ftp site.

  9. How do I submit my results? up

    The submission of your results files is done via ftp. However, prior to submitting your results to our ftp site you are strongly encouraged to perform an on-line format check, to see if your model has produced correctly named and formatted result-files (ideally you check only one sample output file of your model for each of the recommended RAMI measurements). To do this online-format-checking just click on the pulsing online-format-checker link located in the top navigation bar and follow the instructions provided there. Notice that this link will only be visible in the top navigation bar of an active phase of RAMI (i.e., the submission period is still valid).

    Once you have checked the correctness of the filenames and formats of your model outputs (in the sense of the RAMI filenaming and formatting specifications), please contact the RAMI coordinator (here) and request a password for the formal submission of the complete set of your results (and report) files to our ftp site. Having received your password from the RAMI coordinator, the subsequent ftp submission of your result-files can be done using the following guidelines:

    Participants that upload one single file (or data archive - generated with the commands tar, gzip, compress, bzip2, zip, or pkzip), which contains their entire set of RAMI-format compliant measurements and report files, should use:

    ftp h05-ftp.jrc.it
    Login: ramiplayer
    Password: *******
    cd RAMI-IV
    put [your file names]
    bye
    

    Participants that upload multiple files each one of which being a RAMI format compliant measurement (.mes) or report (.rep) file should use:

    ftp -i h05-ftp.jrc.it
    Login: ramiplayer
    Password: *******
    cd RAMI-IV
    mput *mes
    mput *rep
    bye
    

    Notice that you will not be able to see the RAMI-IV directory once the ftp connection has been established. Furthermore, you will not be able to download any data unless you actually 'cd' into the RAMI-IV directory. Again the 'ls' command is disabled but you should be able to see whether you had a successful upload of data from the ftp prompts, i.e. Ok to send data and File receive OK.

    Participants that do not perform the suggested online-format-check on typical output files of their models risk the eventuality that their submitted results files may be rejected by the RAMI coordinator on the grounds of incorrect file naming and/or formatting. In that case the participant will have to engage in the correction and re-submission of (sometimes many hundreds of data) files if he wishes to have his model participating. To avoid such last minute hectics, you are encouraged to use the online-format-checker to make sure that your results adhere to the RAMI formatting and file naming conventions prior to the formal ftp submission of all your results (and report) files.

  10. What if I discover an error in already submitted results? up

    If this happens, then please contact the RAMI coordinator using the email address provided at the bottom of each RAMI web page, and proceed to redo your measurements anew. Once they are finish re-submit them (preferably including those that had been correct the first time round). Again do not forget to make sure that you adhere to the RAMI filenaming and formatting conventions. By default the latest version of any measurement file will be used for subsequent analysis.


    Right navigation