jmetal 튜토리얼 (4) Experimental studies

This section describes the features included in jMetal for performing experimental studies related to assessing and comparing the performance of multi-objective algorithms on a set of problems.

 

Prerequisites

Any multi-objective algorithm in jMetal is aimed at finding a Pareto front approximation of a problem, which is returned at the end of the search as a list of solutions.

Given an algorithm solving a continuous problem, the result of the optimization can be obtained with the getResult() method as follows:

 

“VAR.csv” : contains the values of the solutions (variables) 

“FUN.csv” contains the functions, using a comma as a separator.

Thus, if the list has five solutions of problem (non-dominating) whose formulation requires four decision variables, the contents of a “VAR.tsv” file would look like this:

 

Experimental studies

- make a comparison of a number of algorithms when solving a set of problems.

-  execution and subsequent analysis of the results of R independent runs of A algorithms, each of will solve P problems. This process is carried out in three steps:

 

1. Execution of all the configurations (i.e., all the combinations of A, P, and R)
2. Apply quality indicators to the obtained fronts
3. Performance assessment by using statistical tests and generating stuff summarizing the results (e.g., Latex tables, charts)

These steps are supported by a set of classes included in the jmetal-lab sub-project, package org.uma.jmetal.lab.experiment.

 

This package has the following structure:

└── jmetal-lab: org.uma.jmetal.lab.experiment
    ├── Experiment
    ├── ExperimentBuilder
    ├── component
        ├── experimentComponent
        ├── impl
            ├── ExecuteAlgorithms
            ├── ComputeQualityIndicators
            ├── GenerateLatexTablesWithStatistics
            ├── GenerateHtmlPages (NEW)
            ├── GenerateReferenceParetoFront
            ├── GenerateReferenceParetoSetAndFrontFromDoubleSolutions
            ├── GenerateFriedmanHolmTestTables (NEW)
            ├── GenerateBoxplotsWithR
            └── GenerateWilcoxonTestTablesWithR
    ├── studies
        ├── ZDTStudy
        ├── ZDTComputingReferenceParetoFrontsStudy
        ├── NSGAIIStudy
        ├── NSGAIIComputingReferenceParetoFrontsStudy
        ├── ...
        └── BinaryProblemsStudy

We can observe that there is a class called Experiment (which has an associated ExperimentBuilder class), which can be populated with a number of components.

The mentioned steps are performed by 

1. ExecuteAlgorithms (step 1), 

2. ComputeQualityIndicators (step 2),

3. the rest of components can be selected to produce a variety of elements to analyze the results, such as Latex tables, figures (boxplots), and HTML pages (a new feature in jMetal).

 

To compute quality indicators, it is necessary to have a reference front per problem; when solving benchmark problems, these fronts are usually known (there are located by default in the resources folder of the jMetal project), but this is not the case when dealing with real-world problems.

 

To cope with this issue, we include the GenerateReferenceParetoFront class, which produces reference Pareto fronts from all the results yielded by all the runs of all the algorithms after executing the ExecuteAlgorithms component, and the related GenerateReferenceParetoSetAndFrontFromDoubleSolutions, which does the same if the problems to solve are continuous; in this case, a reference Pareto set is also generated, as well as files with the contributed solutions of each algorithm to this set.

 

To show how these components can be used in an experiment, we have included a number of examples in the studies package. We explain next the ZDTStudy and the NSGAIIComputingReferenceParetoFrontsStudy classes.