In this document
You will learn why accurate solar simulation software and high-quality input data are vital for optimizing energy predictions and system design. Knowing key indicators and related terminology helps understand the expected range of deviations in the calculated PV output.
Overview
Every decision in a solar energy project relies on accurate data. In the context of PV software, accuracy ensures that the calculated values for a power plant are reliable. Reliable data supports objectives, such as achieving an optimal design, securing financing for future projects, and serving as a benchmark for performance assessment of existing power plants.
The most effective way to evaluate the performance of solar radiation models is by comparing their outputs with reference values—a process known as model validation. This comparison yields validation statistics that is crucial for assessing the models' performance and reliability. An understanding of terminology is essential to interpret these statistics effectively and draw reliable conclusions.
Validating the accuracy of expected PV power output from a solar energy simulation software, such as Solargis, is not straightforward. First, it involves validating solar irradiance, meteorological and environmental data input used in the simulation, always ensuring that measured data used as a reference complies with requirements. Second, it involves the verification of models and algorithms involved in the PV calculation chain (ray-tracing, single-diode, etc). All these topics are addressed in the accuracy section of the knowledge base.
Beyond weather data input and model validation, errors in the technical specifications of PV modules and inverters can introduce inaccuracies when running the software. Evaluating the reliability of software outputs requires verifying the confidence class of these components through automated and expert checks—a process known as component verification, which is explained in more detail in the PV component catalog section of the knowledge base.
Finally, analyzing validation statistics reveals a model's strengths and weaknesses, enabling the development of secondary models to estimate output uncertainty.
Terminology
In general, accuracy is measured by how close is to the true value. In practice, this requires identifying reference values that are as close to the truth as possible. Rather than directly comparing with the absolute truth—which is often unreachable—we compare data of unknown accuracy against reference values with known and higher accuracy.
In the context of evaluating accuracy in solar energy systems, understanding key terms is essential for interpreting data correctly and managing expectations for energy yield predictions:
Validation is the process of assessing a model, software, or dataset against reliable reference data or a standard to confirm its correctness and suitability for a specific purpose.
Verification is the process of evaluating whether the provided technical specifications align with the actual reference specifications. This ensures that the characteristics used in energy simulations are compliant with the real-world specifications.
Consistency refers to the reliability of data or predictions over time or across different scenarios. Consistent results with minimal erratic variability reinforce trust in the data and models.
Resolution is the smallest detectable change or increment (temporal or spatial) in the data. For solar measurements, this might represent the smallest unit of irradiance that a sensor can detect, as well as the typical time interval of measurement and spatial (geographical) representation, for example (for the case of grid data) the size of a grid cell.
Uncertainty quantifies the range within which the true value is expected to lie, typically expressed as a margin of error. It accounts for limitations in measurements and models, helping users understand potential deviations.
Confidence represents the degree of certainty that a value lies within a specific range. For example, a 95% confidence interval indicates a 95% probability that the true value falls within the defined bounds.
Validation statistics
When comparing a model's outputs with ground-based data reference, the goal is to assess the similarity between the two datasets for the given site and time period. To ensure meaningful comparison, a set of statistical indicators should be employed. These indicators help quantify the performance and identify potential discrepancies.
Here is a description of key statistical indicators for model-measurement comparisons:
Bias: quantifies the average difference between the model outputs and reference values. Ideally, the bias should be close to zero, indicating minimal systematic error in the model.
Mean Absolute Deviation (MAD): is calculated by averaging the absolute error values, ensuring that all deviations are treated equally. This approach prevents positive and negative errors from canceling each other out during summation, providing a clearer picture of the overall error magnitude.
Root Mean Square Deviation (RMSD): is determined by averaging the squared deviations, which gives greater weight to larger errors. This makes it particularly effective for identifying and emphasizing the impact of significant discrepancies. RMSD can be applied at various aggregation levels, such as sub-hourly, hourly, or daily values, offering flexibility in assessing model performance across different timescales.
To make comparisons meaningful across datasets or metrics, normalization is applied. Normalized indicators are expressed as percentages relative to a reference value, facilitating cross-comparisons between different scenarios or datasets.
Indicator | What does it represent | Expected value | ||
---|---|---|---|---|
Bias | Bias or Mean Bias Deviation | Characterizes systematic model deviation at a given site, i.e. systematic over- or underestimation. | A bias closer to zero means lower deviation | |
MAD | Mean Absolute Deviation | Shows an indication of the spread of error. It is used to avoid deviations with opposite signs canceling each other out. | A MAD closer to zero means a lower error | |
RMSD | Root Mean Square Deviation | Shows an indication of the spread of error, giving higher weight to larger errors. It should be calculated at different granularities (sub-hourly, hourly, daily, monthly). | A lower RMSD means a lower spread of error |
Summary table with main indicators to be calculated for each validation site.
In addition to traditional error metrics, unit-invariant indicators provide valuable insights. These include:
Correlation Coefficient (R): Indicates the strength and direction of the linear relationship between the model and reference data.
Coefficient of Determination (R²): Reflects the proportion of variance in the reference data explained by the model.
Kolmogorov-Smirnov Index: Measures the maximum difference between the cumulative distributions of the model and reference datasets, highlighting distributional discrepancies.
Indicator | What does it represent | Expected value | ||
---|---|---|---|---|
R | Bias or Mean Bias Deviation | Characterizes systematic model deviation at a given site, i.e. systematic over- or underestimation. | A bias closer to zero means lower deviation | |
R2 | R2 | Mean Absolute Deviation | Shows an indication of the spread of error. It is used to avoid deviations with opposite signs canceling each other out. | A MAD closer to zero means a lower error |
KSI | Root Mean Square Deviation | Shows an indication of the spread of error, giving higher weight to larger errors. It should be calculated at different granularities (sub-hourly, hourly, daily, monthly). | A lower RMSD means a lower spread of error |
Once a sufficient number of validation points have been collected, calculating aggregated statistics becomes essential to summarize and analyze the overall model performance. Key aggregated metrics include:
Number of reference samples: The number of reference samples collected and used in the validation study. For example, when validating the performance of irradiance models, this refers to the number of locations where a solar database has been tested and validated against actual ground measurements.
Mean of All Biases: This represents the average bias across all validation points, providing an overall measure of the systematic error in the model. A mean bias close to zero indicates that, on average, the model does not consistently overestimate or underestimate the reference values.
Standard Deviation of Biases: This measures the variability or spread of biases around the mean bias. A lower standard deviation indicates that the model's errors are consistently close to the mean, whereas a higher value suggests significant variability in performance across the validation points.
Mean Absolute Bias (MAB): The average of the absolute values of biases, which accounts for both positive and negative errors without cancellation, offering a clearer representation of the magnitude of error.
Requirements for reference data
Validation is essential for ensuring the accuracy and reliability of results provided by solar and meteorological databases and energy simulation and forecasting software. To achieve this, models are typically compared with ground-measured data from multiple locations, i.e., validation sites. This process enables researchers to assess the performance and accuracy of the models.
Validating models introduces further challenges because measured conditions are never entirely perfect. Solar and meteorological data validation requires expert knowledge to understand the instruments and analyze the measured values after they pass through specific quality control procedures.
For validation results to be meaningful, several key factors must be considered: the quality of the reference data, the number of validation samples, and the variety of situations covered by the validation sites (both geographically and technologically.
Quality of reference data
The quality of validation data is one of the most important aspects of reference data to ensure valid comparisons. Combining high-quality data with lower-quality data can produce skewed results that do not accurately reflect the model's performance.
For solar and meteorological data, this means that the ground-based measurements used to test the model must be accurate, well-maintained, and captured using top-class sensors capable of reflecting actual site conditions. This also includes:
Properly calibrated instruments.
Consistent data collection protocols.
Reliable maintenance of measurement stations.
Quantity of validation samples
The quantity of validation sites refers to the number of reference points where the model is tested. Validation statistics derived from a single reference point cannot provide a representative picture of the model performance. Comparisons should be conducted using a sufficient number of validation points.
Differences in the number of validation sites between models can skew comparison results. For instance, a solar radiation model validated with fewer sites may not fully reflect its true accuracy. Without an adequate number of validation points, especially in key regions where solar projects are planned, developers may face higher uncertainty.
For solar power projects, where every percentage of accuracy matters, it is crucial to estimate accuracy using data from the highest possible number of sites. The complexity of factors affecting solar radiation model uncertainty requires testing with as many validation sites as possible.
Geographic distribution of validation sites
The geographic distribution of validation points is critical in determining the representativeness of validation statistics. Solar radiation patterns vary significantly across regions due to latitude, altitude, climate, and local geography.
A well-distributed set of validation sites ensures the model has been tested across diverse geographic conditions. Solar irradiance is influenced by cloud cover, humidity, aerosol concentration, and local geographic features. By analyzing long-term ground sensor data collected under varied conditions, researchers can ensure that solar radiation models are robust across various scenarios.
If validation sites are clustered in specific regions while others are underrepresented, the resulting statistics may not fully capture the model’s performance in untested areas. For example, a model validated primarily in Europe may not perform as well in tropical or desert regions if those areas are underrepresented.
Variety of situations and technologies tested
When testing a simulation model, it is also important to validate it under various scenarios. For example, different PV module configurations and power plant layouts should be tested when assessing optical simulation models. Similarly, different PV technologies should be tested when evaluating energy conversion inside PV modules.
Testing under diverse conditions leads to a more robust validation process and better accuracy estimates for the model.
From validation to uncertainty
Analyzing validation statistics provides valuable insights into the performance of a model, enabling a deeper understanding of its strengths and limitations. Once sufficient knowledge of a model’s performance is obtained, this information can be leveraged to develop secondary models that estimate the uncertainty of the model outputs.
For specific models, such as satellite-based solar radiation data, the accumulated expertise of organizations like Solargis enables the derivation of reliable uncertainty estimates. In essence, this means using validation results from multiple sites to predict the uncertainty for a specific site. As a result of this effort, we can provide an expected range of deviation for annual satellite-modeled GHI (Global Horizontal Irradiance) and DNI (Direct Normal Irradiance) values as explained in our methodology.
Uncertainty models vary in complexity, ranging from simple approximations to advanced probabilistic frameworks. Despite the widespread accessibility of PV energy yield simulations through advanced software, the uncertainties in these simulations remain underexplored. Key sources of uncertainty include input parameters, model assumptions, and varying operational conditions.