Data post-filtering

Prev Next

In this document

You'll learn how to run Post-filtering, the last QC step, which flags valid values that remain scattered between flagged ones: opening the tool via Quality/Post filtering, selecting which parameters to include, configuring the four filtering thresholds (minimum consecutive valid values, minimum valid values per day, and two high-sun-elevation limits for irradiance), and running the check. Also covers examining results in the Heatmap flag plot (with split-by-year and time zone options), the time series flag plot and histogram, the Quality control summary tables, generating a PDF report, and saving your flags.

Prerequisites

  • An active Solargis Analyst account.

  • Installed an Analyst desktop application.

  • One or more datasets available in the Solargis Analyst application.

Post-filtering QC

Post-filtering quality control (QC) is the last check to detect extremely short consecutively passed values (usually under 10) positioned between otherwise flagged values. These values can not be used for any calculations or in simulations and need to be flagged, too.

Key points to remember:

  • Post-filtering covers both irradiance and meteo parameters. Two of the four configuration thresholds apply to irradiance parameters only.

  • Post-filtering QC check should be done last, after automatic and manual QC checks.

  • Post-filtering QC check adds post-filtering flags to scattered values based on the configuration.

  • Graphs and statistics are displayed as soon as you open the window, before you run the check. Running post-filtering updates them with the newly added flags.

Post-filtering QC tool

You can open the post-filtering QC check tool from the main menu via Quality/Post-filtering or by using the quick access icon located below the main menu. The window interface offers the following:

  1. Run automatic checks: Executes the post-filtering check. The results are displayed in the plot section and the filtered data is flagged with the post-filtering flag (blue by default). This button is highlighted as the primary action before the first run.

  2. Visualization and summary tabs: Toggle between different visualizations to examine the post-filtering flags or view the quality control summary.

  3. Data visualization: Provides visual insights into where the post-filtering has been applied. The flags legend is located above the charts.

  4. Zoomed-in view: Zoomed-in visualization chart to see better where the post-filtering flag has been added.

  5. Available solar and meteo parameters: Select which parameters to include in post-filtering. Deselect the ones you want to skip, for example calculated parameters or parameters that have no QC applied.

  6. Parametrize post-filtering: Set the four thresholds that control which values get flagged. Click the icon next to each field for details.

  7. Plot settings and parameter selector: Select which parameter to display for examination and set how to visualize the data.

Executing the post-filtering QC check

When you open the Post-filtering window, all parameters and all graphs are already displayed, so you can inspect the current state of the dataset before filtering. Configure the parameters and thresholds below, then run the check to add the post-filtering flags.

  1. In the parameter list, deselect any parameters you don't want to post-filter, such as calculated parameters or parameters without QC applied. All parameters are selected by default.

  2. Set the filtering thresholds, or keep the defaults. See the table below for what each one does.

  3. Click the "Run" button to execute the check. A busy indicator appears while post-filtering runs, and a success message confirms when it finishes.

  4. Review the results in the graphs and the Quality control summary, then save.

Filtering thresholds

Four thresholds in the Parametrize post-filtering table control how aggressively post-filtering flags your data. Defaults are sensible for most datasets — adjust them only when you have a reason to.

Threshold

Applies to

Default

What it does

Minimum Consecutive Valid Values

All parameters

Calculated from the dataset time step

Flags short sequences of valid data surrounded by invalid data when the sequence is shorter than the threshold.

Minimum Valid Values Per Day

All parameters

50%

Invalidates a whole day when the share of valid measurements for that day falls below the threshold.

Limit for High Sun Elevation

Irradiance only

50%

Identifies the peak sun period using a quantile approach.

Valid Values for High Sun Elevation

Irradiance only

50%

Flags a whole day when valid measurements during the high-sun period fall below the required percentage.

Tip: Each threshold has an  icon next to it in the interface with additional detail about how the value is applied.

Examining the post-filtering QC results

You can examine the data and flags at any time, both before and after running the check. Several visualizations are at hand to help you analyze the dataset and the flagged data, and you can navigate them using the top navigation menu:

  1. Heatmap flag plot: View the flags in a timeline heatmap. Values flagged with the post-filtering flag are displayed in blue by default (see legend above the chart).

  2. Time series flag plot: Lets you view the post-filtering flags in two charts: a time series chart and a histogram of flag values per month.

  3. Quality control summary: Provides a summary of the checks per parameter performed in the dataset.

Tip: Use the toolbar controls to work with the charts and examine the flags closely. You can learn how to work with them here.

Heatmap flag plot

The heatmap flag plot is the default tab you will see after the post-filtering QC check finishes. It gives you a first look at the data and flags that have been automatically added to the identified values.  

  1. Heatmap: Shows the color-coded data and flags in a timeline. The Y axis represents 24-hour day, and X axis represents the timeline of the dataset. One heatmap contains one year of data by default, which you can change with the Split heatmap by year option. Green color represents correct values, and various color points depict the flags in the timeline. Post-filtering flags are displayed in blue by default.

  2. Identified flags: Flags included in the dataset for the selected parameter, their respective color in the heatmap, and what they represent.

  3. Parameter selector: Select the parameter you want to display in the heatmap flag plot. The first parameter from the top is selected by default.

The heatmap offers two display options:

  • Split heatmap by year: On by default, giving you one heatmap per year of data. Turn it off to show a single heatmap containing all rows.

  • View in time zone: Set by default to the geographical time zone of the data. You can change it manually to any other time zone — useful when you need to cross-check against data recorded in a different reference.

Time series flag plot

The time series flag plot tab lets you examine the flags in two ways - in a time series chart, where you can zoom and see particular flags, and in a monthly statistics chart, where you can visually compare the amount of each flag value per month.

  1. Flagged data: Values flagged with a post-filtering flag.

  2. Flags list: A color-coded list of flags included in the dataset for the selected parameter. The same applies to the second chart.

  3. Histogram: Each color represents a single flag. The amount of each flag is depicted in percent. This can help you quickly spot how much good data (green) is available for each month.

  4. Parameter selector: Select the parameter you want to display in the charts.

Note: Night values are filtered out of the histogram by default. To include them, clear the "Filter out night values" option in the Quality control summary tab.

Quality control summary

The quality control summary tab provides a comprehensive breakdown of the post-filtering QC checks. Flag statistics are presented in two tables:

  • The upper table covers irradiance parameters (and PV parameters where present).

  • The lower table covers meteo parameters.

Use the "Filter out night values" option to exclude values recorded while the sun is below the horizon. This option is on by default and applies to irradiance parameters only, so it affects the upper table.

Generating a post-filtering report

You can generate a report directly from the Post-filtering window, without going back to the main window. Use the "Create report" button to produce a PDF report capturing the post-filtering results, the flag statistics and the charts.

Finishing post-filtering QC check

Once the check has finished, the "Save" button becomes the primary action. Use it to apply the changes and add the post-filtering QC status to the dataset.

If you close the Post-filtering window while you still have unsaved changes, a confirmation dialog asks whether you want to discard them.