Importing and exporting datasets

Prev Next

In this document

We will teach you how to import and export datasets in the Analyst application.

Dataset import manager

The Analyst application includes a dataset import manager, allowing you to add new data to your project or database. The data must be in one of the supported formats. Supported formats are csv, dat, Excell, or any other text files in table format.

You can access the import wizard by navigating to the main menu’s Dataset/Import/Import files or clicking the quick access icon under the main menu.

The importing procedure includes the following steps:

  1. Selecting datasets and data for import: Browse for the datasets you want to import and prepare them for further steps.

  2. Setting up the correct date/time format of the dataset: Define which data/time format the dataset uses.

  3. Setting up timestamp offset and aggregation interval position: Align the timestamp with the timezone and set the timestamp position inside the aggregation interval.

  4. Creating the dataset: Create a new dataset, append the data to an existing one, and add maintenance logs.

Selecting datasets and data to import

As a first step, you must open the dataset import manager and add the datasets you want to import:

  1. Click the “Add files” button to browse for your datasets.

  2. Added datasets will show in the main window, automatically selected for import (checkbox).

The application will analyze the data, identify the used data separators and encoding method, and attempt to parse the file into columns by individual parameters. If this does not happen automatically (e.g., all data is in a single column), you must manually prepare the data for import.

Manual data preparation

If the dataset did not parse correctly and individual parameters are not in separated into columns, you will have to prepare the data manually. Proceed with the following step, checking the dataset rendering changes in the dataset preview window to the right (displays only the first 1000 lines):

  1. Select the correct data separator or add custom (not included in the options) if required.

  2. Exclude the required number of lines from the top of the file in case it contains more header lines or skip lines with a defined special character.

  3. Set correct decimal and thousands separators if required.

  4. Try changing the encoding method if everything else fails to work.

  5. Define which row contains the parameter names if these are not displayed correctly. The column names that will be used are also listed for verification. The parameter names currently selected are displayed in bold.  

Note: You can view the original data file using the “View original file” button at the bottom left to help you identify the structure of the dataset and used separators.  

Selecting parameters to import

If your data is correctly parsed and parameters displayed in separate columns, you can proceed to select the parameters you want to import from each uploaded dataset:

  1. Select the dataset you want to import the data from in the left window.

  2. In the right window, begin selecting parameter columns you want to import from this dataset. Remember to also select every date/time column to pair the data with its timestamps.

  3. Scroll to reveal more parameters to select.

  4. Once you have selected all parameters you want to import from all datasets, proceed to the next step. It may take some time depending on the size of the dataset.

Tip: You can use the “Clear” and “Select all” buttons to quickly select/deselect all parameters at once.

Setting up the correct date/time format of the dataset

Once you have prepared the data and selected the columns to import, the application will try to identify date/time columns and set their format automatically. If the date/time was not recognized, you will need to specify the format used in the dataset and indicate which columns contain this information. The timestamp check window to the right indicates which timestamps have not been recognized:

  1. Select all columns containing timestamp data.

  2. Visually analyze the timestamp format in the columns and select it in the format examples window.

  3. Use the provided button to add the selected format to the Active formats window.

  4. If required, you can manually overwrite the timestamp format to accommodate the dataset if required.

  5. Use advanced options if the date/time is included in the name.

  6. Once all timestamps have been identified correctly, the timestamp check window to the right is empty, and you may proceed to the next step.

Note: When proceeding to the next step you may get notified that some text is still not recognized as a timestamp. It will be due to selected date/time column names that are still displayed in the non-recognized notification window. This is not an issue and you can proceed to the next step.

Setting up timestamp offset and aggregation interval position

After correct recognition and configuration of the timestamps, you will set the timestamp offset and the timestamp’s position in the aggregation interval. This step ensures that your data is accurately aligned with the correct time periods, which is essential for reliable analysis and interpretation.

  1. If required, set the hour/minute time offset for the imported dataset or pick the country where the data was measured.

  2. Select the position within the interval that the dataset's timestamp represents. Center is selected by default. You can change it to start or end of the interval if required.

  3. The timestep values in the data list will reflect your selected interval position. When finished, proceed to the next step.

Note: If you need to reference the original file, use the “View original file” button at the bottom left.

Creating the dataset

In the final step, you will enter the dataset details—such as the name, data source, site name, and other identifiers—to ensure clear identification. At this stage, you will also choose whether to create a new dataset or add data to an existing one, and select the database where the dataset will be stored.

Tip: This window provides an option to save the import settings as a template to speed up your import process for similar datasets. Use the “Save import” button at the top right of the window to save the template.

Creating a new dataset

If you want to create a new dataset and add it to a specific database, follow these steps:

  1. Leave the default “Create a new dataset” option selected.

  2. Select the database you want to store the dataset in, enter the dataset metadata in the dedicated fields, and specify the data type. These will help you identify the dataset in the database. Pay particular attention to GPS coordinates, as these are crucial for ensuring data analytics accuracy.

  3. Add parameter metadata for each parameter in the dataset (select the parameter in the columns window). If you skip this step, you can do it later in the Metadata editor.

  4. When finished, click the “Create new dataset” button to create the dataset and add it to the selected database.

Tip: You can use automatic parameter detection (“Detect param.type” button) to automatically identify the parameters. The color of the text indicates the detection confidence: green represents very high confidence, while gray means the system could not identify the parameter type.

Note: Your dataset is now visible and ready in the Project datasets window. You can edit its details using the Metadata editor it required.

Appending the data to an existing dataset

If you want to add the data to an existing dataset to update it (e.g., upload additional months of data), proceed with the following steps:

  1. Select the “Append rows to existing dataset” option.

  2. Select the dataset you want to append the data to.

  3. Decide what action to take when overlapping timestamps are found.

  4. Verify and match the imported data with the correct columns in the dataset.

  5. When finished, click the “Update existing dataset” button to append the data to the dataset.

Note: If the imported data contains parameters not present in the dataset, application will create new columns to add the new parameter data.

Attaching a maintenance log

If the imported dataset contains maintenance log data, you can use it to add maintenance events to a dataset.

  1. Select the “Attach as maintenance log” option.

  2. Select the dataset you want to attach the maintenance events to.

  3. If required, select a parameter to which to attach the maintenance log. It will attach the maintenance events to all by default.

  4. Select the maintenance log column in the “Filter events by” field and select which value represents the maintenance event.

  5. Click the “Attach maintenance log” button when finished to add maintenance logs to the selected dataset.

Tips:

  • If required, change how to handle overlapping timestamps with the dataset using the selectors at the bottom left.

  • You can save the import as a template to use the same format in the future using the “Save import” button at the top-right.

Importing an exchange file

If you have an Analyst exchange file, you can import it to your project. Exchange files are Analyst-specific files created for seamless data transfers between different Analyst application instances or projects. It is supported in the Analyst application only.

To import an exchange file, navigate to the Dataset/Import/Import Exchange File in the main menu and select the exchange file you want to import.

Exporting the datasets

Analyst offers dataset exports in various formats to accommodate different scenarios. You can export datasets as:

  • CSV file: Standard CSV file format readable by most applications.

  • Exchange file: Creates an Analyst-specific file to transfer data between Analyst applications. Ensures no loss of data and a smaller file size.

  • PVsyst standard format: Exports the dataset in PVsyst-compatible format.

CSV file export

  1. Open the CSV export from the main menu via Dataset/Export/CSV.

  2. Choose how you want to export the header information.

  3. Use the checkboxes in the Columns section to include/exclude parameters from the export.

  4. Select how to export the maintenance logs. The maintenance log header can only be imported if it is included in the data table. It cannot be imported via the importer on its own.  

  5. Browse the folder where the export will be saved.

Click “Export” to initiate the export.

Export exchange file

Exchange files are Analyst-specific files created for seamless import/export of the datasets between various Analyst applications or projects. The exchange file ensures that all metadata is correctly exported and no values will be lost or misinterpreted. Use this export type when you plan to transfer a dataset to another Analyst instance or project. It will save the exchange file to the selected location in the export window.

Note: The exchange file size is smaller compared to the CSV export.

PVsyst Standard Format file export

PVsyst Standard Format export will export the data to the PVsyst-compatible format.

Important: PVsyst Standard Format export supports harmonized datasets without gaps and with hourly temporal resolution only.  

  1. Open the PVsyst export from the main menu via Dataset/Export/PVsyst Standard Format.

  2. Select the dataset to export.

  3. Use the checkboxes in the Columns section to include/exclude parameters from the export.

  4. Browse the folder where the export will be saved.

Click “Export” to initiate the export.