Components

DataSetReader

fides::io::DataSetReader is the main class to be familiar with when using Fides. There are three main phases for this class, Initialization, Reading Metadata, and Reading Data.

Initialization

The DataSetReader is set up by passing the data model to the dataModel argument in the constructor and passing the correct DataModelInput type to the inputType argument in the following ways:

Option 1: Passing a path to JSON file containing the data model

This is the default way to use Fides. A JSON file containing the data model description has been created and the path is passed to the DataSetReader.

std::string jsonPath = "/path/to/data/model/json";
// Default, so second argument is not actually needed
fides::io::DataSetReader reader(jsonPath,
  fides::io::DataSetReader::DataModelInput::JSONFile);

Option 2: Passing a string containing valid JSON describing the data model

In this case, the string passed to the DataSetReader contains the JSON data model.

std::string jsonString = "<string containing valid JSON>";
fides::io::DataSetReader reader(jsonString,
  fides::io::DataSetReader::DataModelInput::JSONString);

Option 3: Passing a path to a BP file

It is also possible to get Fides to automatically generate a data model (see Predefined Data Models). In this case, the string passed to the DataSetReader constructor is a path to a BP file that contains the attribute information that Fides will use to generate the data model.

std::string bpFilePath = "/path/to/bp/file";
fides::io::DataSetReader reader(bpFilePath,
  fides::io::DataSetReader::DataModelInput::BPFile);

Optional Step: Setting ADIOS data source parameters

If you need to pass any engine parameters to the ADIOS engine being used, you should use reader.SetDataSourceParameters(). For instance, if you want to the ADIOS SST engine instead of BP files (which Fides uses by default), you could do the following:

// after creating the DataSetReader object
fides::DataSourceParams params; // std::unordered_map<std::string, std::string>
params["engine_type"] = "SST";
reader.SetDataSourceParameters(source_name, params);

source_name is the name of the source provided in the data model. If you have multiple sources, and want to add options for each data source, you’ll need to call this for each data source that you have.

Another option is to set up the parameters for all sources before the constructor and pass it as the third argument.

fides::Params params;
fides::DataSourceParams src1Params;
fides::DataSourceParams src2Params;
src1Params["OpenTimeoutSecs"] = "10";
params["source1"] = src1Params;
src2Params["OpenTimeoutSecs"] = "20";
params["source2"] = src2Params;
fides::io::DataSetReader reader(jsonPath,
  fides::io::DataSetReader::DataModelInput::JSONFile, params);

Optional Step: Setting data source IO object (for inline engine only)

The last possible step for initialization of the DataSetReader is to set the data source’s IO object using reader.SetDataSourceIO(). This is only necessary when data sources are using ADIOS’ inline engine because the reader and writer need to share the IO object.

std::string sourceName = "source";

// setting up ADIOS IO
adios2::ADIOS adios;
adios2::IO io = adios.DeclareIO("inlineIO");

// define ADIOS variables
...
// setting up inline writer
adios2::Engine writer = io.Open("output.bp", adios2::Mode::Write);

// Setting up Fides for reading
fides::io::DataSetReader reader(jsonPath);
fides::DataSourceParams params;
params["engine_type"] = "Inline";
reader.SetDataSourceParameters(sourceName, params);
reader.SetDataSourceIO(sourceName, &io);

Reading Metadata

The next step is to read the metadata, which will give you info such as number of steps, blocks, and available fields.

auto metadata = reader.ReadMetaData();

If any of the data_sources are set as input filename mode or the specified file is not relative to the JSON file, then you’ll need to first set up the paths for the data source(s) In this case, the metadata can be read as follows:

std::unordered_map<std::string, std::string> paths;
paths["source"] = filePath;
auto metadata = reader.ReadMetaData(paths);

From metadata you can get the following info:

// number of blocks
auto& nBlocks = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_BLOCKS());
std::cout << "Number of blocks " << nBlocks.NumberOfItems << std::endl;

// number of steps
auto& nSteps = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_STEPS());
std::cout << "Number of steps " << nSteps.NumberOfItems << std::endl;

// field information
auto& fields = metadata.Get<fides::metadata::Vector<fides::metadata::FieldInformation>(
  fides::keys::FIELDS());
for (auto& field : fields.Data) // fields.Data is a std::vector
{
  std::cout << field.Name << " has association " << field.Association << std::endl;
}

Reading Data

Now we can read the actual data, in either random access mode or go step by step. In either case, we need to create a new fides::metadata::MetaData object that we can use to give Fides some info on the selections we’d like to make. If a paths object was passed to the fides::io::DataSetReader::ReadMetaData() method, that also needs to be passed to the fides::io::DataSetReader::ReadDataSet() method.

Random access of data steps

In this case, we want to choose a step for Fides to read, so we need to set up this information:

fides::metadata::MetaData selections;
fides::metadata::Index step(2); // we want to read step 2
selections.Set(fides::keys::STEP_SELECTION(), step);

Now we can read the dataset:

vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(selections);

If you provided paths to fides::io::DataSetReader::ReadMetaData(), don’t forget to also provide them here.

vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);

Now you’ve got your data in VTK-m format, so you can use the VTK-m API to access the partitions, use filters, etc.

Step streaming

In this case we don’t need to add a step selection to selections. Before we can read the step though, we’ll need to call reader.PrepareNextStep() and check the return value. PrepareNextStep() will return either OK or EndOfStream. If any data source is not ready, Fides will internally loop on that data source until it returns OK or EndOfStream. In the case of multiple data sources, if some source hits EndOfStream before the others (e.g., mesh is stored in a different data source from the variable data and has only one step, while the variables have multiple steps), Fides caches the data from that data source and doesn’t attempt to read steps that do not exist. When PrepareNextStep() returns EndOfStream that means all data sources have finished streaming.

while (true)
{
  fides::StepStatus status = reader.PrepareNextStep();
  if (status == fides::StepStatus::EndOfStream)
  {
    // all
    break;
  }
  // PrepareNextStep only returns EndOfStream or OK
  vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(selections);
  // perform what ever vis/analysis tasks you want on this step
}

Once again, if you are using a paths object to find BP files/streams, they will also need to be provided to fides::io::DataSetReader::PrepareNextStep() and fides::io::DataSetReader::ReadDataSet().

while (true)
{
  fides::StepStatus status = reader.PrepareNextStep(paths);
  if (status == fides::StepStatus::EndOfStream)
  {
    // all
    break;
  }
  // PrepareNextStep only returns EndOfStream or OK
  vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);
  // perform what ever vis/analysis tasks you want on this step
}

Other possible selections

For either reading method, you can provide Fides with some additional selections instead of reading all data. You can choose specific blocks to read (recall from the section on reading metadata that you can find out the total number of blocks available to be read). If no block selections are provided, then Fides will read all blocks by default.

fides::metadata::Vector<size_t> blockSelection;
blockSelection.Data.push_back(1);
selections.Set(fides::keys::BLOCK_SELECTION(), blockSelection);

You can also choose to select specific fields for reading. If no field selection is made, then Fides will read all fields by default.

fides::metadata::Vector<fides::metadata::FieldInformation> fieldSelection;
fieldSelection.Data.push_back(
  fides::metadata::FieldInformation("dpot", vtkm::cont::Field::Association::Points));
selections.Set(fides::keys::FIELDS(), fieldSelection);

Wildcard Fields

Fides supports Wildcard fields which are mostly useful for the data model generation, but can be used in user-created data models as well. This allows for specifying some basic information about variables in the data model, while allowing the names of the associated variables and their cell/point associations to be specified in ADIOS Attributes.

Wildcard Field Attributes
Attribute Name	Possible types/values	Required
`Fides_Variable_List`	vector<string>: variable names	yes
`Fides_Variable_Associations`	vector<string>: variable associations	yes
`Fides_Variable_Sources`	vector<string>: variable data sources	only for XGC
`Fides_Variable_Array_Types`	vector<string>: variable array types	only for XGC

Example JSON

If using wildcard fields in your own data model (i.e., not generated by Fides), a wildcard field looks like this (except for XGC):

{
    "fields": [
        {
            "variable_list_attribute_name": "Fides_Variable_List",
            "variable_association_attribute_name": "Fides_Variable_Associations",
            "array": {
                "array_type": "basic",
                "data_source": "source",
                "variable": ""
            }
        }
    ]
}

So in your Attributes, you will need an attribute called Fides_Variable_List which is a vector of variable names that you want Fides to read. Then the attribute Fides_Variable_Associations is a vector of those variables associations. Each entry should be either points or cell_set.

For an XGC data model, the field will look like:

{
    "fields": [
        {
            "variable_list_attribute_name": "Fides_Variable_List",
            "variable_association_attribute_name": "Fides_Variable_Associations",
            "variable_sources_attribute_name": "Fides_Variable_Sources",
            "variable_arrays_attribute_name": "Fides_Variable_Array_Types",
            "array": {
                "array_type": "",
                "data_source": "",
                "variable": ""
            }
        }
    ]
}

Since XGC has multiple data sources as well as some special handling for certain variables, two additional Attributes are needed. The attribute Fides_Variable_Sources should have the source name for each variable in Fides_Variable_List. The value for each entry with be either mesh or 3d, depending on whether the variable is contained in the xgc.mesh.bp or xgc.3d.bp data sets, respectively. The entries in attribute Fides_Variable_Array_Type, will be basic if it’s a variable that is for a single plane (e.g., pot0) or xgc_field if it’s for all planes (e.g., dpot).

Data Model Generation

Fides can now generate a data model based on some attributes that are written by ADIOS, instead of requiring the user to write their own data model. Currently they must be written as ADIOS Attributes and not Variables, and they can either be written in the same file as your ADIOS data or you could have ADIOS write them in a separate file that contains only the attributes.

Note

If the attributes are written in a file separate from the actual data, Fides expects that it is in the same directory as the data to be read.

General Attributes

This attribute can be used with any data model to use one of your ADIOS Variables as the time values. For instance, in ParaView (whether it’s post-processing or using Catalyst), specifying the time variable will use the values in the specified array instead of just the ADIOS step numbers.

Attributes
Attribute Name	Possible types/values	Default
`Fides_Time_Variable`	string: name of variable to use for time	none

Supported Data Models

For each supported data model there is a table of attributes and possible types/values listed below. If an attribute does not list a default value, it must be specified in the data given to Fides.

Uniform Data Model

The data model uses uniform point coordinates for the coordinate system, needing the origin and spacing to be specified. The cell set is structured based on the dimensions of the data.

Attributes
Attribute Name	Possible types/values	Default
`Fides_Data_Model`	string: `uniform`	none
`Fides_Origin`	3 integer or floating points	none
`Fides_Spacing`	3 integer or floating points	none
`Fides_Dimension_Variable`	string: name of variable to use for determining dimensions	none

Example JSON