DataSetReader

fides::io::DataSetReader is the main class to be familiar with when using Fides. There are three main phases for this class, Initialization, Reading Metadata, and Reading Data.

Initialization

The DataSetReader is set up by passing the data model to the dataModel argument in the constructor and passing the correct DataModelInput type to the inputType argument in the following ways:

Option 1: Passing a path to JSON file containing the data model

This is the default way to use Fides. A JSON file containing the data model description has been created and the path is passed to the DataSetReader.

std::string jsonPath = "/path/to/data/model/json";
// Default, so second argument is not actually needed
fides::io::DataSetReader reader(jsonPath,
  fides::io::DataSetReader::DataModelInput::JSONFile);

Option 2: Passing a string containing valid JSON describing the data model

In this case, the string passed to the DataSetReader contains the JSON data model.

std::string jsonString = "<string containing valid JSON>";
fides::io::DataSetReader reader(jsonString,
  fides::io::DataSetReader::DataModelInput::JSONString);

Option 3: Passing a path to a BP file

It is also possible to get Fides to automatically generate a data model (see Predefined Data Models). In this case, the string passed to the DataSetReader constructor is a path to a BP file that contains the attribute information that Fides will use to generate the data model.

std::string bpFilePath = "/path/to/bp/file";
fides::io::DataSetReader reader(bpFilePath,
  fides::io::DataSetReader::DataModelInput::BPFile);

Optional Step: Setting ADIOS data source parameters

If you need to pass any engine parameters to the ADIOS engine being used, you should use reader.SetDataSourceParameters(). For instance, if you want to the ADIOS SST engine instead of BP files (which Fides uses by default), you could do the following:

// after creating the DataSetReader object
fides::DataSourceParams params; // std::unordered_map<std::string, std::string>
params["engine_type"] = "SST";
reader.SetDataSourceParameters(source_name, params);

source_name is the name of the source provided in the data model. If you have multiple sources, and want to add options for each data source, you’ll need to call this for each data source that you have.

Another option is to set up the parameters for all sources before the constructor and pass it as the third argument.

fides::Params params;
fides::DataSourceParams src1Params;
fides::DataSourceParams src2Params;
src1Params["OpenTimeoutSecs"] = "10";
params["source1"] = src1Params;
src2Params["OpenTimeoutSecs"] = "20";
params["source2"] = src2Params;
fides::io::DataSetReader reader(jsonPath,
  fides::io::DataSetReader::DataModelInput::JSONFile, params);

Optional Step: Setting data source IO object (for inline engine only)

The last possible step for initialization of the DataSetReader is to set the data source’s IO object using reader.SetDataSourceIO(). This is only necessary when data sources are using ADIOS’ inline engine because the reader and writer need to share the IO object.

std::string sourceName = "source";

// setting up ADIOS IO
adios2::ADIOS adios;
adios2::IO io = adios.DeclareIO("inlineIO");

// define ADIOS variables
...
// setting up inline writer
adios2::Engine writer = io.Open("output.bp", adios2::Mode::Write);

// Setting up Fides for reading
fides::io::DataSetReader reader(jsonPath);
fides::DataSourceParams params;
params["engine_type"] = "Inline";
reader.SetDataSourceParameters(sourceName, params);
reader.SetDataSourceIO(sourceName, &io);

Reading Metadata

The next step is to read the metadata, which will give you info such as number of steps, blocks, and available fields. In order to read metadata/data, you’ll need to first set up the paths for the data source(s), then the metadata can be read as follows:

std::unordered_map<std::string, std::string> paths;
paths["source"] = filePath;
auto metadata = reader.ReadMetaData(paths);

From metadata you can get the following info:

// number of blocks
auto& nBlocks = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_BLOCKS());
std::cout << "Number of blocks " << nBlocks.NumberOfItems << std::endl;

// number of steps
auto& nSteps = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_STEPS());
std::cout << "Number of steps " << nSteps.NumberOfItems << std::endl;

// field information
auto& fields = metadata.Get<fides::metadata::Vector<fides::metadata::FieldInformation>(
  fides::keys::FIELDS());
for (auto& field : fields.Data) // fields.Data is a std::vector
{
  std::cout << field.Name << " has association " << field.Association << std::endl;
}

Reading Data

Now we can read the actual data, in either random access mode or go step by step. In either case, we need the paths for the data source(s) we set up earlier before reading the metadata, as well as creating a new fides::metadata::MetaData object that we can use to give Fides some info on the selections we’d like to make.

Random access of data steps

In this case, we want to choose a step for Fides to read, so we need to set up this information:

fides::metadata::MetaData selections;
fides::metadata::Index step(2); // we want to read step 2
selections.Set(fides::keys::STEP_SELECTION(), step);

Now we can read the dataset:

vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);

Now you’ve got your data in VTK-m format, so you can use the VTK-m API to access the partitions, use filters, etc.

Step streaming

In this case we don’t need to add a step selection to selections. Before we can read the step though, we’ll need to call reader.PrepareNextStep() and check the return value. PrepareNextStep() will return either OK or EndOfStream. If any data source is not ready, Fides will internally loop on that data source until it returns OK or EndOfStream. In the case of multiple data sources, if some source hits EndOfStream before the others (e.g., mesh is stored in a different data source from the variable data and has only one step, while the variables have multiple steps), Fides caches the data from that data source and doesn’t attempt to read steps that do not exist. When PrepareNextStep() returns EndOfStream that means all data sources have finished streaming.

while (true)
{
  fides::StepStatus status = reader.PrepareNextStep(paths);
  if (status == fides::StepStatus::EndOfStream)
  {
    // all
    break;
  }
  // PrepareNextStep only returns EndOfStream or OK
  vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);
  // perform what ever vis/analysis tasks you want on this step
}

Other possible selections

For either reading method, you can provide Fides with some additional selections instead of reading all data. You can choose specific blocks to read (recall from the section on reading metadata that you can find out the total number of blocks available to be read). If no block selections are provided, then Fides will read all blocks by default.

fides::metadata::Vector<size_t> blockSelection;
blockSelection.Data.push_back(1);
selections.Set(fides::keys::BLOCK_SELECTION(), blockSelection);

You can also choose to select specific fields for reading. If no field selection is made, then Fides will read all fields by default.

fides::metadata::Vector<fides::metadata::FieldInformation> fieldSelection;
fieldSelection.Data.push_back(
  fides::metadata::FieldInformation("dpot", vtkm::cont::Field::Association::Points));
selections.Set(fides::keys::FIELDS(), fieldSelection);