DataSetReader

fides::io::DataSetReader is the main class to be familiar with when using Fides. There are three main phases for this class, Initialization, Reading Metadata, and Reading Data.

Initialization

The DataSetReader is set up by passing the data model to the dataModel argument in the constructor and passing the correct DataModelInput type to the inputType argument in the following ways:

Option 1: Passing a path to JSON file containing the data model

This is the default way to use Fides. A JSON file containing the data model description has been created and the path is passed to the DataSetReader.

std::string jsonPath = "/path/to/data/model/json";
// Default, so second argument is not actually needed
fides::io::DataSetReader reader(jsonPath,
  fides::io::DataSetReader::DataModelInput::JSONFile);

Option 2: Passing a string containing valid JSON describing the data model

In this case, the string passed to the DataSetReader contains the JSON data model.

std::string jsonString = "<string containing valid JSON>";
fides::io::DataSetReader reader(jsonString,
  fides::io::DataSetReader::DataModelInput::JSONString);

Option 3: Passing a path to a BP file

It is also possible to pass an ADIOS filename (whether that is an actual BP file, or the name of an SST stream) to Fides. In this case, you must have an ADIOS Attribute called “fides/schema” containing the data model JSON, or you need to have certain attributes that Fides recognizes to automatically generate a data model (see Data Model Generation). In this case, the string passed to the DataSetReader constructor is a path to an ADIOS file/stream that contains the attribute information that Fides will use to read/generate the data model.

std::string bpFilePath = "/path/to/bp/file";
fides::io::DataSetReader reader(bpFilePath,
  fides::io::DataSetReader::DataModelInput::BPFile);

If you are using multiple ADIOS sources, there are some restrictions when providing the data model through an ADIOS attribute. Currently, only one SST source can be used at a time. You can combine the use of BP files with an SST source. For instance, you may have a static mesh which is written to a BP file, while using the SST engine for the array data. In this case, Fides requires that the JSON is in the BP file and not the SST stream.

In the case of a single source which uses the SST engine, then the JSON can be in the SST stream, however, you must write that attribute on every timestep. This is because it’s possible for the reader to connect at a later time, or to disconnect and reconnect later. Writing the JSON attribute(s) on every timestep ensures that Fides can access this.

One final note when using SST engine and the data model is contained in the stream or a BP file, you must set the engines in the Params and pass that to the DataSetReader constructor. It is not possible to set later using reader.SetDataSourceParameters() discussed below.

fides::DataSourceParams meshParams, fieldParams;
meshParams["engine_type"] = "BPFile";
fieldParams["engine_type"] = "SST";
fides::Params params;
params["mesh"] = meshParams;
params["fields"] = fieldParams;

std::string bpFilePath = "/path/to/bp/file";
fides::io::DataSetReader reader(bpFilePath,
  fides::io::DataSetReader::DataModelInput::BPFile, true /*streaming mode*/, params);

Optional Step: Setting ADIOS data source parameters

If you need to pass any engine parameters to the ADIOS engine being used, you should use reader.SetDataSourceParameters(). For instance, if you want to the ADIOS SST engine instead of BP files (which Fides uses by default), you could do the following:

// after creating the DataSetReader object
fides::DataSourceParams params; // std::unordered_map<std::string, std::string>
params["engine_type"] = "SST";
reader.SetDataSourceParameters(source_name, params);

source_name is the name of the source provided in the data model. If you have multiple sources, and want to add options for each data source, you’ll need to call this for each data source that you have.

Another option is to set up the parameters for all sources before the constructor and pass it as the third argument.

fides::Params params;
fides::DataSourceParams src1Params;
fides::DataSourceParams src2Params;
src1Params["OpenTimeoutSecs"] = "10";
params["source1"] = src1Params;
src2Params["OpenTimeoutSecs"] = "20";
params["source2"] = src2Params;
fides::io::DataSetReader reader(jsonPath,
  fides::io::DataSetReader::DataModelInput::JSONFile, params);

Optional Step: Setting data source IO object (for inline engine only)

The last possible step for initialization of the DataSetReader is to set the data source’s IO object using reader.SetDataSourceIO(). This is only necessary when data sources are using ADIOS’ inline engine because the reader and writer need to share the IO object.

std::string sourceName = "source";

// setting up ADIOS IO
adios2::ADIOS adios;
adios2::IO io = adios.DeclareIO("inlineIO");

// define ADIOS variables
...
// setting up inline writer
adios2::Engine writer = io.Open("output.bp", adios2::Mode::Write);

// Setting up Fides for reading
fides::io::DataSetReader reader(jsonPath);
fides::DataSourceParams params;
params["engine_type"] = "Inline";
reader.SetDataSourceParameters(sourceName, params);
reader.SetDataSourceIO(sourceName, &io);

Reading Metadata

The next step is to read the metadata, which will give you info such as number of steps, blocks, and available fields.

auto metadata = reader.ReadMetaData();

If any of the data_sources are set as input filename mode or the specified file is not relative to the JSON file, then you’ll need to first set up the paths for the data source(s) In this case, the metadata can be read as follows:

std::unordered_map<std::string, std::string> paths;
paths["source"] = filePath;
auto metadata = reader.ReadMetaData(paths);

From metadata you can get the following info:

// number of blocks
auto& nBlocks = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_BLOCKS());
std::cout << "Number of blocks " << nBlocks.NumberOfItems << std::endl;

// number of steps
auto& nSteps = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_STEPS());
std::cout << "Number of steps " << nSteps.NumberOfItems << std::endl;

// field information
auto& fields = metadata.Get<fides::metadata::Vector<fides::metadata::FieldInformation>(
  fides::keys::FIELDS());
for (auto& field : fields.Data) // fields.Data is a std::vector
{
  std::cout << field.Name << " has association " << field.Association << std::endl;
}

Reading Data

Now we can read the actual data, in either random access mode or go step by step. In either case, we need to create a new fides::metadata::MetaData object that we can use to give Fides some info on the selections we’d like to make. If a paths object was passed to the fides::io::DataSetReader::ReadMetaData() method, that also needs to be passed to the fides::io::DataSetReader::ReadDataSet() method.

Random access of data steps

In this case, we want to choose a step for Fides to read, so we need to set up this information:

fides::metadata::MetaData selections;
fides::metadata::Index step(2); // we want to read step 2
selections.Set(fides::keys::STEP_SELECTION(), step);

Now we can read the dataset:

viskores::cont::PartitionedDataSet output = reader.ReadDataSet(selections);

If you provided paths to fides::io::DataSetReader::ReadMetaData(), don’t forget to also provide them here.

viskores::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);

Now you’ve got your data in Viskores format, so you can use the Viskores API to access the partitions, use filters, etc.

Step streaming

In this case we don’t need to add a step selection to selections. Before we can read the step though, we’ll need to call reader.PrepareNextStep() and check the return value. PrepareNextStep() will return either OK or EndOfStream. If any data source is not ready, Fides will internally loop on that data source until it returns OK or EndOfStream. In the case of multiple data sources, if some source hits EndOfStream before the others (e.g., mesh is stored in a different data source from the variable data and has only one step, while the variables have multiple steps), Fides caches the data from that data source and doesn’t attempt to read steps that do not exist. When PrepareNextStep() returns EndOfStream that means all data sources have finished streaming.

while (true)
{
  fides::StepStatus status = reader.PrepareNextStep();
  if (status == fides::StepStatus::EndOfStream)
  {
    // all
    break;
  }
  // PrepareNextStep only returns EndOfStream or OK
  viskores::cont::PartitionedDataSet output = reader.ReadDataSet(selections);
  // perform what ever vis/analysis tasks you want on this step
}

Once again, if you are using a paths object to find BP files/streams, they will also need to be provided to fides::io::DataSetReader::PrepareNextStep() and fides::io::DataSetReader::ReadDataSet().

while (true)
{
  fides::StepStatus status = reader.PrepareNextStep(paths);
  if (status == fides::StepStatus::EndOfStream)
  {
    // all
    break;
  }
  // PrepareNextStep only returns EndOfStream or OK
  viskores::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);
  // perform what ever vis/analysis tasks you want on this step
}

Other possible selections

For either reading method, you can provide Fides with some additional selections instead of reading all data. You can choose specific blocks to read (recall from the section on reading metadata that you can find out the total number of blocks available to be read). If no block selections are provided, then Fides will read all blocks by default.

fides::metadata::Vector<size_t> blockSelection;
blockSelection.Data.push_back(1);
selections.Set(fides::keys::BLOCK_SELECTION(), blockSelection);

You can also choose to select specific fields for reading. If no field selection is made, then Fides will read all fields by default.

fides::metadata::Vector<fides::metadata::FieldInformation> fieldSelection;
fieldSelection.Data.push_back(
  fides::metadata::FieldInformation("dpot", viskores::cont::Field::Association::Points));
selections.Set(fides::keys::FIELDS(), fieldSelection);