API

User API

DataSetReader class

class DataSetReader

General purpose reader for data described by an Fides data model.

fides::io::DataSetReader reads data described by an Fides data model and creates datasets in a backend-agnostic manner. See the Fides schema definitionfor the supported data model. DataSetReader also supports reading meta-data.

Public Types

enum class DataModelInput

Input types when setting up the DataSetReader

Values:

enumerator JSONFile: Brief input is path to a JSON file with the data model

enumerator JSONString: Brief input is JSON containing the data model stored in a string

enumerator BPFile: Brief input is a BP file that contains attributes that provide details for the predefined data model to be generated by Fides

Public Functions

DataSetReader(const std::string &dataModel, DataModelInput inputType = DataModelInput::JSONFile, const Params &params = Params(), bool createSharedPoints = false)

Constructor to set up the Fides reader

See also

DataModelInput

Parameters:

dataModel – the value should be 1) a path to a JSON file describing the data model to be used by the reader, 2) a string containing valid JSON, 3) a path to a BP file containing attributes that Fides can use to generate a data model, or 4) a path to a BP file that contains a fides/schema attribute that contains the full JSON for the data model.
inputType – specifies what is stored in the dataModel arg. Optional
streamSteps – if true, the reader will be set up to use a streaming mode (even if BP engine)
params – a map of ADIOS engine parameters to be used for each data source. Optional

void SetDataSourceParameters(const std::string &source, const DataSourceParams &params)

Sets the parameters for a given data source. Currently, only the inline engine requires this to be called, which must happen before attempting to read data.

Parameters:

source – name of the DataSource, which should match a data_sources name given in the data model JSON.
params – a map of parameters and their values

void SetDataSourceIO(const std::string &source, void *io)

Set the IO for a given source. This call should only be used when using the inline engine and must be called before attempting to read data or metadata.

Parameters:

source – name of the DataSource, which should match a data_sources name given in the data model JSON.
io – pointer to the ADIOS IO object

void SetDataSourceIO(const std::string &source, const std::string &io)

Set the IO for a given source. This call should only be used when using the inline engine and must be called before attempting to read data or metadata.

Parameters:

source – name of the DataSource, which should match a data_sources name given in the data model JSON.
io – the address to an ADIOS IO object, stored in a string

fides::metadata::MetaData ReadMetaData(const std::unordered_map<std::string, std::string> &paths = {}, const std::string &groupName = "")

Read and return meta-data. This includes information such as the number of blocks, available fields etc.

Parameters:

paths – a map that provides the paths (filenames usually) corresponding to each data source.
groupName – looks for metadata on variables inside the given group.

std::unique_ptr<fides::DataContainer> ReadDataSet(const std::unordered_map<std::string, std::string> &paths, const fides::metadata::MetaData &selections, fides::DataSetType dsType = fides::DataSetType::Viskores)

Read heavy-data and return an opaque wrapper containing the backend- specific dataset.

Parameters:

paths – a map that provides the paths (filenames usually) corresponding to each data source.
selections – provides support for reading a subset of the data by providing choices for things such as time and blocks.
dsType – Dataset type to produce. Allowed values depend on backend support selected at configure time (currently possibilities are Viskores and VTK).

std::unique_ptr<fides::DataContainer> ReadDataSet(const fides::metadata::MetaData &selections, fides::DataSetType dsType = fides::DataSetType::Viskores)

Read and return heavy-data.

Parameters:: selections – provides support for reading a subset of the data by providing choices for things such as time and blocks.

StepStatus PrepareNextStep(const std::unordered_map<std::string, std::string> &paths = {})

When reading in streaming mode, this method has to be called before reading any meta-data or heavy data. It will also move the reader to the next step. Fides will loop on a data source while ADIOS reports that it is NotReady, but the user should also check the return which could return fides::StepStatus::OK or fides::StepStatus::EndOfStream. If EndOfStream, all steps have been read.

Parameters:: paths – a map that provides the paths (filenames usually) corresponding to each data source.

std::vector<std::string> GetDataSourceNames(): Get std::vector of DataSource names.

std::set<std::string> GetGroupNames(const std::unordered_map<std::string, std::string> &paths): Get all available group names.

std::vector<std::string> GetDataSetNames(const std::unordered_map<std::string, std::string> &paths = {}): Unified item-name enumeration for both schema shapes. For a multi-dataset (datasets[]) schema returns the schema-declared names in declaration order; paths is ignored. For a single-dataset schema read against a multi-group .bp, opens the step source and returns the discovered ADIOS variable groups in sorted order. With an empty paths and a single-dataset schema, no source can be opened so the result is empty (legacy behaviour). The returned names are exactly the keys consumed by DATASET_SELECTION.

void Close(): Closes all open DataSources.

Public Static Functions

static bool CheckForDataModelAttribute(const std::string &filename, const std::string &attrName = "Fides_Data_Model")

Checks a bp file for an attribute containing information that Fides can use to generate the data model. Static so that it doesn’t require setting up the DataSetReader first. Useful for applications like ParaView, where it wants to check if it can use Fides to read a file without needing to configure Fides first.

Parameters:

filename – Name of file to check
attrName – Name of attribute to look for

class DataSetReaderImpl

Public Functions

inline bool IsMultiDataset() const: True when the schema declared a datasets[] block (multi-dataset PDC). More than one entry, or one entry with a non-empty name.

inline fides::datamodel::DataObjectModel &PrimaryModel() const: The first/primary model — the single model for legacy schemas, or the first dataset of a datasets[] schema (used for whole-schema queries like step/time that are shared across PDC peers).

inline std::unique_ptr<fides::datamodel::DataObjectModel> BuildModelFor(const rapidjson::Value &obj): Build a model for a schema body, dispatching on required members: cell_attributes → CellGridModel, coordinate_system → DataSetModel. Works for legacy bodies and datasets[] entry bodies alike.

inline void ParseAssemblyTree(const rapidjson::Value &json, fides::OutputBuilder::AssemblyNode &out, const std::set<std::string> &declaredNames): Recursively translate the JSON assembly tree into an OutputBuilder::AssemblyNode, validating every leaf datasets reference against the declared dataset names.

inline std::vector<std::string> GetDataSetNames(const std::unordered_map<std::string, std::string> &paths): Unified item enumeration. For a datasets[] schema returns the schema-declared names in declaration order; paths is ignored. For a single-dataset schema read against a multi-group .bp, queries the step source via GetGroupNames and returns the discovered groups in name-sorted order. With an empty paths and a single-dataset schema, no source can be opened so the result is empty.

inline bool RequiresVTK() const: Any dataset requiring VTK forces the whole collection to VTK output.

inline void FilterFieldsForModel(fides::metadata::MetaData &sel, fides::datamodel::DataObjectModel &model): Drop FIELDS entries the model does not own, so a caller selection spanning PDC peers doesn’t trip a sub-model’s “unknown field” guard.

Public Members

std::vector<DatasetEntry> Datasets: Schema-declared datasets. Single-dataset schema = 1 entry, empty Name; datasets[] schema = N named entries.

std::vector<std::string> DataSetNames: Schema-declared dataset names (datasets[] order). Empty for a legacy single-dataset schema (the “is this a PDC schema?” probe).

bool HasAssembly = false: Optional assembly tree (multi-dataset schemas only).

std::vector<fides::datamodel::DataObjectModel*> ItemModels: Model backing each collection item, in CreateItem order. Populated by ReadDataSetInternal so PostRead can run per item.

struct DatasetEntry: One schema-declared dataset. A legacy single-dataset schema yields one entry with an empty Name; a datasets[] schema yields one entry per child, Name doubling as the ADIOS variable group prefix and the DATASET_SELECTION key.

Keys and MetaData

KeyType fides::keys::NUMBER_OF_BLOCKS(): Key used for storing number of blocks meta-data. Uses fides::metadata::Size

KeyType fides::keys::NUMBER_OF_STEPS(): Key used for storing number of steps meta-data. Uses fides::metadata::Size

KeyType fides::keys::BLOCK_SELECTION(): Key used for selecting a set of blocks. Uses fides::metadata::Vector<size_t>

KeyType fides::keys::FIELDS(): Key used for available array meta-data and array selection. Uses fides::metadata::Vector<fides::metadata::FieldInformation>

KeyType fides::keys::STEP_SELECTION(): Key used for selecting time step. Uses fides::metadata::Index

KeyType fides::keys::PLANE_SELECTION(): Key used for selecting planes for XGC data. Should only be used internally. Uses fides::metadata::Set

struct Size : public fides::metadata::MetaDataItem

Meta-data item to store size of things such as number of blocks.

Public Functions

inline Size(size_t nItems): constructor

Public Members

size_t NumberOfItems: Number of items (e.g., blocks)

struct Index : public fides::metadata::MetaDataItem

Meta-data item to store an index to a container.

Public Functions

inline Index(size_t idx)

Public Members

size_t Data

struct FieldInformation

Simple struct representing field information.

Public Functions

inline FieldInformation(std::string name, fides::FieldAssociation assoc)

Public Members

std::string Name: Name of the field.

fides::FieldAssociation Association: Association of the field.

template<typename T> struct Vector : public fides::metadata::MetaDataItem

Meta-data item to store a vector.

Public Functions

inline Vector(std::vector<T> vec)

Vector() = default

Public Members

std::vector<T> Data

template<typename T> struct Set : public fides::metadata::MetaDataItem

Meta-data item to store a set.

Public Functions

inline Set(std::set<T> data)

Set() = default

Public Members

std::set<T> Data

class MetaData

Container of meta-data items. This class is a simple wrapper around an std::map that makes setting/getting a bit easier. Internally, it stores objects using unique_ptrs but the interface uses stack objects.

Public Functions

template<typename T> inline void Set(fides::keys::KeyType key, const T &item): Add a meta-data item to the map. Supports subclasses of MetaDataItem only.

template<typename T> inline const T &Get(fides::keys::KeyType key) const: Given a type, returns an object if it exists. Raises an exception if the item does not exist or if the provided template argument is incorrect.

inline void Remove(fides::keys::KeyType key): Given a key, removes the item from the map.

inline bool Has(fides::keys::KeyType key) const: Given a key, checks whether an item exists.

Useful enums and typedefs

enum class fides::StepStatus

Possible return values when using Fides in a streaming mode.

Values:

enumerator OK

enumerator NotReady

enumerator EndOfStream

enumerator OtherError

using fides::DataSourceParams = std::unordered_map<std::string, std::string>: Parameters for an individual data source, e.g., Parameters needed by ADIOS for configuring an Engine.

using fides::Params = std::unordered_map<std::string, DataSourceParams>: Parameters for all data sources mapped to their source name. The key must match the name given for the data source in the JSON file.