API

User API

DataSetReader class

class DataSetReader

General purpose reader for data described by an Fides data model.

fides::io::DataSetReader reads data described by an Fides data model and creates Viskores datasets. See the Fides schema definition for the supported data model. DataSetReader also supports reading meta-data.

Public Types

enum class DataModelInput

Input types when setting up the DataSetReader

Values:

enumerator JSONFile

Brief input is path to a JSON file with the data model

enumerator JSONString

Brief input is JSON containing the data model stored in a string

enumerator BPFile

Brief input is a BP file that contains attributes that provide details for the predefined data model to be generated by Fides

Public Functions

DataSetReader(const std::string &dataModel, DataModelInput inputType = DataModelInput::JSONFile, const Params &params = Params(), bool createSharedPoints = false)

Constructor to set up the Fides reader

See also

DataModelInput

Parameters:

• dataModel – the value should be 1) a path to a JSON file describing the data model to be used by the reader, 2) a string containing valid JSON, 3) a path to a BP file containing attributes that Fides can use to generate a data model, or 4) a path to a BP file that contains a fides/schema attribute that contains the full JSON for the data model.

• inputType – specifies what is stored in the dataModel arg. Optional

• params – a map of ADIOS engine parameters to be used for each data source. Optional

void SetDataSourceParameters(const std::string &source, const DataSourceParams &params)

Sets the parameters for a given data source. Currently, only the inline engine requires this to be called, which must happen before attempting to read data.

Parameters:

• source – name of the DataSource, which should match a data_sources name given in the data model JSON.

• params – a map of parameters and their values

void SetDataSourceIO(const std::string &source, void *io)

Set the IO for a given source. This call should only be used when using the inline engine and must be called before attempting to read data or metadata.

Parameters:

• source – name of the DataSource, which should match a data_sources name given in the data model JSON.

• io – pointer to the ADIOS IO object

void SetDataSourceIO(const std::string &source, const std::string &io)

Set the IO for a given source. This call should only be used when using the inline engine and must be called before attempting to read data or metadata.

Parameters:

• source – name of the DataSource, which should match a data_sources name given in the data model JSON.

• io – the address to an ADIOS IO object, stored in a string

fides::metadata::MetaData ReadMetaData(const std::unordered_map<std::string, std::string> &paths = {}, const std::string &groupName = "")

Read and return meta-data. This includes information such as the number of blocks, available fields etc.

Parameters:

• paths – a map that provides the paths (filenames usually) corresponding to each data source.

• groupName – looks for metadata on variables inside the given group.

viskores::cont::PartitionedDataSet ReadDataSet(const std::unordered_map<std::string, std::string> &paths, const fides::metadata::MetaData &selections)

Read and return heavy-data.

Parameters:

• paths – a map that provides the paths (filenames usually) corresponding to each data source.

• selections – provides support for reading a subset of the data by providing choices for things such as time and blocks.

viskores::cont::PartitionedDataSet ReadDataSet(const fides::metadata::MetaData &selections)

Read and return heavy-data.

Parameters:

selections – provides support for reading a subset of the data by providing choices for things such as time and blocks.

StepStatus PrepareNextStep(const std::unordered_map<std::string, std::string> &paths = {})

When reading in streaming mode, this method has to be called before reading any meta-data or heavy data. It will also move the reader to the next step. Fides will loop on a data source while ADIOS reports that it is NotReady, but the user should also check the return which could return fides::StepStatus::OK or fides::StepStatus::EndOfStream. If EndOfStream, all steps have been read.

Parameters:

paths – a map that provides the paths (filenames usually) corresponding to each data source.

FIDES_DEPRECATED_SUPPRESS_BEGIN viskores::cont::PartitionedDataSet ReadStep (const std::unordered_map< std::string, std::string > &paths, const fides::metadata::MetaData &selections)

Same as ReadDataSet except that it works in streaming mode and needs to be preceeded by PrepareStep.

Parameters:

• paths – a map that provides the paths (filenames usually) corresponding to each data source.

• selections – provides support for reading a subset of the data by providing choices for things such as time and blocks.

FIDES_DEPRECATED_SUPPRESS_END FIDES_DEPRECATED_SUPPRESS_BEGIN std::shared_ptr< fides::datamodel::FieldDataManager > GetFieldData ()

Get a pointer to the field data manager

See also

FieldDataManager, FieldData

FIDES_DEPRECATED_SUPPRESS_END std::vector< std::string > GetDataSourceNames ()

Get std::vector of DataSource names.

std::set<std::string> GetGroupNames(const std::unordered_map<std::string, std::string> &paths)

Get all available group names.

Public Static Functions

static bool CheckForDataModelAttribute(const std::string &filename, const std::string &attrName = "Fides_Data_Model")

Checks a bp file for an attribute containing information that Fides can use to generate the data model. Static so that it doesn’t require setting up the DataSetReader first. Useful for applications like ParaView, where it wants to check if it can use Fides to read a file without needing to configure Fides first.

Parameters:

• filename – Name of file to check

• attrName – Name of attribute to look for

class DataSetReaderImpl

struct GetTimeValueFunctor

Keys and MetaData

KeyType fides::keys::NUMBER_OF_BLOCKS()

Key used for storing number of blocks meta-data. Uses fides::metadata::Size

KeyType fides::keys::NUMBER_OF_STEPS()

Key used for storing number of steps meta-data. Uses fides::metadata::Size

KeyType fides::keys::BLOCK_SELECTION()

Key used for selecting a set of blocks. Uses fides::metadata::Vector<size_t>

KeyType fides::keys::FIELDS()

Key used for available array meta-data and array selection. Uses fides::metadata::Vector<fides::metadata::FieldInformation>

KeyType fides::keys::STEP_SELECTION()

Key used for selecting time step. Uses fides::metadata::Index

KeyType fides::keys::PLANE_SELECTION()

Key used for selecting planes for XGC data. Should only be used internally. Uses fides::metadata::Set

struct Size : public fides::metadata::MetaDataItem

Meta-data item to store size of things such as number of blocks.

Public Functions

inline Size(size_t nItems)

constructor

Public Members

size_t NumberOfItems

Number of items (e.g., blocks)

struct Index : public fides::metadata::MetaDataItem

Meta-data item to store an index to a container.

Public Functions

inline Index(size_t idx)

Public Members

size_t Data

struct FieldInformation

Simple struct representing field information.

Public Functions

inline FieldInformation(std::string name, viskores::cont::Field::Association assoc)

inline FIDES_DEPRECATED_SUPPRESS_BEGIN FIDES_DEPRECATED (1.1, "fides::Association is no longer used. Use viskores::cont::Field::Association directly.") FieldInformation(std

Public Members

FIDES_DEPRECATED_SUPPRESS_END std::string Name

Name of the field.

viskores::cont::Field::Association Association

Association of the field. See Viskores field association for details.

template<typename T>
struct Vector : public fides::metadata::MetaDataItem

Meta-data item to store a vector.

Public Functions

inline Vector(std::vector<T> vec)

Vector() = default

Public Members

std::vector<T> Data

template<typename T>
struct Set : public fides::metadata::MetaDataItem

Meta-data item to store a set.

Public Functions

inline Set(std::set<T> data)

Set() = default

Public Members

std::set<T> Data

class MetaData

Container of meta-data items. This class is a simple wrapper around an std::map that makes setting/getting a bit easier. Internally, it stores objects using unique_ptrs but the interface uses stack objects.

Public Functions

template<typename T>
inline void Set(fides::keys::KeyType key, const T &item)

Add a meta-data item to the map. Supports subclasses of MetaDataItem only.

template<typename T>
inline const T &Get(fides::keys::KeyType key) const

Given a type, returns an object if it exists. Raises an exception if the item does not exist or if the provided template argument is incorrect.

inline void Remove(fides::keys::KeyType key)

Given a key, removes the item from the map.

inline bool Has(fides::keys::KeyType key) const

Given a key, checks whether an item exists.

Useful enums and typedefs

enum class fides::StepStatus

Possible return values when using Fides in a streaming mode.

Values:

enumerator OK

enumerator NotReady

enumerator EndOfStream

using fides::DataSourceParams = std::unordered_map<std::string, std::string>

Parameters for an individual data source, e.g., Parameters needed by ADIOS for configuring an Engine.

using fides::Params = std::unordered_map<std::string, DataSourceParams>

Parameters for all data sources mapped to their source name. The key must match the name given for the data source in the JSON file.