*************
DataSetReader
*************

``fides::io::DataSetReader`` is the main class to be familiar with when using Fides.
There are three main phases for this class, :ref:`dataset-reader-init`, :ref:`dataset-reader-read-metadata`, and :ref:`dataset-reader-read-data`.

.. _dataset-reader-init:

Initialization
==============

The ``DataSetReader`` is set up by passing the data model to the ``dataModel`` argument in the constructor and passing the correct ``DataModelInput`` type to the ``inputType`` argument in the following ways:

**Option 1: Passing a path to JSON file containing the data model**

This is the default way to use Fides.
A JSON file containing the data model description has been created and the path is passed to the ``DataSetReader``.

.. code-block:: c++

    std::string jsonPath = "/path/to/data/model/json";
    // Default, so second argument is not actually needed
    fides::io::DataSetReader reader(jsonPath,
      fides::io::DataSetReader::DataModelInput::JSONFile);


**Option 2: Passing a string containing valid JSON describing the data model**

In this case, the string passed to the ``DataSetReader`` contains the JSON data model.

.. code-block:: c++

    std::string jsonString = "<string containing valid JSON>";
    fides::io::DataSetReader reader(jsonString,
      fides::io::DataSetReader::DataModelInput::JSONString);


**Option 3: Passing a path to a BP file**

It is also possible to get Fides to automatically generate a data model (see Predefined Data Models).
In this case, the string passed to the ``DataSetReader`` constructor is a path to a BP file that contains the attribute information that Fides will use to generate the data model.

.. code-block:: c++

    std::string bpFilePath = "/path/to/bp/file";
    fides::io::DataSetReader reader(bpFilePath,
      fides::io::DataSetReader::DataModelInput::BPFile);


**Optional Step: Setting ADIOS data source parameters**

If you need to pass any engine parameters to the ADIOS engine being used, you should use ``reader.SetDataSourceParameters()``.
For instance, if you want to the ADIOS SST engine instead of BP files (which Fides uses by default), you could do the following:

.. code-block:: c++

    // after creating the DataSetReader object
    fides::DataSourceParams params; // std::unordered_map<std::string, std::string>
    params["engine_type"] = "SST";
    reader.SetDataSourceParameters(source_name, params);

``source_name`` is the name of the source provided in the data model.
If you have multiple sources, and want to add options for each data source, you'll need to call this for each data source that you have.

Another option is to set up the parameters for all sources before the constructor and pass it as the third argument.

.. code-block:: c++

    fides::Params params;
    fides::DataSourceParams src1Params;
    fides::DataSourceParams src2Params;
    src1Params["OpenTimeoutSecs"] = "10";
    params["source1"] = src1Params;
    src2Params["OpenTimeoutSecs"] = "20";
    params["source2"] = src2Params;
    fides::io::DataSetReader reader(jsonPath,
      fides::io::DataSetReader::DataModelInput::JSONFile, params);


**Optional Step: Setting data source IO object (for inline engine only)**

The last possible step for initialization of the ``DataSetReader`` is to set the data source's IO object using ``reader.SetDataSourceIO()``.
This is only necessary when data sources are using ADIOS' inline engine because the reader and writer need to share the ``IO`` object.


.. code-block:: c++

    std::string sourceName = "source";

    // setting up ADIOS IO
    adios2::ADIOS adios;
    adios2::IO io = adios.DeclareIO("inlineIO");

    // define ADIOS variables
    ...
    // setting up inline writer
    adios2::Engine writer = io.Open("output.bp", adios2::Mode::Write);

    // Setting up Fides for reading
    fides::io::DataSetReader reader(jsonPath);
    fides::DataSourceParams params;
    params["engine_type"] = "Inline";
    reader.SetDataSourceParameters(sourceName, params);
    reader.SetDataSourceIO(sourceName, &io);


.. _dataset-reader-read-metadata:

Reading Metadata
================

The next step is to read the metadata, which will give you info such as number of steps, blocks, and available fields.
In order to read metadata/data, you'll need to first set up the paths for the data source(s), then the metadata can be read as follows:

.. code-block:: c++

    std::unordered_map<std::string, std::string> paths;
    paths["source"] = filePath;
    auto metadata = reader.ReadMetaData(paths);


From ``metadata`` you can get the following info:

.. code-block:: c++

    // number of blocks
    auto& nBlocks = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_BLOCKS());
    std::cout << "Number of blocks " << nBlocks.NumberOfItems << std::endl;

    // number of steps
    auto& nSteps = metadata.Get<fides::metadata::Size>(fides::keys::NUMBER_OF_STEPS());
    std::cout << "Number of steps " << nSteps.NumberOfItems << std::endl;

    // field information
    auto& fields = metadata.Get<fides::metadata::Vector<fides::metadata::FieldInformation>(
      fides::keys::FIELDS());
    for (auto& field : fields.Data) // fields.Data is a std::vector
    {
      std::cout << field.Name << " has association " << field.Association << std::endl;
    }


.. _dataset-reader-read-data:

Reading Data
============

Now we can read the actual data, in either random access mode or go step by step.
In either case, we need the ``paths`` for the data source(s) we set up earlier before reading the metadata, as well as creating a new ``fides::metadata::MetaData`` object that we can use to give Fides some info on the selections we'd like to make.

**Random access of data steps**

In this case, we want to choose a step for Fides to read, so we need to set up this information:

.. code-block:: c++

    fides::metadata::MetaData selections;
    fides::metadata::Index step(2); // we want to read step 2
    selections.Set(fides::keys::STEP_SELECTION(), step);


Now we can read the dataset:

.. code-block:: c++

    vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);


Now you've got your data in VTK-m format, so you can use the VTK-m API to access the partitions, use filters, etc.

**Step streaming**

In this case we don't need to add a step selection to ``selections``.
Before we can read the step though, we'll need to call ``reader.PrepareNextStep()`` and check the return value.
``PrepareNextStep()`` will return either ``OK`` or ``EndOfStream``.
If any data source is not ready, Fides will internally loop on that data source until it returns ``OK`` or ``EndOfStream``.
In the case of multiple data sources, if some source hits ``EndOfStream`` before the others (e.g., mesh is stored in a different data source from the variable data and has only one step, while the variables have multiple steps), Fides caches the data from that data source and doesn't attempt to read steps that do not exist.
When ``PrepareNextStep()`` returns ``EndOfStream`` that means all data sources have finished streaming.

.. code-block:: c++

    while (true)
    {
      fides::StepStatus status = reader.PrepareNextStep(paths);
      if (status == fides::StepStatus::EndOfStream)
      {
        // all
        break;
      }
      // PrepareNextStep only returns EndOfStream or OK
      vtkm::cont::PartitionedDataSet output = reader.ReadDataSet(paths, selections);
      // perform what ever vis/analysis tasks you want on this step
    }


**Other possible selections**

For either reading method, you can provide Fides with some additional selections instead of reading all data.
You can choose specific blocks to read (recall from the section on reading metadata that you can find out the total number of blocks available to be read).
If no block selections are provided, then Fides will read all blocks by default.

.. code-block:: c++

    fides::metadata::Vector<size_t> blockSelection;
    blockSelection.Data.push_back(1);
    selections.Set(fides::keys::BLOCK_SELECTION(), blockSelection);

You can also choose to select specific fields for reading.
If no field selection is made, then Fides will read all fields by default.

.. code-block:: c++

    fides::metadata::Vector<fides::metadata::FieldInformation> fieldSelection;
    fieldSelection.Data.push_back(
      fides::metadata::FieldInformation("dpot", vtkm::cont::Field::Association::Points));
    selections.Set(fides::keys::FIELDS(), fieldSelection);