Skip to content

Reading Data from DataFrames

First things first

The TruEra client appropriate to your environment must be installed before proceeding with data ingestion. If you have not yet installed the client, see Installation and Access for guidance.

The following discussion covers various ways in which Pandas DataFrames can be ingested into the TruEra ecosystem. A working knowledge of project, data collection, and data split structures is recommended (see Project Structure for a quick review). Knowledge of basic ingestion commands and concepts covered in Diagnostics Quickstart will also be helpful to you.

Important Classes

Depending on your use case, ingesting data locally from a
Pandas DataFrame

Pandas DataFrame

A two-dimensional data structure in tabular format organizing the data into rows and columns.
hinges on one or more of three important SDK
classes

Python Class

A means of bundling data and functionality together; each class creates a new type of object, allowing new instances of that type to be made. Each class instance can have attributes attached to it for maintaining its state. Class instances can also have methods (defined by its class) for modifying its state.
:

Click a class above to see its parameters.

Ingesting Feature Data, Extra Data, and Labels

Let's assume that we've already created a project and specified a data collection. You can now set them for the current workspace.

tru.set_project(project_name)
tru.set_data_collection(data_collection_name)

Next, create a basic DataFrame with an id column, feature data, extra data, and labels using the following code strictly as an example.

import pandas as pd

data = pd.DataFrame({
    "id": ["1", "2", "3"],
    "feature1": ["a", "a" ,"b"],
    "feature2": [0.5, 0.7, 0.8],
    "extra": [True, True, False],
    "label": [1, 1, 0],
})

Ingest this using the tru.add_data method, which creates a data split based on the data_split_name and data you specify. Specify columns with the ColumnSpec class.

from truera.client.ingestion import ColumnSpec

tru.add_data(
    data=data,
    data_split_name="split_1",
    column_spec=ColumnSpec(
        id_col_name="id",
        pre_data_col_names=["feature1", "feature2"],
        extra_data_col_names=["extra"],
        label_col_names="label"
    )
)

Alternatively, use a Python dictionary to specify your columns in lieu of the ColumnSpec class.

tru.add_data(
    data=data,
    data_split_name="split_1",
    column_spec={
        "id_col_name":"id",
        "pre_data_col_names":["feature1", "feature2"],
        "extra_data_col_names":["extra"],
        "label_col_names":"label"
    }
)

Verify successful ingestion with the following code:

tru.get_xs(extra_data=True)
tru.get_ys()

Ingesting Model Predictions

When ingesting model predictions, a model needs to be associated with the data. Assuming you've already created one, set the model for your workspace by replacing model_1 with the name of your model.

tru.set_model("model_1")

Next, define the predictions and ingest them using the SDK's add_data() method.

data_pred = pd.DataFrame({
    "id": ["1", "2", "3"],
    "prediction": [1, 0, 0]
})

tru.add_data(
    data=data_pred,
    data_split_name="split_1",
    column_spec=ColumnSpec(
        id_col_name="id",
        prediction_col_names="prediction"
    )
)

Since we have both the project and model defined in the workspace, the workspace automatically infers the model and score type. In cases where an override of these parameters is desired, specify this with the ModelOutputContext class.

from truera.client.ingestion import ModelOutputContext

tru.add_data(
    data=data_pred,
    data_split_name="split_1",
    column_spec=ColumnSpec(
        id_col_name="id",
        prediction_col_names="prediction"
    ),
    model_output_context=ModelOutputContext(
        model_name="model_2",
        score_type="probits"
    )
)

See the Python SDK Technical Reference for additional details.

Ingesting Feature Influences

The method for ingesting feature influences is similar to ingesting model predictions. Although you could have TruEra compute feature influences, for this example, assume you've already computed influences, which you now wish to ingest. First, define the influences, replacing the example values below with your values, then ingest them using the add_data() method.

data_fi = pd.DataFrame({
    "id": ["1", "2", "3"],
    "feature1": [0.7, 0.4, 0.5],
    "feature2": [0.3, 0.6, 0.5],
})

tru.add_data(
    data=data_fi,
    data_split_name="split_1",
    column_spec=ColumnSpec(
        id_col_name="id",
        feature_influence_col_names=["feature1", "feature2"]
    )
)

In this case, the workspace automatically inferred information based on its context. In cases for which you want to override these parameters, specify them with ModelOutputContext class.

tru.add_data(
    data=data_fi,
    data_split_name="split_1",
    column_spec=ColumnSpec(
        id_col_name="id",
        feature_influence_col_names=["feature1", "feature2"]
    )
    model_output_context=ModelOutputContext(
        model_name="model_1",
        score_type="regression",
        background_split_name="split_1",
        influence_type="truera-qii"
    )
)

Again, refer to the Python SDK Technical Reference for additional details.

Alternatively, we can specify both ColumnSpec and ModelOutputContext as Python dictionaries.

tru.add_data(
    data=data_fi,
    data_split_name="split_1",
    column_spec={
        "id_col_name": "id",
        "feature_influence_col_names": ["feature1", "feature2"]
    },
    model_output_context={
        "model_name": "model_1",
        "score_type": "regression",
        "background_split_name": "split_1",
        "influence_type": "truera-qii"
    }
)

If you choose to ingest both feature data and feature influences together in one DataFrame, you can add suffixes to the feature influence columns to avoid duplicate name issues.

Here, make the suffix either _truera-qii_influence or _shap_influence as shown in the following example.

data_all = pd.DataFrame({
    "id": ["1", "2", "3"],

    "feature1": ["a", "a" ,"b"],
    "feature2": [0.5, 0.7, 0.8],

    # Feature influence columns have a suffix
    "feature1_truera-qii_influence": [0.7, 0.4, 0.5],
    "feature2_truera-qii_influence": [0.3, 0.6, 0.5]
})

tru.add_data(
    data=data_all,
    data_split_name="split_1",
    column_spec=ColumnSpec(
        id_col_name="id",
        pre_data_col_names=[
            "feature1", 
            "feature2"
        ],
        feature_influence_col_names=[
            "feature1_truera-qii_influence", 
            "feature1_truera-qii_influence"
        ]
    )
)

Ingesting Production Data for Monitoring

Ingesting production data for monitoring is almost identical to the previous examples. with the only differences being that the add_production_data() method is used instead and that a timestamp column must be specified.

production_data = pd.DataFrame({
    ...
    "timestamp": pd.Timestamp.now(),    # Required for monitoring
    ...
})

tru.add_production_data(
    data=production_data,
    column_spec=ColumnSpec(
        ...
        timestamp_col_name:"timestamp", # Required for monitoring
        ...
    ),
)

Click Next below to continue.