Skip to content

Monitoring Quickstart

In this quickstart, you'll create a monitoring dashboard that can be accessed by a URL.

Before you begin – Important Information

Interested in TruEra Monitoring? Request a demo and early access at https://go.truera.com/get-demo.

Restrictions

The dataset used here is hosted under the Creative Commons 4.0 license by UCI. It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the Creative Commons 4.0 License for more details:

System Requirements

Although TruEra's Python SDK can interact with web services via the Python environment of your choice, before continuing with installation, please review the system requirements for installing the TruEra package here.

To get started quickly with TruEra Monitoring, make sure the Python SDK is installed (see Installation and Access), then complete each of the following steps:

  1. Connect to TruEra and create your workspace.
  2. Download and process the sample data provided.
  3. Set up a TruEra Diagnostics project for subsequent model analysis, bearing in mind that a monitoring dashboard can track models from more than one Diagnostics project.
  4. Add a data collection and model to the project.
  5. Ingest the simulated prod data into the production datastore.
  6. Ingest custom metrics (optional).
  7. Create a dashboard.

Guidance follows below for each step in the sequence outlined above.

Step 1. Connect and Create Your Workspace

To access TruEra Monitoring, you'll need a TruEra account. Sign-up/sign-in at app.truera.net.

Next, generate your authentication, connect via SDK, and create your workspace. Here's how:

  1. Open your avatar in the Web App toolbar (pictured below in the top-left).
  2. Click Authentication, then click GENERATE CREDENTIALS.

    generate auth token credentials example
    click and hold to enlarge

  3. Click the clipboard icon to copy the token.

  4. Connect and authenticate using the Python SDK by replacing your-auth-token in the code snippet below with the copied token.
TRUERA_URL = "https://app.truera.net"
AUTH_TOKEN = "your-auth-token"

from truera.client.truera_workspace import TrueraWorkspace
from truera.client.truera_authentication import TokenAuthentication

auth = TokenAuthentication(AUTH_TOKEN)
tru = TrueraWorkspace(TRUERA_URL, auth, ignore_version_mismatch=True)

You're now ready to download the sample data provided.

Two sample projects are provided by TruEra for download — Housing Price Forecast and Diabetes Readmission.

Step 2. Download and Process Sample Data

Throughout this quickstart, you'll work with the Diabetes Readmission project. It includes a formatted version of the data and a pickled scikit-learn Python classification model to illustrate the model ingestion process. For other frameworks, the process is similar.

Regression Models

See Monitoring Quickstart for Regression Models in the Jupyter Notebook Library for guidance on getting started quickly with a sample regression model and data.

Add the sample model and dataset to your remote workspace as follows:

import pandas as pd
import numpy as np
from itertools import combinations

# common parameters & constants
_ID_COLUMN_NAME = 'id'
_TIMESTAMP_COLUMN_NAME = 'timestamp'
_LABEL_COLUMN_NAME = 'readmission'
_PREDICTION_COLUMN_NAME = 'prediction'
_PREDICTION_THRESHOLD = 0.6

# prod data
prod_data = pd.read_parquet("https://quickstart-sample-data.s3.us-west-2.amazonaws.com/diabetes_readmission_v2/prod.parquet")

# If __id__ in the data, remove this. 
prod_data.rename(columns={'__id__': _ID_COLUMN_NAME}, inplace=True)

Next, create a new TruEra project.

Step 3. Create a TruEra project

Add the following code to create your first project.

# adding project
project_name = "PROJECT_NAME_X" # Replace with a project name of your choice
if project_name not in tru.get_projects():
    tru.add_project(project=project_name, score_type="probits")
else:
    tru.set_project(project_name)

Step 4. Add a Data Collection and Model

Give the model a name and associate it with a data collection.

MODEL_NAME = "lr"

data_collection_name = "{}_dataset".format(MODEL_NAME)

# Add data collection.
if data_collection_name not in tru.get_data_collections():
    tru.add_data_collection(data_collection_name=data_collection_name)
else:
    tru.set_data_collection(data_collection_name)

# Add models.
if MODEL_NAME not in tru.get_models():
    tru.add_model(MODEL_NAME)
else:
    tru.set_model(MODEL_NAME)

The model and data added above are now available for production monitoring as soon as you create a dashboard.

Step 5. Add Production Data

Adding production data (typically from a production datastore) requires setting a prediction threshold, creating a schema, and adding a score type. For example:

from typing import Sequence
from truera.client.ingestion.util import ColumnSpec, ModelOutputContext


def columns_excluding(df: pd.DataFrame, exclude_columns: Sequence[str]):
    column_names = df.columns.tolist()
    for column in exclude_columns:
        if column in column_names:
            column_names.remove(column)
    return column_names

def add_prediction_classification_column(data: pd.DataFrame, threshold):
    data[_PREDICTION_CLASSIFICATION_COLUMN_NAME] = ["1" if score > threshold else "0" for score in data[_PREDICTION_COLUMN_NAME] ]

pre_column_names = columns_excluding(
        prod_data, [
            _ID_COLUMN_NAME, _TIMESTAMP_COLUMN_NAME,
            _PREDICTION_COLUMN_NAME, _LABEL_COLUMN_NAME
        ]
    )

#prod data Threshold column
add_prediction_classification_column(prod_data, _PREDICTION_THRESHOLD)

# Create Schema 
tru.add_production_data(
    prod_data,
    column_spec=ColumnSpec(
        id_col_name=_ID_COLUMN_NAME,
        pre_data_col_names=pre_column_names,
        label_col_names=[_LABEL_COLUMN_NAME],
        prediction_col_names=[_PREDICTION_COLUMN_NAME],
        timestamp_col_name=_TIMESTAMP_COLUMN_NAME
    ),
    model_output_context=ModelOutputContext(MODEL_NAME,score_type="probits")
)

# adding classification_score
tru.add_production_data(
    prod_data,
    column_spec=ColumnSpec(
        id_col_name=_ID_COLUMN_NAME,
        prediction_col_names=[_PREDICTION_CLASSIFICATION_COLUMN_NAME],
        timestamp_col_name=_TIMESTAMP_COLUMN_NAME
    ),
    model_output_context=ModelOutputContext(MODEL_NAME,score_type="classification")
)

Next, you have the option of ingesting custom metrics.

Step 6. Ingest Custom Metrics (optional)

Entirely optional, see Ingesting Custom Metrics for a detailed discussion and guidance on adding your own custom metrics. For purposes of this quickstart, here's a simplified template:

import datetime, time
from truera.client.metrics import getModelMetricReporter
from truera.client.metrics import getGeneralMetricReporter

# Reset the timestamp for each model to have same start time
start_time = pd.to_datetime(prod_data[_TIMESTAMP_COLUMN_NAME]).min().to_pydatetime()
end_time = pd.to_datetime(prod_data[_TIMESTAMP_COLUMN_NAME]).max().to_pydatetime()

# Add models.
if MODEL_NAME not in tru.get_models():
    tru.add_model(MODEL_NAME)
else:
    tru.set_model(MODEL_NAME)

general_metric_reporter = getGeneralMetricReporter(tru)
model_metric_reporter = getModelMetricReporter(tru)

while (start_time <= end_time):
    model_metrics = {"model_metric_1": np.random.random(), "model_metric_2": np.random.random()}
    general_metrics = {"general_metric_1": np.random.random(), "general_metric_2": np.random.random()}
    model_metric_reporter.sendMetrics(metrics=model_metrics, time=start_time)
    general_metric_reporter.sendMetrics(metrics=general_metrics, time= start_time)
    time.sleep(0.4)
    start_time = start_time + datetime.timedelta(days = 1)

You're now ready to create a dashboard to monitor your newly added sample model and data.

Click Next below to continue.