Creating/Changing a Dashboard

Sign-in at http://app.truera.net, then make sure the MONITORING tab is open. If it isn't, click it.

Right off the bat, your list of Dashboards is displayed (if you're creating your first dashboard, the list will be empty).

click and hold to enlarge

Here's the information you'll need at hand, along with the setup decisions you'll need to make, when creating your new dashboard:

• Basic identifying information – name, description, who created it, default time behavior (default time window, time granularity of the dashboard), and how often to auto-refresh the dashboard.
• Model(s) to include – choose the models in selected projects to track, then select the baseline splits to use for drift calculations (typically the training split but can be a known good production time window).
• Panels to configure – select which metrics to display in the dashboard, organized as follows:
  — Model Output – output statistics, trends, drift (requires baseline data), and performance (requires labels)
  Model Input – volume trends, drift, and drift quality metrics
  — Custom data – for custom metrics (see Ingesting Custom Metrics) and/or segment tag performance see [guidance not yet available]

After initial dashboard creation, you can make changes/additions in accordance with the guidance presented below.

Navigating Your Dashboards

As pictured above, select a dashboard by clicking its row. You can also search for a particular dashboard by entering either a partial or full Dashboard Name. Click the icon in the last column on the right to edit or delete a respective dashboard. Change the number of dashboards to display per page at the bottom of the page. Go back to the previous page or move to the next page by clicking the or icon, respectively.

See Panel Organization for more on the arrangement of dashboard panels).

Click CREATE NEW DASHBOARD to get started. When you do, you'll see a split screen. The panel on the left lists your existing dashboards, if any. The panel on the right is where you configure your New Monitoring Dashboard.

Step 1. Enter Dashboard Information

As pictured below, you first need to identify the dashboard and its general purpose, model output type, refresh rate (update frequency), time granularity (period used to aggregate values together for display), and a default rolling window (time period over which performance is measured).

click and hold to enlarge

To complete this step, do the following:

1. Enter the Dashboard Name (required) — something intuitive or easily remembered is recommended, although whether you implement a complex or simple dashboard naming scheme — cryptic or straightforward — the optional Description should at least hint at its purpose to avoid confusion among the MLOps project collaborators and stakeholders that will eventually use it.
2. Enter a dashboard Description (optional).
3. Select a Model output type, either Classification or Regression.
4. Select a Dashboard Refresh Rate. Off = never refresh; otherwise, choose a supported interval between 5 minutes (5m) and 1 hour (1h).
5. Select a minimum Time Granularity for aggregating the values to display. Supported increments range from 15 minutes (15m) to 1 month (1mo).
6. Select a Default Dashboard Rolling Window, which creates "summary historical" visualizations of your data at fixed intervals in the past — a week (Last 1 week), a month (Last 1 month), 3 months previously (Last 3 months), the year prior (Last 1 year), or you can specify a Custom time window with its own start and end dates — for comparison with more recent model outputs.
7. Click CONTINUE to move on to the next step.

As you'll discover, the Admin (userid) field is automatically populated and cannot be changed. Once you save the new dashboard, changes to the Model output type are also disallowed. Otherwise, after completing the remaining configuration steps below, you can edit this or a different dashboard at any time. See Navigating Your Dashboards).

Step 2. Select Project Models

If you're using TruEra Monitoring as a standalone — i.e., not as part of a TruEra Diagnostics and Monitoring bundle — you must first ingest the models you want to include in the new dashboard. On the other hand, if you're using TruEra Diagnostics bundled with TruEra Monitoring, you can select from models already in the system for diagnostic evaluation, as well as those ingested specifically for production monitoring.

click and hold to enlarge

To add a model to the new dashboard, do the following:

1. Select an existing TruEra Project name (required).
2. Select a Model name (required) from the list of models currently belonging to the project you specified in (a).
3. Select a Production Datastore (required).
   For purposes of this quickstart, the datastore is the sample data collection you ingested earlier. Otherwise, it's production data ingested by batch or scheduled pull.
4. Select a baseline split — one of the train/test splits or a time range/window of the production data (prod) you specfied in (c) above.
5. Click ADD TO DASHBOARD.

Next, add another model by clicking ADD MODEL or click CONTINUE to advance to Step 3.

Step 3. Select Dashboard Panels

You're now ready to choose the dashboard panels you want to see to visually monitor the performance and data quality of models you selected in the previous step.

click and hold to enlarge

To select a panel for inclusion in your dashboard, click its respective checkbox.

Click here for the list of available panels by model type and metric.

An example of what each standard panel looks like is available in Exploring the Dashboard Panels. Because custom panels are unique to your particular use case, be sure to follow the onscreen instructions with regard to the distinction between segment tag performance and custom-level metrics.

After you make your selections, click CONTINUE to advance to the next step.

Step 4. Confirm the Dashboard Configuration

The final step in creating a dashboard is to review a summary of your new dashboard configuration.

click and hold to enlarge

Click SAVE DASHBOARD to confirm the configuration summary. Click BACK to go to the previous step and make changes.

Once your dashboard is saved, you're ready to set up real-time monitoring alerts and notifications.

Click Next below to continue.

What is a production datastore?✕

The early days of machine learning distinguished between batch processing and streaming processing with a line that was rarely crossed. Generally, pipelines and architecture were built for one and could not interact with the other. Today it’s becoming more and more difficult to maintain the distinction between batch and streamed data.

Even so, knowing when your use case calls for batch or stream processing helps you plan projects, tooling, and storage, making it essential to understand the difference between the two.

Batch processing is asynchronous

This means data accumulates in a storage repository until a certain condition is met, then that data is processed through a pipeline and delivered to its endpoint. This type of data typically does not arrive in real time and it also does not need to be processed in real-time. Hence, pipelines process accumulated data in a batch.

A newer approach schedules jobs closer together in micro-batches, running every few minutes or seconds. This delivers more data results at a higher frequency than users could previously expect to access, which makes it especially popular in customer-facing applications.

As a rule, batch systems run at set times — every hour, half hour, every 10 minutes, and so forth. When you experience a delay in running a query in an app or a dashboard, it is typically because a batch process is still being run for your result.

See Batch Ingestion for more on ingesting a batch datastore.

Streaming processing is continuous

Generally speaking, streaming is capable of handling millions of events at scale, in real time.

In other words, data is processed as a constant flow as it is ingested into your data pipelines. Referred to as “always on” or real-time processing, streaming data is almost always highly perishable — moment-to-moment, like financial market conditions.

This means that only certain data is taken from the stream and processed by your model to serve predictions, reporting, or other outputs.

See SageMaker Scheduled Pull for more on ingesting a streaming datastore.

Panels by Metric and Model Type✕

The table below lists the available dashboard panels organized by model type and metric.

Panel Type	Regression Model Panels	Classification Model Panels
Model Output	Mean model score midpoint values for a time range Volume number of records generated over time Predictions and Lables reports model results against ground truth labels tracked over time Distribution tabulation of results by percentile (Min, 5th, 25th, Median, 75th, 95th, and Max) for a time range Model Drift aka concept drift; tracks decay of a model's predictive power as a result of the changes in real world environments : Difference of Means (X1, X2):=| 𝐄X1 – 𝐄X2 | for a time range Model Drift aka concept drift; tracks decay of a model's predictive power as a result of the changes in real world environments : Wasserstein distance between probability distributions on a given metric space M over a time range	Mean model score midpoint values for a time range Volume number of records generated over time Distribution tabulation of results by percentile (Min, 5th, 25th, Median, 75th, 95th, and Max) for a time range Model Drift aka concept drift; tracks decay of a model's predictive power as a result of the changes in real world environments : Difference of Mean (X1, X2):=| 𝐄X1 – 𝐄X2 | for a time range Model Drift aka concept drift; tracks decay of a model's predictive power as a result of the changes in real world environments : Wasserstein distance between probability distributions on a given metric space M over a time range Decisions and Labels By Class distribution of model decisions (post-decision threshold) for all models, and the distribution of labels for each model Class Distribution tabulaion of results by percentile (Min, 5th, 25th, Median, 75th, 95th, and Max) for a time range by class Model Score Distribution tabulation of model scores by percentile (Min, 5th, 25th, Median, 75th, 95th, and Max) for a time range
Labels (requires labels)	Label Volume reports the volume of ground truth labels assigned Label Distributions reports the percentage of records over time with ground truth labels assigned Label Drift calculates by the distance between the baseline (training split or known good production time window) and the selected time window using the specified distance metric	Label Volume reports the volume of ground truth labels assigned Label Class Distributions reports the percentage of records over time with ground truth labels assigned to the target class
Model Performance (requires labels)	RMSE Root Mean Square Error also known as root mean square deviation – commonly used regression metric for evaluating the quality of predictions; measures the Euclidean distance between the prediction and the ground truth. WMAPE Weighted Mean Absolute Percentage Error sometimes abbreviated wMAPE – a variant of MAPE, absolute percentage errors are weighted by volume; used to investigate the average error of model predictions over time.	AUC Area Under the Curve also known as C-statistics, ranges from 0 to 1; measures the two-dimensional area underneath the ROC curve.
Data Input	Input Volume volume of data inputs processed by the model over time Data Drift unexpected changes to data structure, semantics, and/or infrastructure : Difference of Means (X1, X2):=| 𝐄X1 – 𝐄X2 | for a time range Data Drift unexpected changes to data structure, semantics, and/or infrastructure : Wasserstein distance between probability distributions on a given metric space M over a time range Out of Range Values tracks results that are too far above or below a statistically reliable range based on the threshold defined
Data Quality	Unrecognized Categories data that violates explicit or inferred cardinality; cardinality refers to the number of possible values for a particular category Numerical Issues counts the instances in which numerical features exhibit NAN, NULL, Inf, or -Inf for the time range in question Schema Mismatch database constraint violations, such as a multi-valued attribute with duplicate values Missing Values errors tracked due to the absence of expected records during model processing of ingested data DQ Explorations tracks data quality errors per feature per model across the time range specified by the panel’s time selector
Custom panels you design by applying custom metrics from your datastore. You can also monitor selected data segments by applying tags for standard metrics	Segment tag performance Custom Level Metrics   – General any business metric not explicitly tied to one model   – Model any business metric tied to a given model   – Record a metric at the record level