Production Monitoring

An LLM application once developed and tested is deployed to production. TruEra's Monitoring Dashboards can be configured to monitor your LLM application over time at scale. To monitor your llm applications, you need to ingest data in production config. Read more about production data ingestion.

Setup Monitoring Dashboard

From your TruEra projects page, switch over to Monitoring tab and click on CREATE NEW DASHBOARD and choose LLM Application

Fill the required details such as,

1. Name
2. Default Dashboard Rolling Window
3. Minimum Time Granularity for aggregating metrics

and other optional fields

click and hold to enlarge

Add LLM Application

Click on + LLM APPLICATION and in the popup pick the project and application(s) you wish to monitor in this dashboard

click and hold to enlarge

Choose Panels

Click on + NEW PANEL and in the popup,

1. Give a name to the panel
2. Choose the Metric that this panel should track. This dropdown is populated by all the feedback evaluations ingested for this application along with other meta-data such as cost and latency
3. Choose the aggregation (count, min, max, sum, average) that you wish to perform on the time granularity chosen for the dashboard

click and hold to enlarge

Now you can save the Dashboard and you can open it from your Dashboards page.

RCA Snapshots

If you notice an issue on the dashboard while monitoring your application, you can create a snapshot of the data which can then be investigated like the experimental data by looking closely at the individual traces in a given time period.

To create an RCA snapshot,

1. Click on Open in Diagnostics button on any of the panels
2. In the pop up, pick the application you wish to do an RCA on
3. Select the time range and click SET TIME RANGE
4. Give you snapshot a name and Description and click on SAVE AND OPEN IN DIAGNOSTICS
5. You will be re-directed to the Production section of the project in Diagnostics

click and hold to enlarge

Click Next below to continue.