Features¶
Data scientists like to organize things into tables, where a row is an instance/record/ observation/trial representing a single datapoint on a graph, i.e., the person or thing (a single unit thereof) being measured. Each column is an applicable feature or variable (a characteristic) of the row entry — name, age, weight, education, temperature, viscosity, voltage, salinity ... you name it.
But what does it mean?
It means that features constitute what your dataset is comparing, row by row.
It also means that the quality of the features in your dataset impacts the efficacy of the insights gained when the dataset is used for machine learning. Business problems within the same industry do not necessarily require the same features. Everything that can be measured may not be necessary to answer the business problem posed, making it important to have a strong understanding of the business goals and priorities of your modeling project before merely including all available features, i.e., all measurable attributes. Certain attributes simply are not relevant to the analysis being undertaken. In other words, it's often a fine line that separates "too much" from "not enough."
Hence, it's important to weigh the quality of your dataset’s features and their pertinence/relevance to the business problem at hand. Feature selection and feature engineering, notoriously difficult and tedious, are therefore vital to the verity of your model's results. Done well, your continually optimized dataset should contain only those features having a bearing on the problem you're trying to solve. Including more could impose an undue influence on the model. Including less could skew the results in an entirely different way.
TruEra's Features diagnostics can help you determine the "Goldilocks Zone" for your model — not too much, not too little — just right.
Assessing Feature Behavior¶
To begin, click Features under Diagnostics in the nav panel on the left. This displays a page headed by three tabs: Overview, All features, and Feature Groups.
As discussed next, the Overview tab opens by default, showing feature information for the last Model and Split name selected.
Overview¶
- Importance – shows the top-five features ranked by highest average influence measured as L1 norm, calculated as the sum of the absolute vector values or the Manhattan distance from the origin of the vector space.The L1 norm is calculated as the sum of the absolute values of the vector. The L2 norm is calculated as the square root of the sum of the squared vector values.
- Outliers – presents the top five outlier features in terms of:
- Influence – highest influence of a single point, a metric which helps identify those features which, for certain points, drive extreme model decisions.
- Category – calculated as the normalized ratio of the highest and second highest absolute influence categories, these are features for which a single category stands out from all other categories in terms of influence.
- Trend-line – calculated by finding the average residual for the top 10 percentile of points as measured by distance from the trend line and comparing this to the average residual for the rest of the points in order to surface those points that have irregular interactions in that they diverge from a typical trend; i.e., these features have points that lie far from the best-fit spline.
- Trends – ranks the top five features with the highest R-squared value of the best-fit spline (influence variances well-explained by the spline).
See Influence Sensitivity Plots for additional guidance on outliers and trends.
Insight gained at this level of analysis involves the influence of input variables on outputs, measuring the model's behavior in terms of input value change, noise tolerance, data quality, internal structure, and more, as a means of uncovering abnormal or unexpected behavior.
To reveal additional feature detail, click a plot in the overview. This switches the view to the All features tab.
Pictured above, the left-side panel under the All features tab contains the current list of features for the selected Model and Split name. (Remember, you can change the model and/or split at any time.) The feature you clicked in Overview is highlighted.
Influence Sensitivity Plots¶
Here, the right-side panel shows greater detail for the selected feature in the form of an Influence Sensitivity Plot (ISP). ISPs show the relationship between a feature’s value and its contribution to the model output, or its influence — a point-level visualization that allows you to examine any feature in detail. Added in composite overlay to contextualize the ISP by showing particularly sparse or dense regions is a distribution of the feature values (shown at the top) and the distribution of influences (shown on the right).
When enabled within an ISP, you can also examine (click a link for additional detail):
- Best-fit spline – shows a polynomial trend line that is fit to the ISP to tease out a clear relationship between a feature's value and its contribution to the model score.
- Overfitting records – identifies influential points of the ISP occurring in low-density regions. Points that drive model scores despite occurring in low-density regions can indicate overfitting.
- Error rate – shows the model's error for bucketized ranges of the feature's value, which can help contextualize the ISP by sanity checking regions against the error rate graph. Error is measured using either mean absolute error or mean squared error, depending on your project settings.
- Ground truth rate – shows the ground truth rate of the given data split for bucketized ranges of the feature's value. This helps contextualize the ISP with a sanity check of feature behavior for regions associated with ground truth labels.
How is overfitting calculated?
The overfitting diagnostic identifies influential points which occur in low-density or sparse regions of data. Data points with feature values in regions with less than 3% of the data population and with influences at or above the 95th percentile are classified as overfitting. This indicates that the feature value of the point in question is driving a model's prediction, despite being a low-density region of data. If the model overfits on these low-density regions of data, it can fail to generalize, leading to poor test and production performance.
Click SHOW ALL FEATURES to return to the ISPs for all features. You can then sort by submetrics for the primary measurements reported in the Overview tab.
Curve fitting constructs a curve of the best fit to a series of data points. Interpolation is used to estimate the values of unknown data points that fall in between known (ingested) data points.
A spline is a piecewise polynomial parametric curve that minimizes a weighted combination of the average squared approximation error over observed data and the roughness measure. The term comes from the flexible (analog) spline devices used by shipbuilders and draftsmen to draw smooth shapes.
Essentially, spline interpolation fits low-degree polynomials to small subsets of points instead of fitting a single large-degree polynomial to all of them.
It is a generally preferred interpolation method because the interpolation error can be made small.
Overfitting occurs when the model fits exactly against its training data and cannot perform accurately against unseen data, defeating the purpose of machine learning.
More specifically, when the model trains for too long on sample data or when the model is too complex, it can start to learn “noise” (information that isn't relevant) within the dataset. When it memorizes the noise, fitting too closely to the training set, the model becomes “overfitted” — unable to generalize well to new data — making it unable to perform the classification or prediction tasks for which it is intended.
Generally speaking, low error rates and a high variance are good indicators of overfitting. To prevent this type of behavior, you should set up a test data split to check for overfitting. A training split with a low error rate and a test split with a high error rate signals overfitting.
How is the overfitting diagnostic calculated?
Data points with feature values in regions with less than 3% of the data population and with influences at or above the 95th percentile are classified as overfitting.
This indicates that the feature value of the point in question is driving the model's predictions, despite being in a low-density region of data. When the model overfits on these low-density regions of data, it is failing to generalize.
Model accuracy is the number of correctly predicted points out of all the data points modelled, whereas the error rate is a measure of how often your model classifies data incorrectly; in other words, the number of times it is wrong. More formally, model accuracy is defined as the number of true positives and true negative divided by the total number of true positives, true negatives, false positives and false negatives.
Error rate is calculated as the total number of two incorrect predictions (FN + FP) divided by the total number of data points (TP + TN + FP + FN).
Measuring model performance by quantifying the difference between predicted probabilities and actual values, Log loss (logarithmic loss or cross-entropy loss) is a common evaluation metric for binary classification models.
If the error rate of your model is very high — meaning the accuracy of its predictions are very low — then there is likely to be a lot you missed when fitting the model to the dataset you've chosen. Called "underfitting," this often occurs when your model:
- didn’t find any trend in the dataset
- is fitted to the wrong data; i.e., fitting a linear model to nonlinear data.
Otherwise, you can reduce underfitting by increasing the number of data points and reducing the number of superfluous (unecessary/redundant) features in the training split used.
At the other end of the error spectrum is "overfitting," which can happen when there are too many explanatory features selected.
Prevent overfitting by:
- Pausing training before your model starts learning noise — i.e., features unnecessary to the model's predictive purpose.
- Expanding the training split to include more data.
- Selecting features to include in the training data judiciously, identifying the most important ones, eliminating those that are irrelevant or redundant.
- Regularizing your model by applying L1 to feature selection (Lasso regression; shrinking coefficients to zero) or L2 (Ridge regression; shrinking coefficients equally) if features are collinear/codependent.
- Incorporating ensemble methods with a set of classifiers — e.g., decision trees that employ bagging and boosting.
Last but not least, check out this blog about the optimal error rate in model training.
Applicable to supervised algorithms, ground truth is the target for training and/or validating your model with a labeled (annotated) dataset — results already determined to be accurate; i.e., the "reality."
However, only model stakeholders can define the objective for the ground truth algorithm, which is almost always subjective, and on which decision-makers can disagree. Frankly, there are no hard-and-fast rules for defining ground truth labels. Nevertheless, the more high-quality labelled data you make available for training, the better your model will ultimately perform.
You should initially train your model on data with ground truth labels, evaluating it on test datasets for which the model doesn't know the ground truth label, then compare the model's predictions to the actual ground truth label to determine model performance.
The ground truth rate plots the difference (loss/error) between the model's prediction and the ground truth, where ground-truth refers to the "officially correct" label (categorical or numerical) for a given input.
Partial Dependence Plots (PDPs) show you how a given feature affects the model's predictions, helping to extract valuable insights with respect to explainability you can relay to model stakeholders. For TruEra to display a feature's partial dependence plot, however, you must first ingest PDPs for the features you're most interested in analyzing for partial dependency.
To make a partial dependence plot with sklearn, for example, see the scikit-learn guidance, then use the Python SDK's get_partial_dependencies() method (see the Python SDK Technical Reference to ingest the PDPs.
Essentially, PDPs plot the average affect on predictions as a particular feature value changes.
All features¶
As shown next, the All features tab contains the complete list of features being modeled.
Tip
You can control the number of features displayed per page by setting a new value at the bottom of the left-side panel.
Use 
to toggle between ISP and distribution graphs.
To sort the list:
- Click in the Sort by drop-down and select your criteria.
- Click the arrow on the right to toggle the sort direction.
To focus on a single feature:
Click on the desired feature in the left-side panel.
--OR--
- Begin typing in the Find features... search box until you see a match.
- Select a search result by clicking it.
The right-side panel is then refreshed with the corresponding information (defined above) for the feature you want to see.
Feature Groups¶
Feature groups comprise individual features that are closely related or which capture similar information (e.g., "demographic features"). When ingested with the model (optional), these groups of features are displayed by group importance under the Feature Groups tab. Hence, if your model has features which are closely related or otherwise reflect shared characteristics, the computed influences in the final model outcome are scored for each feature group rather than for each individual feature. This is especially valuable when the selected split includes a high number of features.
Filtering by Segments¶
Accessible under all three feature diagnostic tabs is a drop-down control called Filter by segments.
Important
In order to apply segment filters, you must first create and define segments (see Managing Segments for guidance).
Assuming you've already defined segments for your model, click Filter by segments (located just below the Overview tab). Here, you'll find your defined Segment groups for the selected model. To reset your filtering criteria at any time, click reset.
Otherwise, take the following steps:
To filter by feature value:
- Click Feature and select one from the list.
-
Click Filter and select the desired operator. Your choices are as follows:
operation operator equal to == not equal to != less than < less than or equal to <= greater than > greater than or equal to >= -
Click Value and enter a number within the defined range.
- Click the search icon to see the results.
To filter by single segment or combined segments:
- Click a single segment and the current display will update accordingly.
- Click one or more additional segments to see the combined results (one OR the other OR both/all).
Click Next below to continue.
- Click the Model selector.
- Deselect the currently selected model(s), then select the desired model(s).
- Click SELECT.
- Click the Split name selector.
- Select the desired split.
Search for a model or split by entering a parial or full string in the respective search box.
Features that affect or impact the outcome of your model more than other features, either positively or negatively, are said to have influence. The most obvious way to identify an "influential" feature is to delete it from the model's training data. Should this produce signicant changes in model output, the deleted feature can safely be considered influential.
When a single datapoint is an outlier, it is called an "interesting" instance. An interesting instance can skew the influence of a feature.
Hence, it follows that the more a model's parameters or predictions change when the model is retrained with the interesting instance removed, the more influential that instance is.