Features
Called "Features" page up to 1.42 release
Data scientists like to organize things into tables, where a row is an instance/record/ observation/trial representing a single datapoint on a graph, i.e., the person or thing (a single unit thereof) being measured. Each column is an applicable feature or variable (a characteristic) of the row entry — name, age, weight, education, temperature, viscosity, voltage, salinity ... you name it.
But what does it mean?
It means that features constitute what your dataset is comparing, row by row.
It also means that the quality of the features in your dataset impacts the efficacy of the insights gained when the dataset is used for machine learning. Business problems within the same industry do not necessarily require the same features. Everything that can be measured may not be necessary to answer the business problem posed, making it important to have a strong understanding of the business goals and priorities of your modeling project before merely including all available features, i.e., all measurable attributes. Certain attributes simply are not relevant to the analysis being undertaken. In other words, it's often a fine line that separates "too much" from "not enough."
Hence, it's important to weigh the quality of your dataset’s features and their pertinence/relevance to the business problem at hand. Feature selection and feature engineering, notoriously difficult and tedious, are therefore vital to the verity of your model's results. Done well, your continually optimized dataset should contain only those features having a bearing on the problem you're trying to solve. Including more could impose an undue influence on the model. Including less could skew the results in an entirely different way.
TruEra's Explainability diagnostics can help you determine the "Goldilocks Zone" for your model — not too much, not too little — just right. It also helps users determine if the behavior learnt by the model for a particulare feature is expected as per human/expert intuition.
Assessing Feature Behavior¶
To begin, click Explainability under Diagnostics in the nav panel on the left. This displays a page headed by two tabs: Features, and Feature Groups.
Insight gained at this level of analysis involves the influence of input variables on outputs, measuring the model's behavior in terms of input value change, noise tolerance, data quality, internal structure, and more, as a means of uncovering abnormal or unexpected behavior.
Pictured above, the left-side panel under the Features tab contains the current list of features for the selected Model and Split name. (Remember, you can change the model and/or split at any time.) You can click on a feature to enlarge the graph.
Influence Sensitivity Plots¶
Here, the right-side panel shows greater detail for the selected feature in the form of an Influence Sensitivity Plot (ISP). ISPs show the relationship between a feature’s value and its contribution to the model output. Added in composite overlay to contextualize the ISP by showing the influence density (shown on the right).
You can also turn on the Show sampled record view to see point-wise influence visualization for a random sample of 1000 - 10,000 records (configurable in project setting) from your selected split.
When enabled within an ISP, you can also examine (click a link for additional detail):
- Error rate – shows the model's error for bucketized ranges of the feature's value, which can help contextualize the ISP by sanity checking regions against the error rate graph to see as the influence changes does the error rate also change which can help understand whether this feature is aiding or hindering the accuracy of the model.
- Ground truth rate – shows the ground truth rate of the given data split for bucketized ranges of the feature's value. This helps contextualize the ISP with a sanity check of feature behavior for regions associated with ground truth labels and see if the influence relationship correlates to the ground truth in the data. Ex: If as the feature value increases, the influence increases towards a particular class, it would be reasonable to expect (not always) that ground truth for that class also increases
Additionally in the sampled record view, you can also examine
- Best-fit spline – shows a polynomial trend line that is fit to the ISP to tease out a clear relationship between a feature's value and its contribution to the model score.
- Overfitting records – identify influential points of the ISP occurring in low-density regions. Points that drive model scores despite occurring in low-density regions can indicate overfitting.
How is overfitting calculated?
The overfitting diagnostic identifies influential points which occur in low-density or sparse regions of data. Data points with feature values in regions with less than 3% of the data population and with influences at or above the 95th percentile are classified as overfitting. This indicates that the feature value of the point in question is driving a model's prediction, despite being a low-density region of data. If the model overfits on these low-density regions of data, it can fail to generalize, leading to poor test and production performance.
Click SHOW ALL FEATURES to return to the ISPs for all features.
Curve fitting constructs a curve of the best fit to a series of data points. Interpolation is used to estimate the values of unknown data points that fall in between known (ingested) data points.
A spline is a piecewise polynomial parametric curve that minimizes a weighted combination of the average squared approximation error over observed data and the roughness measure. The term comes from the flexible (analog) spline devices used by shipbuilders and draftsmen to draw smooth shapes.
Essentially, spline interpolation fits low-degree polynomials to small subsets of points instead of fitting a single large-degree polynomial to all of them.
It is a generally preferred interpolation method because the interpolation error can be made small.
Overfitting occurs when the model fits exactly against its training data and cannot perform accurately against unseen data, defeating the purpose of machine learning.
More specifically, when the model trains for too long on sample data or when the model is too complex, it can start to learn “noise” (information that isn't relevant) within the dataset. When it memorizes the noise, fitting too closely to the training set, the model becomes “overfitted” — unable to generalize well to new data — making it unable to perform the classification or prediction tasks for which it is intended.
Generally speaking, low error rates and a high variance are good indicators of overfitting. To prevent this type of behavior, you should set up a test data split to check for overfitting. A training split with a low error rate and a test split with a high error rate signals overfitting.
How is the overfitting diagnostic calculated?
Data points with feature values in regions with less than 3% of the data population and with influences at or above the 95th percentile are classified as overfitting.
This indicates that the feature value of the point in question is driving the model's predictions, despite being in a low-density region of data. When the model overfits on these low-density regions of data, it is failing to generalize.
Model accuracy is the number of correctly predicted points out of all the data points modelled, whereas the error rate is a measure of how often your model classifies data incorrectly; in other words, the number of times it is wrong. More formally, model accuracy is defined as the number of true positives and true negative divided by the total number of true positives, true negatives, false positives and false negatives.
Error rate is calculated as the total number of two incorrect predictions (FN + FP) divided by the total number of data points (TP + TN + FP + FN).
Measuring model performance by quantifying the difference between predicted probabilities and actual values, Log loss (logarithmic loss or cross-entropy loss) is a common evaluation metric for binary classification models.
If the error rate of your model is very high — meaning the accuracy of its predictions are very low — then there is likely to be a lot you missed when fitting the model to the dataset you've chosen. Called "underfitting," this often occurs when your model:
- didn’t find any trend in the dataset
- is fitted to the wrong data; i.e., fitting a linear model to nonlinear data.
Otherwise, you can reduce underfitting by increasing the number of data points and reducing the number of superfluous (unecessary/redundant) features in the training split used.
At the other end of the error spectrum is "overfitting," which can happen when there are too many explanatory features selected.
Prevent overfitting by:
- Pausing training before your model starts learning noise — i.e., features unnecessary to the model's predictive purpose.
- Expanding the training split to include more data.
- Selecting features to include in the training data judiciously, identifying the most important ones, eliminating those that are irrelevant or redundant.
- Regularizing your model by applying L1 to feature selection (Lasso regression; shrinking coefficients to zero) or L2 (Ridge regression; shrinking coefficients equally) if features are collinear/codependent.
- Incorporating ensemble methods with a set of classifiers — e.g., decision trees that employ bagging and boosting.
Last but not least, check out this blog about the optimal error rate in model training.
Applicable to supervised algorithms, ground truth is the target for training and/or validating your model with a labeled (annotated) dataset — results already determined to be accurate; i.e., the "reality."
However, only model stakeholders can define the objective for the ground truth algorithm, which is almost always subjective, and on which decision-makers can disagree. Frankly, there are no hard-and-fast rules for defining ground truth labels. Nevertheless, the more high-quality labelled data you make available for training, the better your model will ultimately perform.
You should initially train your model on data with ground truth labels, evaluating it on test datasets for which the model doesn't know the ground truth label, then compare the model's predictions to the actual ground truth label to determine model performance.
The ground truth rate plots the difference (loss/error) between the model's prediction and the ground truth, where ground-truth refers to the "officially correct" label (categorical or numerical) for a given input.
Partial Dependence Plots (PDPs) show you how a given feature affects the model's predictions, helping to extract valuable insights with respect to explainability you can relay to model stakeholders. For TruEra to display a feature's partial dependence plot, however, you must first ingest PDPs for the features you're most interested in analyzing for partial dependency.
To make a partial dependence plot with sklearn, for example, see the scikit-learn guidance, then use the Python SDK's get_partial_dependencies() method (see the Python SDK Technical Reference to ingest the PDPs.
Essentially, PDPs plot the average affect on predictions as a particular feature value changes.
Feature Groups¶
Feature groups comprise individual features that are closely related or which capture similar information (e.g., "demographic features"). When ingested with the model (optional), these groups of features are displayed by group importance under the Feature Groups tab. Hence, if your model has features which are closely related or otherwise reflect shared characteristics, the computed influences in the final model outcome are scored for each feature group rather than for each individual feature. This is especially valuable when the selected split includes a high number of features.
Filtering by Segments¶
Accessible above feature diagnostic tabs is a drop-down control called Segments.
Assuming you've already defined segments for your model, click Segments. Here, you'll find your defined Segment groups for the selected model. To reset your filtering criteria at any time, click Clear Filters.
Otherwise, click on New Segment Filter and define a segment.
Click Next below to continue.
- Click the Model selector.
- Deselect the currently selected model(s), then select the desired model(s).
- Click SELECT.
- Click the Split name selector.
- Select the desired split.
Search for a model or split by entering a parial or full string in the respective search box.
Features that affect or impact the outcome of your model more than other features, either positively or negatively, are said to have influence. The most obvious way to identify an "influential" feature is to delete it from the model's training data. Should this produce signicant changes in model output, the deleted feature can safely be considered influential.
When a single datapoint is an outlier, it is called an "interesting" instance. An interesting instance can skew the influence of a feature.
Hence, it follows that the more a model's parameters or predictions change when the model is retrained with the interesting instance removed, the more influential that instance is.