Skip to content

Python SDK Technical Reference

This reference provides pertinent technical specifications regarding each TruEra Python SDK method and function. Each API call is listed in the navigator panel on the right under Table of contents in alphabetical order, organized by class.

TrueraWorkspace (BaseTrueraWorkspace)

Workspace for Truera computations.

__init__(self, connection_string, authentication, log_level=20, **kwargs) special

Construct a new TruEra workspace.

Parameters:

Name Type Description Default
connection_string str

URL of the TruEra deployment. Defaults to None.

required
authentication TrueraAuthentication

Credentials to connect to TruEra deployment. Defaults to None.

required
log_level int

Log level (defaults to logging.INFO).

20
**verify_cert bool|str

When set to False certificate verification failures will be ignored (not recommended). A path to certificate file or directory can also be provided. If verify_cert is set to a path to a directory, the directory must have been processed using the c_rehash utility supplied with OpenSSL.

required

Exceptions:

Type Description
ValueError

Raised if exactly one of connection_string and authentication is None.

activate_client_setting(self, setting_name) inherited

Activates a setting for client side behavior.

Parameters:

Name Type Description Default
setting_name str

Client setting to activate.

required

add_credential(self, name, secret, identity=None)

[Alpha] Add a new credential to TruEra product. The credential is saved in a secure manner and is used to authenticate with the data source to be able to perform various operations (read, filter, sample etc.).

Parameters:

Name Type Description Default
name

Friendly name of the credential.

required
secret

The secret to be stored.

required
identity

Identity portion of the secret. Not needed in all cases. Defaults to None.

None

Returns:

Type Description
Credential

Returns an object with the credential name and id. The secret is not stored in this object.

add_data(self, data, *, data_split_name, column_spec, model_output_context=None, is_production_data=False, **kwargs)

Add data by either creating a new split or appending to an existing split. The split will be set in the current context.

ColumnSpec/NLPColumnSpec and ModelOutputContext classes can be imported from truera.client.ingestion. Alternatively column_spec and model_output_context can be specified as Python dictionaries.

Parameters:

Name Type Description Default
data Union[pd.DataFrame, 'Table']

A pd.DataFrame or Table containing the data to be added.

required
data_split_name str

The name of the split to be created or appended to.

required
column_spec Union[ColumnSpec, NLPColumnSpec, Mapping[str, Union[str, Sequence[str]]]]

The ColumnSpec or NLPColumnSpec mapping column names in the data to corresponding data kind. Parameters include: id_col_name, timestamp_col_name, pre_data_col_names, post_data_col_names, prediction_col_names, label_col_names, extra_data_col_names, feature_influence_col_names, token_influence_col_names, tags_col_name, token_col_name, sentence_embeddings_col_name

required
model_output_context Optional[Union[ModelOutputContext, dict]]

Contextual information about data involving a model, such as the model name and score type. This argument can be omitted in most cases, as the workspace infers the appropriate values from the context.

None

add_data_collection(self, data_collection_name, pre_to_post_feature_map=None, provide_transform_with_model=None)

Creates and sets the current data collection to use for all operations in the workspace. Must specify feature mapping if some data transformation is being done.

Parameters:

Name Type Description Default
data_collection_name str

Name of the data_collection.

required
pre_to_post_feature_map Optional[Mapping[str, Sequence[str]]]

One-to-many mapping from a single pre-processed human-readable feature name to post-processed model-readable feature names. Ignored if post-processed data is not provided for the data collection.

None
provide_transform_with_model Optional[bool]

Specifies if transform function will be mentioned for each model using this data collection. If True, then feature mapping is via a transform function and if False, then separate pre-/post-transform data is expected in all splits. Defaults to None. Must be set, if feature map is provided for the data collection.

None

Exceptions:

Type Description
ValueError

Raised if no project is associated with the current workspace. Use set_project to set the correct project.

Examples:

>>> tru.set_project("Project Name")
>>> feature_map = {"numeric_feature":["numeric_feature"],
                "numeric_feature":["bucket_1","bucket_2"],
                "categorical feature":["categorical_onehot_1", "categorical_onehot_2"]}
>>> tru.add_data_collection("Data Collection Name", pre_to_post_feature_map = feature_map)

add_data_source(self, name, uri, credential=None, **kwargs)

Add a new data source in the system.

Parameters:

Name Type Description Default
name str

Friendly name of the data source.

required
uri str

URI describing the location of the data source. For local files this can be file:///path/to/my/file or /path/to/my/file For files stored in Azure Storage Blobs the expected path is wasb://container@account.blob.core.windows.net/blob

required
credential Credential

Provide the credential object if the data source requires authentication to read from it. Defaults to None.

None
**format str

The format in which the file (local) or blob (AWS S3, Azure WASB etc.) are stored in.

required
**first_row_is_header bool

For text based delimited files (csv, tsv etc.), indicates if the first row provides header information. Defaults to True.

required
**column_delimiter str

For text based delimited files (csv, tsv etc.), provides the delimiter to separate column values. Defaults to ','.

required
**quote_character str

For text based delimited files (csv, tsv etc.), if quotes are used provide the quote character. Defaults to '"'.

required
**null_value str

For text based delimited files (csv, tsv etc.), the string that signifies null value. Defaults to 'null'.

required
**empty_value str

For text based delimited files (csv, tsv etc.), the string that signifies empty value. Defaults to '""'.

required
**date_format str

For text based delimited files (csv, tsv etc.), if any column has date time, provide the format string. Defaults to 'yyyy-MM-dd-HH🇲🇲ssZZ'.

required
**account_key str

For reading from Azure Storage Blob (WASB), provide the account_key to be used to read the blob. Not required if credential object is provided.

required
**database_name str

For reading from sql databases, provide the database name to use. Required for MySQL or Hive data source.

required
**table_name str

For reading from sql databases, provide the table name to use. Required for MySQL or Hive data source.

required

Exceptions:

Type Description
ValueError

Raised if the current project is a local project.

Returns:

Type Description
Table

Returns a Table object which allows interaction with the attached data.

add_extra_data(self, extra_data, extras_col_names, id_col_name, **kwargs)

[Alpha] Add extra data to an existing data split.

Parameters:

Name Type Description Default
extra_data Union[Table, str]

A Table or URI of file containing the label data. Table: Use add_data_source or get_data_source to get a Table. You can optionally futher filter the Table by applying filter. URI: Used to load external data source into the system. This is similar to ingesting via add_data_source and Table, without the filters. The data in the external data source will be sub-sampled (random or first N) and ingested into the system. By default up to 5000 rows are ingested as the split. You can override this by specifying sample_count=X. Sampling is either done by using first N rows or sampling rows randomly (without replacement). Defaults to random. You can override this by specifying sample_kind="first".

required
extras_col_names Union[str, Sequence[str]]

Column name(s) for the extra data columns to be ingested.

required
id_col_name str

Column name for the unique row identifier in the provided label_data. Used to match labels with corresponding data points.

required
**credential Credential

Provide the credential object if the data source requires authentication to read from it. Defaults to None.

required
**format str

The format in which the file (local) or blob (AWS S3, Azure WASB etc.) are stored in.

required
**first_row_is_header bool

For text based delimited files (csv, tsv etc.), indicates if the first row provides header information. Defaults to True.

required
**column_delimiter str

For text based delimited files (csv, tsv etc.), provides the delimiter to separate column values. Defaults to ','.

required
**quote_character str

For text based delimited files (csv, tsv etc.), if quotes are used provide the quote character. Defaults to '"'.

required
**null_value str

For text based delimited files (csv, tsv etc.), the string that signifies null value. Defaults to 'null'.

required
**empty_value str

For text based delimited files (csv, tsv etc.), the string that signifies empty value. Defaults to '""'.

required
**date_format str

For text based delimited files (csv, tsv etc.), if any column has date time, provide the format string. Defaults to 'yyyy-MM-dd HH🇲🇲ssZZ'.

required
**account_key str

For reading from Azure Storage Blob (WASB), provide the account_key to be used to read the blob. Not required if credential object is provided.

required
**access_key_id str

For reading from a s3 bucket, provide the access key id to be used to read the blob. Not required if credential object is provided or the underlying deployment is in a role with access to the bucket.

required
**secret_access_key str

For reading from a s3 bucket, provide the secret access key to be used to read the blob. Not required if credential object is provided or the underlying deployment is in a role with access to the bucket.

required
**database_name str

For reading from MySQL database, provide the database name to use. Required for MySQL data source.

required
**table_name str

For reading from MySQL database, provide the table name to use. Required for MySQL data source.

required
**sample_count int

Maximum rows to use when creating the split. Defaults to 5000.

required
**sample_kind str

Specifies the strategy to use while sub-sampling the rows. Defaults to "random".

required
**timeout_seconds int

Number of seconds to wait for data source. Defaults to 300.

required

add_feature_metadata(self, feature_description_map=None, group_to_feature_map=None, missing_values=None, force_update=False)

Upload metadata describing features and feature groupings to the server.

Parameters:

Name Type Description Default
feature_description_map Optional[Mapping[str, str]]

Map from pre-processed feature name, as provided in the data, to the description of the feature.

None
group_to_feature_map Optional[Mapping[str, Sequence[str]]]

Grouping of pre-features for analysis purposes. A key of the map will be a name for the collection of pre-features mapped to. If given, all pre-features must appear in exactly one of the map's value lists.

None
missing_values Optional[Sequence[str]]

List of strings to be registered as missing values when reading split data.

None
force_update bool

Overwrite any existing feature metadata.

False

add_labels(self, label_data, label_col_name, id_col_name, **kwargs)

[Alpha] Add labels to an existing data split.

Parameters:

Name Type Description Default
label_data Union[Table, str]

A Table or URI of file containing the label data. Table: Use add_data_source or get_data_source to get a Table. You can optionally futher filter the Table by applying filter. URI: Used to load external data source into the system. This is similar to ingesting via add_data_source and Table, without the filters. The data in the external data source will be sub-sampled (random or first N) and ingested into the system. By default up to 5000 rows are ingested as the split. You can override this by specifying sample_count=X. Sampling is either done by using first N rows or sampling rows randomly (without replacement). Defaults to random. You can override this by specifying sample_kind="first".

required
label_col_name str

Column name for the labels or ground truth in the provided label_data.

required
id_col_name str

Column name for the unique row identifier in the provided label_data. Used to match labels with corresponding data points.

required
**credential Credential

Provide the credential object if the data source requires authentication to read from it. Defaults to None.

required
**format str

The format in which the file (local) or blob (AWS S3, Azure WASB etc.) are stored in.

required
**first_row_is_header bool

For text based delimited files (csv, tsv etc.), indicates if the first row provides header information. Defaults to True.

required
**column_delimiter str

For text based delimited files (csv, tsv etc.), provides the delimiter to separate column values. Defaults to ','.

required
**quote_character str

For text based delimited files (csv, tsv etc.), if quotes are used provide the quote character. Defaults to '"'.

required
**null_value str

For text based delimited files (csv, tsv etc.), the string that signifies null value. Defaults to 'null'.

required
**empty_value str

For text based delimited files (csv, tsv etc.), the string that signifies empty value. Defaults to '""'.

required
**date_format str

For text based delimited files (csv, tsv etc.), if any column has date time, provide the format string. Defaults to 'yyyy-MM-dd HH🇲🇲ssZZ'.

required
**account_key str

For reading from Azure Storage Blob (WASB), provide the account_key to be used to read the blob. Not required if credential object is provided.

required
**access_key_id str

For reading from a s3 bucket, provide the access key id to be used to read the blob. Not required if credential object is provided or the underlying deployment is in a role with access to the bucket.

required
**secret_access_key str

For reading from a s3 bucket, provide the secret access key to be used to read the blob. Not required if credential object is provided or the underlying deployment is in a role with access to the bucket.

required
**database_name str

For reading from MySQL database, provide the database name to use. Required for MySQL data source.

required
**table_name str

For reading from MySQL database, provide the table name to use. Required for MySQL data source.

required
**sample_count int

Maximum rows to use when creating the split. Defaults to 5000.

required
**sample_kind str

Specifies the strategy to use while sub-sampling the rows. Defaults to "random".

required
**timeout_seconds int

Number of seconds to wait for data source. Defaults to 300.

required

add_model(self, model_name, train_split_name=None, train_parameters=None)

Registers and adds a new model in TruEra. By default, the model is "virtual" in that it does not have an executable model object attached. To add the model object itself, see add_python_model().

Parameters:

Name Type Description Default
model_name

Name of model to create

required
train_split_name Optional[str]

The name of the train split of the model

None
train_parameters Optional[Mapping[str, Any]]

Train parameters of the model. Ex. {'n_estimators": 10}"}

None

add_model_error_influences(self, error_influence_data, score_type=None, *, data_split_name=None, background_split_name=None, id_col_name=None, timestamp_col_name=None, influence_type=None)

Adds error influence of given score type for the current model and split.

Parameters:

Name Type Description Default
error_influence_data pd.DataFrame

Feature influence data to add. Must be aligned with the pre-processed data of the given split.

required
score_type Optional[str]

Score type of the influences, either mean_absolute_error_for_classification or mean_absolute_error_for_regression, depending on project type. Inferred if None. Defaults to None.

None
id_col_name Optional[str]

Column name for the unique identifier of each data point.

None
data_split_name Optional[str]

Data split that influences are associated with. If None, defaults to split set in the current context.

None
background_split_name Optional[str]

Background data split that influences are computed against. If None, defaults to the base split of the data collection (if this is not explicitly set, it is an ingested split of type "all" or "train").

None
influence_type Optional[str]

Influence algorithm used to generate influences. If influence type of project is set to "truera-qii", assumes that explanations are generated using truera-qii. If influence type of project is set to "shap", then influence_type must be passed in as one of ["tree-shap-tree-path-dependent", "tree-shap-interventional", "kernel-shap"].

None
timestamp_col_name Optional[str]

Column name for the timestamp of each data point. Must be a column of type string or pd.DateTime. Defaults to None.

None

add_model_feature_influences(self, feature_influence_data, *, id_col_name, data_split_name=None, background_split_name=None, timestamp_col_name=None, influence_type=None, score_type=None)

Adds feature influence for the current model and split. Assumes influences are calculated for the default score type of the project.

Parameters:

Name Type Description Default
feature_influence_data pd.DataFrame

Feature influence data to add. Must be aligned with the pre-processed data of the given split.

required
id_col_name str

Column name for the unique identifier of each data point.

required
data_split_name Optional[str]

Data split that influences are associated with. If None, defaults to split set in the current context.

None
background_split_name Optional[str]

Background data split that influences are computed against. If None, defaults to the base split of the data collection (if this is not explicitly set, it is an ingested split of type "all" or "train").

None
timestamp_col_name Optional[str]

Column name for the timestamp of each data point. Must be a column of type string or pd.DateTime. Defaults to None.

None
influence_type Optional[str]

Influence algorithm used to generate influences. If influence type of project is set to "truera-qii", assumes that explanations are generated using truera-qii. If influence type of project is set to "shap", then influence_type must be passed in as one of ["tree-shap-tree-path-dependent", "tree-shap-interventional", "kernel-shap"].

None
score_type Optional[str]

The score type to use when computing influences. If None, uses default score type of project. Defaults to None. For a list of valid score types, see list_valid_score_types.

None

add_model_metadata(self, train_split_name=None, train_parameters=None, overwrite=False)

Add or update metadata for the current model in context.

Parameters:

Name Type Description Default
train_split_name Optional[str]

The name of the train split of the model

None
train_parameters Optional[Mapping[str, Any]]

Train parameters of the model. Ex. {'n_estimators": 10}"}

None
overwrite bool

Overwrite existing values (if exist).

False

add_model_predictions(self, prediction_data, id_col_name=None, *, prediction_col_name=None, data_split_name=None, ranking_group_id_column_name=None, ranking_item_id_column_name=None, score_type=None)

Adds prediction data for the current model. Assumes predictions are calculated for the default score type of the project.

Parameters:

Name Type Description Default
prediction_data Union[pd.DataFrame, Table]

Prediction data to add.

required
id_col_name str

Column name for the unique row identifier in the provided prediction_data. Used to match predictions with corresponding split data.

None
prediction_col_name Optional[str]

Column name from which to pull prediction data. Only required if prediction_data is a Table or URI.

None
data_split_name Optional[str]

Data split that predictions are associated with. If None, defaults to split set in the current context.

None
ranking_group_id_column_name Optional[str]

Column name for group id for ranking projects.

None
ranking_item_id_column_name Optional[str]

Column name for item id for ranking projects.

None
score_type Optional[str]

Specifies the score type for prediction data, if provided. Defaults to None, in which case the score type of the project is used.

None

add_nn_data_split(self, data_split_name, truera_wrappers, split_type='all', *, pre_data=None, label_data=None, label_col_name=None, id_col_name=None, extra_data_df=None)

[Alpha] Upload NN data split to TruEra server.

Examples:

# During NN Ingestion to add a split you will create wrappers
>>> from truera.client.nn.wrappers.autowrap import autowrap
>>> truera_wrappers = autowrap(...) # Use the appropriate NN Diagnostics Ingestion to create this

# Add the data split to the truera workspace
>>> tru.add_nn_data_split(
>>>     data_split_name="<split_name>",
>>>     truera_wrappers,
>>>     split_type="<split_type_train_or_test>"
>>> )

Parameters:

Name Type Description Default
data_split_name str

Name of the split to be uploaded.

required
truera_wrappers base.WrapperCollection

A Base.WrapperCollection housing a base.Wrappers.SplitLoadWrapper that helps load data files from files. This must be implemented via subclassing the truera.client.nn.wrappers.Base.SplitLoadWrapper

required
split_type Optional[str]

The type of the data split. Options are ["all", "train", "test", "validate", "oot", "prod", "custom"]. Defaults to "all".

'all'
pre_data Optional[Union[np.ndarray, pd.DataFrame]]

Data.

None
label_data Optional[pd.DataFrame]

Label data.

None
label_col_name Optional[str]

The column name in pre_data containig label data

None
id_col_name Optional[str]

The column name in pre_data containing record ID data

None
extra_data_df Optional[pd.DataFrame]

Extra columns which are not used / consumed by the model, but could be used for other analysis like defining segments.

None

add_nn_model(self, model_name, truera_wrappers, attribution_config, model=None, train_split_name=None, train_parameters=None, **kwargs)

[Alpha] Upload a NN model to TruEra server. The model is also attached to the workspace as the current model.

Examples:

# During NN Ingestion you will create two objects
>>> from truera.client.nn.client_configs import NLPAttributionConfiguration
>>> attr_config = NLPAttributionConfiguration(...)

>>> from truera.client.nn.wrappers.autowrap import autowrap
>>> truera_wrappers = autowrap(...) # Use the appropriate NN Diagnostics Ingestion to create this

# Add the model to the truera workspace
>>> tru.add_nn_model(
>>>     model_name="<model_name>",
>>>     truera_wrappers,
>>>     attr_config
>>> )

Parameters:

Name Type Description Default
model_name str

Name assigned to the model.

required
truera_wrappers base.WrapperCollection

A set of wrappers to help truera run your model. The tutorial should help you get them.

required
attribution_config dict

An attribution config containing attribution run parameters.

required
model Optional[Any]

Your model object.

None
train_split_name Optional[str]

The name of the train split of the model.

None
train_parameters Optional[Mapping[str, Any]]

Train parameters of the model. Ex. {'n_estimators": 10}"}

None

Exceptions:

Type Description
ValueError

Raised if no project is associated with the current workspace. Use set_project to set the correct project.

ValueError

Raised if workspace is not attached to a data_collection. Either attach workspace to a data-collection or provide the data_collection_name.

ValueError

Raised if the provided data_collection_name does not exist in the current project.

add_packaged_python_model(self, model_name, model_dir, *, data_collection_name=None, train_split_name=None, train_parameters=None, verify_model=True, compute_predictions=None, compute_feature_influences=False, compute_for_all_splits=False)

Registers and adds a new model, along with a pre-serialized and packaged executable Python model object. The model is also attached to the workspace as the current model.

Parameters:

Name Type Description Default
model_name str

Name assigned to the model.

required
model_dir str

Directory where packaged model is located.

required
data_collection_name Optional[str]

Data collection to attach to the model, by default the data collection attached to the workspace will be used. Defaults to None.

None
train_split_name Optional[str]

The name of the train split of the model

None
train_parameters Optional[Mapping[str, Any]]

Train parameters of the model. Ex. {'n_estimators": 10}"}

None
verify_model bool

Locally verify the model is packaged properly and can operate on existing split data. Defaults to True.

True
compute_predictions Optional[bool]

Trigger computations of model predictions on the base/default split of the model's data collection, if such a split exists. Ignored for local models. Defaults to True when using local_execution for remote workspace.

None
compute_feature_influences bool

Trigger computations of model feature influences on the base/default split of the model's data collection, if such a split exists. Ignored for local models.

False
compute_for_all_splits bool

If compute_predictions and/or compute_feature_influences, triggers computations for all existing data splits (not solely the base/default split). Ignored for local models.

False

add_production_data(self, data, *, column_spec, model_output_context=None, **kwargs)

Add production data.

ColumnSpec and ModelOutputContext classes can be imported from truera.client.ingestion. Alternatively column_spec and model_output_context can be specified as Python dictionaries.

Parameters:

Name Type Description Default
data Union[pd.DataFrame, 'Table']

A pd.DataFrame or Table containing the data to be added.

required
column_spec Union[ColumnSpec, NLPColumnSpec, Mapping[str, Union[str, Sequence[str]]]]

The ColumnSpec mapping column names in the data to corresponding data kind. Parameters include: id_col_name, timestamp_col_name, pre_data_col_names, post_data_col_names, prediction_col_names, label_col_names, extra_data_col_names, feature_influence_col_names, token_influence_col_names, tags_col_name, token_col_name, sentence_embeddings_col_name

required
model_output_context Optional[Union[ModelOutputContext, dict]]

Contextual information about data involving a model, such as the model name and score type. This argument can be omitted in most cases, as the workspace infers the appropriate values from the context.

None

add_project(self, project, score_type, input_type='tabular', num_default_influences=None)

Adds and sets project to use for the current workspace environment. This will unset the rest of the context (data collection, data split, model, etc) if set prior.

Parameters:

Name Type Description Default
project str

Name of the project.

required
score_type str

Scorer type configuration for the project. Options are ["logits", "probits", "classification", "regression"].

required
input_type Optional[str]

Input data type for the project. Must be one of ["tabular", "time_series_tabular"]. Defaults to "tabular".

'tabular'
num_default_influences Optional[int]

Number of influences used by default for most influence-requiring graphs, computations, etc. Note that this will take the first of the provided many from the data split --- therefore, shuffling data splits is generally advised prior to ingestion. If creating a project and left as None, then will be set as 1000.

None

Examples:

# Create a probits project
>>> tru.add_project("Project Name", score_type = "probits")

add_python_model(self, model_name, model, transformer=None, *, additional_pip_dependencies=None, additional_modules=None, classification_threshold=None, train_split_name=None, train_parameters=None, verify_model=True, compute_predictions=None, compute_feature_influences=False, compute_for_all_splits=False, **kwargs)

Registers and adds a new model, including the executable model object provided. This method deduces the model framework to appropriately serialize and upload the model object to TruEra server. Models of supported frameworks can be passed directly. Supported Model Frameworks: sklearn, xgboost, catboost, lightgbm, pyspark (tree models only). If you cannot ingest your model via this function due to custom logic, feature transforms, etc., see create_packaged_python_model(). [ALPHA] For frameworks that are not yet supported, or for custom model implementations the prediction function for the model can be provided as the model. For binary classifiers, the prediction function should accept a pandas DataFrame as input and produce a pandas DataFrame as output with the class probabilities and [0, 1] as the column header. For regression models, the prediction function should accept a pandas DataFrame as input and produce the result as a pandas DataFrame with "Result" as the column header. All required dependencies to execute the prediction function should be provide as additional_pip_dependencies. For example:

add_segment_group(self, name, segment_definitions)

[Alpha] Create a segment group where each segments is defined by an SQL expression.

Parameters:

Name Type Description Default
name str

The name of the segment group.

required
segment_definitions Mapping[str, str]

A dictionary containing the name of the segment and an SQL expression that defines the segment. Supported expressions: = : filters for equality, ex: amount = 1000 != : filters for inequality, ex: amount != 1000 < : filters for less-than, ex: amount < 1000 <= : filters for less-than-or-equal, ex: amount <= 1000 > : filters for greater-than, ex: amount > 1000 >= : filters for great-than-or-equal, ex: amount >= 1000 NOT: filters records if the inner condition is not true, ex: NOT(amount >= 1000) AND: filters records if both the conditions are true, ex: (amount >= 1000) AND (state = 'WA') OR : filters records if any of the two conditions is true, ex: (amount >= 1000) OR (state = 'WA') String literals should be within quotes (''), numeric literals should not have quotes(''). Left side of a binary expression should be either column name (without quotes) or one of the special keywords (see below for the list of supported special keywords). Right side of an expression should be a literal. For example, (amount < salary) is not a valid expression, as both left and right side of the expression are column-names. Supported special keywords: DATA_GROUND_TRUTH : filters using the data label, ex: _DATA_GROUND_TRUTH == 1 _MODEL: filters using the of the model where is "REGRESSION" for regression models or one of ["PROBITS", "LOGITS", "CLASSIFICATION"] for classification models, ex: MODEL_REGRESSION >= 5 _MODEL\(<model_name>: filters using the <score_type> of the model with name <model_name> where <score_type> is "REGRESSION" for regression models or one of ["PROBITS", "LOGITS", "CLASSIFICATION"] for classification models, example to filter by "REGRESSION" score of "model_a": _MODEL_REGRESSION\)model_a >= 5 _RANKING_GROUP_ID: filters using the ranking group id, ex: _RANKING_GROUP_ID = "group_id_1"

required

Examples:

>>> tru.set_project("Project Name")
>>> tru.set_data_collection("Data Collection Name")

# Add a segment group for Sex
>>> tru.add_segment_group(name = "Sex", segment_definitions = {"Male": "Sex == 'Male'", "Female": "Sex == 'Female'"})

# Add a segment group for Language at Home
>>> tru.add_segment_group("Language at Home", {"English": "LANX == 1", "Not English": "LANX == 2"})

attach_packaged_python_model_object(self, model_object_dir, verify_model=True)

Attaches a pre-serialized and packaged executable model object to the current model, which must be virtual. This effectively "converts" the virtual model to a non-virtual one, as the system can now call the model to generate predictions.

Parameters:

Name Type Description Default
model_object_dir str

Directory where packaged model object is located.

required
verify_model bool

Locally verify the model is packaged properly and can operate on existing split data. Defaults to True.

True

attach_python_model_object(self, model_object, additional_pip_dependencies=None, verify_model=True)

Attaches the provided executable model object to the current model, which must be virtual. This effectively "converts" the virtual model to a non-virtual one, as the system can now call the model to generate predictions. Supported Model Frameworks: sklearn, xgboost, catboost, lightgbm, pyspark (tree models only). If you cannot ingest your model object via this function due to custom logic, feature transforms, etc., see attach_packaged_python_model_object().

[ALPHA] For frameworks that are not yet supported, or for custom model implementations the prediction
function for the model can be provided as the model.
For binary classifiers, the prediction function should accept a pandas DataFrame as input and produce
a pandas DataFrame as output with the class probabilities and [0, 1] as the column header.
For regression models, the prediction function should accept a pandas DataFrame as input and produce
the result as a pandas DataFrame with "Result" as the column header.
All required dependencies to execute the prediction function should be provide as additional_pip_dependencies.
For example:
```python
def predict(df):
    return pd.DataFrame(my_model.predict_proba(df, validate_features=False), columns=[0, 1])
tru.add_python_model("my_model", predict, additional_pip_dependencies=["xgboost==1.3.1", "pandas==1.1.1"])
```

Parameters:

Name Type Description Default
model_object Any

The Python model object or the prediction function to attach. Supported frameworks are catboost, lightgbm, sklearn and xgboost, and tree-based PySpark models. For prediction function, please see the description above.

required
additional_pip_dependencies Optional[Sequence[str]]

List of pip dependencies required to execute the model object. If the model object is from a supported framework, the pip dependency for that framework is automatically inferred. If a prediction function is provided as the model, additional pip dependencies are not automatically inferred and must be explicitly provided. Defaults to None. Example: ["pandas==1.1.1", "numpy==1.20.1"]

None
verify_model bool

[Alpha] Locally verify the model is packaged properly and can operate on existing split data. Defaults to True.

True

cancel_scheduled_ingestion(self, workflow_id)

[Alpha] Cancel a scheduled ingestion.

Parameters:

Name Type Description Default
workflow_id str

The id of the scheduled ingestion workflow.

required

Returns:

Type Description
str

Returns an object containing the canceled_on timestamp of a workflow.

compute_all(self, computations=['predictions', 'feature influences', 'error influences'], models=None, data_splits=None, data_collection=None, num_influences=None) inherited

Executes and ingests all specified computations over models and data-splits in a data collection

Parameters:

Name Type Description Default
computations Optional[Sequence[str]]

Types of computations. Defaults to ["predictions", "feature influences", "error influences"].

['predictions', 'feature influences', 'error influences']
models Optional[Sequence[str]]

Models to compute with. All models in data collection if None. Defaults to None.

None
data_splits Optional[Sequence[str]]

Data-splits to compute with. All data-splits in data collection if None. Defaults to None.

None
data_collection Optional[str]

Data collection to compute over. Uses data collection in context if None. Defaults to None.

None
num_influences Optional[int]

Number of influences to calculate for influence computations. Uses project settings if None. Defaults to None.

None

compute_error_influences(self, start=0, stop=None, score_type=None, system_data=False, by_group=False, num_per_group=None, wait=True, ingest=True)

Compute the error QIIs/shapley-values associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
score_type Optional[str]

The score type to use when computing error influences. If None, infers error score type based on project configuration. Defaults to None.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None
wait bool

Whether to wait for the job to finish. Defaults to True.

True
ingest bool

Whether to ingest computed feature influences. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values associated with the current data-split.

compute_feature_influences(self, start=0, stop=None, score_type=None, system_data=False, by_group=False, num_per_group=None, wait=True, ingest=True)

Compute the QIIs/shapley-values associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
score_type Optional[str]

The score type to use when computing influences. If None, uses default score type of project. Defaults to None. For a list of valid score types, see list_valid_score_types.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None
wait bool

Whether to wait for the job to finish. Defaults to True.

True
ingest bool

Whether to ingest computed feature influences. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values associated with the current data-split.

compute_predictions(self, start=0, stop=None, system_data=False, wait=True, ingest=True)

Compute predictions over the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of rows.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

required
num_per_group

For ranking projects and when by_group is True, the number of points per group to return.

required
wait bool

Whether to wait for the job to finish. Defaults to True.

True
ingest bool

Whether to ingest predictions. Defaults to True.

True

Returns:

Type Description

The predictions for the current data-split.

create_packaged_python_model(self, output_dir, model_obj=None, additional_pip_dependencies=None, additional_modules=None, model_path=None, model_code_files=None, **kwargs)

Creates a template of a packaged Python model object to be edited and then uploaded to TruEra server. Can be used to package either Python model objects, or serialized model data. This workflow should only be used for custom models or to debug model ingestion. To upload a models of a known framework, it is recommended to use the add_python_model() function.

Parameters:

Name Type Description Default
output_dir str

Path to the directory to create template. Cannot be an existing directory.

required
model_obj Optional[Any]

The Python model object to package. Supported frameworks are catboost, lightgbm, sklearn, xgboost, and tree-based PySpark models.

None
additional_pip_dependencies Optional[Sequence[str]]

List of pip dependencies required to execute the model. When model object is from a supported framework, pip dependency for that framework is automatically inferred. If a prediction function is provided as the model, additional pip dependencies are not automatically inferred and must be explicitly provided. Defaults to None. Example: ["pandas==1.1.1", "numpy==1.20.1"].

None
additional_modules Optional[Sequence[Any]]

List of modules not available as pip packages required for the model. These must already be imported. Defaults to None.

None
model_path Optional[str]

Path to a model file or directory. Can be a serialized model or a directory containing multiple files of serialized data. Ignored if model_obj is passed in.

None
model_code_files Optional[Sequence[str]]

List of paths to additional files to be packaged with the model. Ignored if model_obj is passed in.

None

deactivate_client_setting(self, setting_name) inherited

Deactivates a setting for client side behavior.

Parameters:

Name Type Description Default
setting_name str

Client setting to deactivate.

required

delete_credential(self, name)

[Alpha] Removes a credential from the TruEra product.

Parameters:

Name Type Description Default
name

Friendly name of the credential.

required

delete_data_collection(self, data_collection_name=None, *, recursive=False)

Delete a data collection from the current TruEra workspace. This will only delete artifacts within the current location context (either local or remote). Note: Data collection can be deleted only after all the data splits in the data collection have been deleted.

Parameters:

Name Type Description Default
data_collection_name Optional[str]

Name of the data collection to be deleted. By default the currently set data collection will be deleted.

None
recursive bool

Whether to delete any data splits as well in the data collection. Defaults to False.

False

delete_data_source(self, name)

Delete a data source that was already created in the system.

Parameters:

Name Type Description Default
name str

The friendly name of the data source.

required

delete_data_split(self, data_split_name=None, *, recursive=False)

Delete a data split from the current TruEra workspace. This will only delete artifacts within the current location context (either local or remote).

Parameters:

Name Type Description Default
data_split_name Optional[str]

Name of the data split to be deleted. By default the currently set data split will be deleted.

None
recursive bool

Whether to delete any model tests associated with the data split. Defaults to False.

False

delete_model(self, model_name=None, *, recursive=False)

Delete a model from the current TruEra workspace. This will only delete artifacts within the current location context (either local or remote).

Parameters:

Name Type Description Default
model_name Optional[str]

Name of the model to be deleted. By default the currently set model will be deleted.

None
recursive bool

Whether to delete any model tests associated with the model. Defaults to False.

False

delete_model_metadata(self)

Unset train_split_name and train_parameters for the current model in context.

delete_project(self, project_name=None)

Delete a project from the current TruEra workspace. This will only delete artifacts within the current location context (either local or remote).

Parameters:

Name Type Description Default
project_name Optional[str]

Name of the project to be deleted. By default the currently set project will be deleted.

None

delete_segment_group(self, name)

[Alpha] Delete a segment group.

Parameters:

Name Type Description Default
name str

The name of the segment group.

required

get_client_setting_value(self, setting_name) inherited

Gets current value of a setting for client side behavior.

get_credential_metadata(self, name)

[Alpha] Get metadata about a credential in the TruEra product. The credential details are not returned.

Parameters:

Name Type Description Default
name

Friendly name of the credential.

required

Returns:

Type Description
Credential

Returns an object with the credential name and id.

get_data_collections(self)

Get all data-collections in the connected project.

Exceptions:

Type Description
ValueError

Raised if the workspace isn't connected to any project.

Returns:

Type Description
Sequence[str]

Name of data-collections in the project.

get_data_source(self, name)

Get a data source that was already created in the system.

Parameters:

Name Type Description Default
name str

The friendly name of the data source.

required

Exceptions:

Type Description
ValueError

Raised if the current project is a local project.

Returns:

Type Description
Table

Returns a Table object which allows interaction with the attached data.

get_data_sources(self)

Get list of data sources attached in the current project.

Exceptions:

Type Description
ValueError

Raised if the current project is a local project.

get_data_splits(self)

Get all data-splits in the connected data-collection.

Exceptions:

Type Description
ValueError

Raised if the workspace isn't connected to any project.

ValueError

Raised if the workspace isn't connected to any data-collection.

Returns:

Type Description
Sequence[str]

Name of data-splits in the project.

get_default_performance_metrics(self)

Get the default performance metrics of the currently set project.

Returns:

Type Description
Sequence[str]

Default performance metrics.

get_error_influences(self, start=0, stop=None, score_type=None, system_data=False, by_group=False, num_per_group=None)

Get the error QIIs/shapley-values associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
score_type Optional[str]

The score type of error influences to retrieve. If None, infers error score type based on project configuration. Defaults to None.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values associated with the current data-split.

get_explainer(self, base_data_split=None, comparison_data_splits=None)

Get the explainer associated with the TruEra workspace.

Parameters:

Name Type Description Default
base_data_split Optional[str]

The name of the data split to set as the base split for explainer operations.

None
comparison_data_splits Optional[Sequence[str]]

The name(s) of the data splits to set as the comparison splits for explainer operations.

None

Returns:

Type Description
Explainer

Explainer for current context of the TruEra workspace.

get_feature_influences(self, start=0, stop=None, score_type=None, system_data=False, by_group=False, num_per_group=None)

Get the QIIs/shapley-values associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the project setting for "Number of default influences".

None
score_type Optional[str]

The score type to use when computing influences. If None, uses default score type of project. Defaults to None. For a list of valid score types, see list_valid_score_types.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values associated with the current data-split.

get_feature_names(self) inherited

Get the feature names associated with the current data-collection.

Returns:

Type Description
Sequence[str]

Feature names.

get_influence_type(self)

Get the influence algorithm type of the currently set project.

get_influences_background_data_split(self, data_collection_name=None)

Get the background data split used for computing feature influences.

Parameters:

Name Type Description Default
data_collection_name Optional[str]

(Optional) Name of the data collection. Defaults to the current data collection in context.

None

Returns:

Type Description
str

Name of the background data split.

get_ingestion_client(self)

[Alpha] Get the data ingestion client associated with the TruEra workspace. Valid only for "remote" workspace. The ingestion client can be used to pull data from different data sources into TruEra to perform analytics.

Exceptions:

Type Description
ValueError

Raised if no project is associated with the current workspace. Use set_project to set the correct project.

ValueError

Raised if no data collection is associated with the current workspace. Use set_data_collection to set the correct data collection.

ValueError

Raised if the current project is a local project.

Returns:

Type Description
IngestionClient

IngestionClient for current context of the TruEra workspace.

get_maximum_model_runner_failure_rate(self)

Get the maximum model runner failure rate (fraction of points on which the model can fail for a model run to be considered successful) for the current project.

get_model_metadata(self)

Get model metadata for the current model in context.

get_model_threshold(self)

Gets the model threshold for the currently set model and score type in the TruEra workspace.

Returns:

Type Description
Optional[float]

The model threshold.

get_models(self)

Get all models in the connected project.

Exceptions:

Type Description
ValueError

Raised if the workspace isn't connected to any project.

Returns:

Type Description
Sequence[str]

Name of models in the project.

get_nn_user_configs(self)

Get NN user configurations for project and model set in the current context.

Examples:

>>> from truera.client.nn.client_configs import NLPAttributionConfiguration
>>> attr_config = NLPAttributionConfiguration(
>>>     token_embeddings_layer=token_embeddings_layer_name,
>>>     token_embeddings_anchor=token_embeddings_layer_tensor_anchor,
>>>     n_output_neurons=n_output_neurons,
>>>     n_metrics_records=n_metrics_records,
>>>     rebatch_size=rebatch_size,)

# View the ingested config after add_nn_model in NN Ingestion...
>>> tru.get_nn_user_configs()

Returns:

Type Description
Union[AttributionConfiguration, RNNUserInterfaceConfiguration]

NN user config.

get_num_default_influences(self)

Get the number of influences computed by default of the currently set project.

Returns:

Type Description
int

Number of default influences.

get_num_internal_qii_samples(self)

Get the number of samples used internally in influence computations of the currently set project.

Returns:

Type Description
int

Number of samples to be used internally for influence computations.

get_predictions(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None)

Get the model predictions associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The model predictions associated with the current data-split.

get_projects(self)

Get all projects accessible by current user for the current workspace environment.

get_ranking_k(self)

Gets the ranking k for the current project.

Returns:

Type Description
int

Ranking k.

get_scheduled_ingestion(self, workflow_id)

[Alpha] Get the metadata about a scheduled ingestion from a workflow_id

Parameters:

Name Type Description Default
workflow_id str

The id of the scheduled ingestion workflow.

required

Returns:

Type Description
GetScheduleResponse

Returns an object containing the request_template, schedule, and run_results of a workflow.

get_segment_groups(self)

[Alpha] Get all segment groups associated with the current TruEra workspace, along with their respective segments.

Returns:

Type Description
Mapping[str, Mapping[str, str]]

Mapping of segment group names to the corresponding segment group definition.

get_xs(self, start=0, stop=None, extra_data=False, system_data=False, by_group=False, num_per_group=None)

Get the inputs/data/x-values associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
extra_data bool

Include extra data columns in the response. Defaults to False.

False
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The inputs/data/x-values associated with the current data-split.

get_ys(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None)

Get the targets/y-values associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The targets/y-values associated with the current data-split.

get_ys_pred(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None, wait=True)

Get the model predictions associated with the current data-split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The model predictions associated with the current data-split.

list_monitoring_tables(self)

lists monitoring tables relevant to current project

Returns:

Type Description
str

Returns a json of objects containing the monitoring tables for a project

list_performance_metrics(self)

Lists the available metrics that can be supplied to compute performance, and be set as the project default.

Returns:

Type Description
Sequence[str]

Available metrics.

list_scheduled_ingestions(self, last_key=None, limit=50)

[Alpha] List workflows.

Parameters:

Name Type Description Default
last_key str

The last id to fetch workflows after. Defaults to None.

None
limit int

The number of workflows to fetch. Defaults to 50.

50

Returns:

Type Description
str

Returns a list of objects containing the workflow_id and active state of all workflows.

list_valid_score_types(self) inherited

List the valid score types that can be set for the currently set project.

Returns:

Type Description
Sequence[str]

Valid score types.

register_schema(self, schema)

Register a schema for streaming.

Exceptions:

Type Description
ValueError

Raised on validation error.

schedule_existing_data_split(self, split_name, cron_schedule, override_split_name=None, append=True)

[Alpha] Schedule a new scheduled ingestion based off an existing split.

Parameters:

Name Type Description Default
split_name str

The name of an already materialized split.

required
cron_schedule str

The schedule for the periodic ingestion. Follows cron unix format: ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of the month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday; │ │ │ │ │ 7 is also Sunday on some systems) │ │ │ │ │ │ │ │ │ │


For example, to run a cron on the first of every month: "0 0 1 * *"

required
override_split_name str

The name of the new splits to be created. For information on templating, see the schedule_ingestion function.

None

Returns:

Type Description
str

A workflow_id for looking up the workflow.

schedule_ingestion(self, raw_json, cron_schedule)

[Alpha] Schedule a new scheduled ingestion based off a JSON request tree.

Templating

Templating is supported for uris, split names, and filter expessions for scheduled ingestion. The scheduler passes in several variables and functions when evaluating the supported field, which can be accessed by using the syntax: ${}.

For example, to add the run date to the split name you can add the following suffix when submitting a split name to scheduled ingestion: "split_name_${formatDate("yyyy-MM-dd", now)}"

Supported variables: - now: The Date of the run. - last: The Date of the last run. - lastSuccess: The Date of the last successful run.

Supported functions: - uuid(): Generate a random v4 uuid. - formatDate(, ): Format the given Date as a string. Format follows Java SimpleDateFormat (https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). - epochMillis(): The epoch millis of the given Date.

Parameters:

Name Type Description Default
raw_json str

The JSON string representation of the request tree. To build a json string, you can use the serialize_split function.

required
cron_schedule str

The schedule for the periodic ingestion. Follows cron unix format: ┌───────────── minute (0 - 59) │ ┌───────────── hour (0 - 23) │ │ ┌───────────── day of the month (1 - 31) │ │ │ ┌───────────── month (1 - 12) │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday; │ │ │ │ │ 7 is also Sunday on some systems) │ │ │ │ │ │ │ │ │ │


For example, to run a cron on the first of every month: "0 0 1 * *"

required

Returns:

Type Description
str

A workflow_id for looking up the workflow.

serialize_split(self, split_name, override_split_name=None)

[Alpha] Build a request tree represented as a JSON string.

Templating

Templating is supported for uris, split names, and filter expressions for scheduled ingestion. The scheduler passes in several variables and functions when evaluating the supported field, which can be accessed by using the syntax: ${}.

For example, to add the run date to the split name you can add the following suffix when submitting a split name to scheduled ingestion: "split_name_${formatDate("yyyy-MM-dd", now)}"

Supported variables: - now: The Date of the run. - last: The Date of the last run. - lastSuccess: The Date of the last successful run.

Supported functions: - uuid(): Generate a random v4 uuid. - formatDate(, ): Format the given Date as a string. Format follows Java SimpleDateFormat (https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html). - epochMillis(): The epoch millis of the given Date.

Parameters:

Name Type Description Default
split_name str

The name of an already materialized split.

required
override_split_name str

The optional name to replace the original split name. For information on templating, see the schedule_ingestion function.

None

Returns:

Type Description
str

A JSON string representing the request tree that can be used for scheduled ingestion.

set_as_protected_segment(self, segment_group_name, segment_name)

Sets the provided segment as a "protected" segment. This enables fairness analysis for this segment.

Parameters:

Name Type Description Default
segment_group_name

Name of segment group.

required
segment_name str

Name of segment in provided segment group.

required

Examples:

>>> tru.set_project("Project Name")
>>> tru.set_data_collection("Data Collection Name")

# Add a segment group for Sex
>>> tru.add_segment_group(name = "Sex", segment_definitions = {"Male": "Sex == 'Male'", "Female": "Sex == 'Female'"})

# Set the Female Segment as a Protected Segment
>>> tru.set_as_protected_segment(segment_group_name = "Sex", segment_name = "Female")

set_data_collection(self, data_collection_name)

Set the current data collection to use for all operations in the workspace. This will also unset the current model if it is not associated with the provided data collection.

Parameters:

Name Type Description Default
data_collection_name str

Name of the data_collection. If None, will unset the data collection.

required

Exceptions:

Type Description
ValueError

Raised if no project is associated with the current workspace. Use set_project to set the correct project.

ValueError

Raised if there is no such data_collection in the project.

set_data_split(self, data_split_name)

Set the current data split to use for all operations in the current workspace.

Parameters:

Name Type Description Default
data_split_name str

Name of the data_split. If None, will unset the data split.

required

Exceptions:

Type Description
ValueError

Raised if no project is associated with the current workspace. Use set_project to set the correct project.

ValueError

Raised if no data_collection is associated with the current workspace. Use set_data_collection to set the correct data_collection.

ValueError

Raised if there is no such data_split in the data_collection.

set_default_performance_metrics(self, performance_metrics)

Set the default performance metrics of the currently set project.

Parameters:

Name Type Description Default
performance_metrics Sequence[str]

Performance metrics to use by default.

required

set_influence_type(self, algorithm)

Set the influence algorithm type of the currently set project.

Parameters:

Name Type Description Default
algorithm str

Algorithm type. Options are ["truera-qii", "shap"]. Defaults to "truera-qii" if available, and otherwise "shap".

required

set_influences_background_data_split(self, data_split_name, data_collection_name=None)

Set the background data split used for computing feature influences.

Parameters:

Name Type Description Default
data_split_name str

Name of the data split.

required
data_collection_name Optional[str]

(Optional) Name of the data collection. Defaults to the current data collection in context.

None

set_maximum_model_runner_failure_rate(self, maximum_model_runner_failure_rate)

Sets the maximum model runner failure rate (fraction of points on which the model can fail for a model run to be considered successful) for the current project.

Parameters:

Name Type Description Default
maximum_model_runner_failure_rate float

Maximum failure rate. Must be in [0, 1). By default, it is set to 0.

required

set_model(self, model_name)

Set the current model to use for all operations in the current workspace. This will also change the data collection to the one corresponding to the provided model if different than the priorly set data collection.

Parameters:

Name Type Description Default
model_name str

Name of the model. If None, will unset the model.

required

Exceptions:

Type Description
ValueError

Raised if no project is associated with the current workspace. Use set_project to set the correct project.

ValueError

Raised if there is no such model in the project.

set_model_execution(self, environment)

Set the environment (either local or remote) to execute models in.

Parameters:

Name Type Description Default
environment str

Environment to execute models in. Either "local" or "remote".

required

set_num_default_influences(self, num_default_influences)

Set the number of influences computed by default of the currently set project.

Parameters:

Name Type Description Default
num_default_influences int

Number of influences used by default for most influence-requiring graphs, computations, etc. Note that this will take the first of the provided many from the data split --- therefore, shuffling data splits is generally advised prior to ingestion.

required

set_num_internal_qii_samples(self, num_samples)

Set the number of samples used internally in influence computations of the currently set project.

Parameters:

Name Type Description Default
num_samples int

Number of samples to be used internally for influence computations.

required

set_project(self, project)

Set the current project to use for the current workspace environment. This will unset the rest of the context (data collection, data split, model, etc.) if set prior.

Parameters:

Name Type Description Default
project str

Name of the project.

required

Exceptions:

Type Description
ValueError

Raised if the project does not exist.

set_ranking_k(self, ranking_k)

Sets the ranking k for the current project.

Parameters:

Name Type Description Default
ranking_k int

Must be in >= 0.

required

set_score_type(self, score_type)

Set the score type of the currently set project.

Parameters:

Name Type Description Default
score_type str

Scorer type configuration. Options are ["logits", "probits", "classification", "regression", None]. Defaults to None.

required

update_model_threshold(self, classification_threshold)

Update the classification threshold for the model associated with the TruEra workspace. A model score (probits, logits) that is greater than or equal to the threshold is assigned a positive classification outcome.

Parameters:

Name Type Description Default
classification_threshold float

New threshold to update. Ignored for regression models.

required

update_nn_user_config(self, config)

Update NN user configurations for project and model set in the current context.

Examples:

# If you need to make changes to the attributions config after you have run add_nn_model in NN Ingestion...
>>> from truera.client.nn.client_configs import NLPAttributionConfiguration
>>> updated_attr_config = NLPAttributionConfiguration(
>>>     token_embeddings_layer=token_embeddings_layer_name,
>>>     token_embeddings_anchor=token_embeddings_layer_tensor_anchor,
>>>     n_output_neurons=n_output_neurons,
>>>     n_metrics_records=n_metrics_records,
>>>     rebatch_size=rebatch_size,
>>> )

>>> tru.update_nn_user_config(updated_attr_config)
>>> tru.get_nn_user_configs() # Will return the updated attr_config

Parameters:

Name Type Description Default
config Union[AttributionConfiguration, RNNUserInterfaceConfiguration]

Config to set.

required

verify_nn_wrappers(self, *, clf, attr_config=None, truera_wrappers)

Validates that all wrappers and methods are well formed.

Examples:

# During NN Ingestion you will create two objects
>>> from truera.client.nn.client_configs import NLPAttributionConfiguration
>>> attr_config = NLPAttributionConfiguration(...)

>>> from truera.client.nn.wrappers.autowrap import autowrap
>>> truera_wrappers = autowrap(...) # Use the appropriate NN Diagnostics Ingestion to create this

# Check if ingestion is set up correctly
>>> tru.verify_nn_wrappers(
>>>     clf=model,
>>>     attr_config=attr_config,
>>>     truera_wrappers=truera_wrappers
>>> )

Parameters:

Name Type Description Default
clf NNBackend.Model

The model object.

required
truera_wrappers Optional[base.WrapperCollection]

A collection of wrappers

required

verify_packaged_model(self, model_path)

Locally verifies a packaged Python model by loading the model and, if available, running it on split data ingested into the TruEra system. The model must already be packaged, e.g. via create_packaged_python_model(). The project and data collection for the model must also be set in the current workspace context. This function assumes that it is running an environment with any model dependencies/packages installed.

Parameters:

Name Type Description Default
model_path str

Path to packaged model directory.

required

BasicAuthentication (TrueraAuthentication)

Basic authentication for BaseTrueraWorkspace.

__init__(self, username, password) special

Construct BasicAuth for BaseTrueraWorkspace.

Parameters:

Name Type Description Default
username str

Username for the basic auth credentials.

required
password str

Password for the basic auth credentials.

required

Note: Most users will use TokenAuthentication rather than BasicAuthentication.

Examples:

# import BasicAuthentication and TruEraWorkspace
>>> from truera.client.truera_authentication import BasicAuthentication
>>> from truera.client.truera_workspace import TrueraWorkspace

# Create authentication object
>>> auth = BasicAuthentication(username="My Username", password="My Password")

# Create TruEra Workspace
>>> tru = TrueraWorkspace(connection_string="https://myconnectionstring", authentication=auth)

ServiceAccountAuthentication (TrueraAuthentication)

Service Account Authentication for BaseTrueraWorkspace.

__init__(self, client_id, client_secret, *, token_endpoint=None, additional_payload=None, verify_cert=True) special

Construct ServiceAccountAuth for BaseTrueraWorkspace.

Parameters:

Name Type Description Default
client_id str

Client ID from service account credentials.

required
client_secret str

Client secret from service account credentials.

required
token_endpoint Optional[str]

Optional override of the endpoint to retrieve token from.

None
additional_payload Optional[Mapping[str, str]]

Optional payload to include in request to retrieve token.

None

set_token_endpoint(self, endpoint, append_path=True, overwrite=False)

Set token endpoint. Appends '/oauth/token' to endpoint if append_path is True. Does nothing if token_endpoint is already defined unless overwrite is True.

Token

__str__(self) special

Support implicit usage of this class in str(Token) or f"{Token}".

TokenAuthentication (TrueraAuthentication)

Token authentication for BaseTrueraWorkspace.

__init__(self, token) special

Construct TokenAuth for BaseTrueraWorkspace.

Parameters:

Name Type Description Default
token str

Token containing credentials to authenticate with the services.

required

Note: Most users will use TokenAuthentication rather than BasicAuthentication.

Examples:

# import TokenAuthentication and TruEraWorkspace
>>> from truera.client.truera_authentication import TokenAuthentication
>>> from truera.client.truera_workspace import TrueraWorkspace

# Create authentication object
>>> auth = TokenAuthentication(token="My Token From the TruEra Web App")

# Create TruEra Workspace
>>> tru = TrueraWorkspace(connection_string="https://myconnectionstring", authentication=auth)

Explainer (ABC)

clear_segment(self)

Clears any set segments from all explainer operations.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")
>>> explainer.set_segment("segment_group1", "segment1_in_segment_group1")

# This will return only the xs in "split1" filtered to the segment "segment1_in_segment_group1".
>>> explainer.get_xs()

>>> explainer.clear_segment()

# This will return all the xs in "split1".
>>> explainer.get_xs()

compute_feature_influences(self, start=0, stop=None, score_type=None, system_data=False, by_group=False, num_per_group=None, wait=True)

Compute the QIIs/shapley-values for this explainer's currently set data split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the project setting for "Number of default influences".

None
score_type Optional[str]

The score type to use when computing influences. If None, defaults to score type of project. Defaults to None. For a list of valid score types, see tru.list_valid_score_types.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")

# This will return "model1"'s feature influences on "split1".
>>> explainer.compute_feature_influences()

compute_feature_influences_for_data(self, pre_data, post_data=None, ys=None, score_type=None, comparison_post_data=None, num_internal_qii_samples=1000, algorithm='truera-qii')

Compute the QIIs/shapley-values for the provided data.

Parameters:

Name Type Description Default
pre_data pd.DataFrame

A pandas DataFrame containing the human-readable data for which to compute influences. If post_data is not specified, pre_data is assumed to be both human- and model-readable.

required
post_data Optional[pd.DataFrame]

A pandas DataFrame containing the model-readable post-processed data that is aligned with the pre-processed data. Can be ignored if model-readable pre-processed data is provided. If providing different pre- and post-processed data, be sure the mapping between them adheres to the feature map of the data collection specified during the data collection's creation.

None
ys Optional[Union[np.ndarray, pd.Series]]

Labels for which to compute influences if required by the provided score_type. Defaults to None.

None
score_type Optional[str]

The score type to use when computing influences. If None, defaults to score type of project. Defaults to None. For a list of valid score types, see tru.list_valid_score_types.

None
comparison_post_data Optional[pd.DataFrame]

The comparison data to use when computing influences. If None, defaults to a data split of the data collection of type "all" or "train" and failing that uses the base split currently set in this explainer. Defaults to None.

None
num_internal_qii_samples int

Number of samples used internally in influence computations.

1000
algorithm str

Algorithm to use during computation. Must be one of ["truera-qii", "tree-shap-tree-path-dependent", "tree-shap-interventional", "kernel-shap"]. Defaults to "truera-qii".

'truera-qii'

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")
>>> pre_data = ...
>>> post_data = ...

# This will compute feature influences on `pre_data`/`post_data`. Note that the feature mapping from
# `pre_data` to `post_data` must be the same as that expected by "model1".
>>> explainer.compute_feature_influences_for_data(
        pre_data=pre_data,
        post_data=post_data,
        score_type="probits",
        comparison_post_data=post_data
    )

compute_performance(self)

Computes performance metrics from labels and predictions. To see the list of available metrics, use list_performance_metrics

Examples:

# Set your project, data collection and model
>>> tru.set_project("Project Name")
>>> tru.set_data_collection("data collection name")
>>> tru.set_model("model v1")

# Get the explainer for the base split
>>> explainer = tru.get_explainer("train split name")

# Compute performance for the base split using the explainer object
>>> explainer.compute_performance(metric_type = "MAE", plot = False)

get_base_data_split(self)

Get the base data split used by explainer.

Returns:

Type Description
str

The name of the base data split.

get_comparison_data_splits(self)

Gets the comparison data splits used by the explainer.

Returns:

Type Description
Sequence[str]

The names of the comparison data splits.

get_data_collection(self)

Get the data collection name used by explainer.

get_feature_names(self)

Get the feature names associated with the current data-collection.

Returns:

Type Description
Sequence[str]

Feature names.

get_spline_fitter(self, start=0, stop=None)

Get the spline-fitter using the provided range of points to fit splines.

Parameters:

Name Type Description Default
start int

The lower bound (inclusive) of the index of points to use during spline fitting. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to use during spline fitting. Defaults to the number of Number of default influences for the project.

None

Returns:

Type Description
SplineFitter

Spline-fitter.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
tru.set_model("model1")
explainer = tru.get_explainer("split1")

# Construct a spline fitter and use it to construct a GAM.
sf = explainer.get_spline_fitter("segment_group1", "segment1_in_segment_group1")
gam = sf.construct_gam(
    n_splines=10,
    spline_orders=3,
    constraints={"feature2": "monotonic_inc", "feature5": "monotonic_dec"}
)

# Add GAM model into TruEra.
tru.add_python_model("GAM from model1", gam)

get_xs(self, start=0, stop=None, extra_data=False, system_data=False, by_group=False, num_per_group=None)

Get the inputs/data/x-values.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
extra_data bool

Include extra data columns in the response.

False
system_data bool

Include system data columns (unique ID) in the response.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The inputs/data/x-values.

get_ys(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None)

Get the targets/y-values.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data columns (unique ID) in the response.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The targets/y-values as a pd.DataFrame.

get_ys_pred(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None, wait=True)

Get the model predictions.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The model predictions as a pd.DataFrame.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")

# This will return "model1"'s predictions on "split1".
>>> explainer.get_ys_pred()

list_performance_metrics(self)

Lists the available metrics that can be supplied to compute_performance.

set_base_data_split(self, data_split_name=None)

Set the base data split to use for all operations in the explainer.

Parameters:

Name Type Description Default
data_split_name Optional[str]

Name of the data split.

None

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")

# Gets labels for data-split "split1".
>>> explainer.get_ys()

>>> explainer.set_base_data_split("split2")

# Gets labels for data-split "split2".
>>> explainer.get_ys()

set_comparison_data_splits(self, comparison_data_splits=None, use_all_data_splits=False)

Sets comparison data splits to use for all operations in the explainer.

Parameters:

Name Type Description Default
comparison_data_splits Optional[Sequence[str]]

List of data split names for comparison. This is ignored if use_all_data_split is set to True. (Optional)

None
use_all_data_splits bool

If set to True, set comparison data splits as all of the data splits in the data collection except the base data split. (Optional)

False

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")
>>> explainer = tru.set_comparison_data_splits(["split2", "split3"])

# This will compute AUC metrics for "split1" and compare to "split2" and "split3".
>>> explainer.compute_performance("AUC")

set_segment(self, segment_group_name, segment_name)

Sets and applies a segment filter to all explainer operations.

Parameters:

Name Type Description Default
segment_group_name str

Name of segment group under which the segment is defined.

required
segment_name str

Name of the segment.

required

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")

# This will return all the xs in "split1".
>>> explainer.get_xs()

>>> explainer.set_segment("segment_group1", "segment1_in_segment_group1")

# This will return only the xs in "split1" filtered to the segment "segment1_in_segment_group1".
>>> explainer.get_xs()

NonTabularExplainer (Explainer)

clear_segment(self)

Not Available for NonTabularExplainer

compute_feature_influences(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None, wait=True)

Gets the input influences.

Parameters:

Name Type Description Default
- start (int

A starting offset of the data records

required
- stop (int

A stop offset of the data records

required

Examples:

# During NN Ingestion you will create two objects
>>> from truera.client.nn.client_configs import NLPAttributionConfiguration
>>> attr_config = NLPAttributionConfiguration(...)

>>> from truera.client.nn.wrappers.autowrap import autowrap
>>> truera_wrappers = autowrap(...) # Use the appropriate NN Diagnostics Ingestion to create this

# Check if ingestion is set up correctly
>>> tru.verify_nn_wrappers(
        clf=model,
        attr_config=attr_config,
        truera_wrappers=truera_wrappers
    )

# Add the model and data to the truera workspace
>>> tru.add_nn_model(
        model_name="<model_name>",
        truera_wrappers,
        attr_config
    )
>>> tru.add_nn_data_split(
        data_split_name="<split_name>",
        truera_wrappers,
        split_type="<split_type_train_or_test>"
    )

# Compute influences
>>> tru.compute_feature_influences()

compute_feature_influences_for_data(self)

Not Available for NonTabularExplainer

compute_performance(self) inherited

Computes performance metrics from labels and predictions. To see the list of available metrics, use list_performance_metrics

Examples:

# Set your project, data collection and model
>>> tru.set_project("Project Name")
>>> tru.set_data_collection("data collection name")
>>> tru.set_model("model v1")

# Get the explainer for the base split
>>> explainer = tru.get_explainer("train split name")

# Compute performance for the base split using the explainer object
>>> explainer.compute_performance(metric_type = "MAE", plot = False)

get_base_data_split(self)

Not Available for NonTabularExplainer

get_comparison_data_splits(self)

Not Available for NonTabularExplainer

get_data_collection(self) inherited

Get the data collection name used by explainer.

get_feature_names(self) inherited

Get the feature names associated with the current data-collection.

Returns:

Type Description
Sequence[str]

Feature names.

get_spline_fitter(self, start=0, stop=None)

Not Available for NonTabularExplainer

get_xs(self, start=0, stop=None, extra_data=False, system_data=False, by_group=False, num_per_group=None) inherited

Get the inputs/data/x-values.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
extra_data bool

Include extra data columns in the response.

False
system_data bool

Include system data columns (unique ID) in the response.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The inputs/data/x-values.

get_ys(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None) inherited

Get the targets/y-values.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data columns (unique ID) in the response.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The targets/y-values as a pd.DataFrame.

get_ys_pred(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None, wait=True) inherited

Get the model predictions.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The model predictions as a pd.DataFrame.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")

# This will return "model1"'s predictions on "split1".
>>> explainer.get_ys_pred()

list_performance_metrics(self) inherited

Lists the available metrics that can be supplied to compute_performance.

set_base_data_split(self, data_split_name=None)

Not Available for NonTabularExplainer

set_comparison_data_splits(self)

Not Available for NonTabularExplainer

set_segment(self, segment_group_name=None, segment_name=None)

Not Available for NonTabularExplainer

TabularExplainer (Explainer)

Contains methods to provide explanations for tabular models.

Examples:

# Assuming `tru` is a `TrueraWorkspace` with a tabular project 

# Set your project, data collection and model
>>> tru.set_project("Project Name")
>>> tru.set_data_collection("data collection name")
>>> tru.set_model("model v1")

# Get the explainer for the base split
>>> explainer = tru.get_explainer("train split name")

clear_segment(self) inherited

Clears any set segments from all explainer operations.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")
>>> explainer.set_segment("segment_group1", "segment1_in_segment_group1")

# This will return only the xs in "split1" filtered to the segment "segment1_in_segment_group1".
>>> explainer.get_xs()

>>> explainer.clear_segment()

# This will return all the xs in "split1".
>>> explainer.get_xs()

compute_fairness(self, segment_group, segment1, segment2=None, fairness_type='DISPARATE_IMPACT_RATIO', threshold=None, threshold_score_type=None)

Compares the fairness of outcomes for two segments within a segment group using the provided fairness type.

Parameters:

Name Type Description Default
segment_group str

Name of segment group that the two segments are defined under.

required
segment1 str

Name of first segment (must belong to provided segment group).

required
segment2 Optional[str]

Name of second segment (must belong to provided segment group). If None, then uses the complement of segment1.

None
fairness_type Optional[str]

Name of fairness metric. Must be one of the options returned by list_fairness_metrics.

'DISPARATE_IMPACT_RATIO'
threshold Optional[float]

Optional model threshold for classification models. If None, defaults to pre-configured threshold for the model. Ignored for regression models.

None
threshold_score_type Optional[str]

If threshold is provided, the score type to apply the threshold to (probits or logits). If None, defaults to pre-configured score type for the model. Ignored for regression models.

None

Returns:

Type Description
BiasResult

Computed fairness metric along with information about which group is favored.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.

# Setup the project
>>> project_name = 'test_explainer_compute_fairness'
>>> tru.add_project(project_name, score_type='classification', input_type='tabular')
>>> tru.add_data_collection('dc')

# Create train data where membership 1 is unfairly treated
>>> wealth_max = 5000
>>> rng = np.random.default_rng(seed=42)
>>> wealth = rng.uniform(0.0,wealth_max,size=100)
>>> random_approvals = rng.binomial(1,0.25,size=50)
>>> xs_train = pd.DataFrame({
        "membership": [0]*50 + [1]*50,
        "wealth": wealth,
        "approved": np.concatenate([[1]*50, random_approvals])
    }).astype({
        "membership": "int",
        "wealth": "float",
        "approved": "bool",
        })

# Create data split. 
>>> xs = xs_train.drop(['approved'], axis=1)
>>> labels = xs_train['approved']
>>> tru.add_data_split('train', pre_data = xs, label_data = labels, split_type = "train")

# Create xgb model
>>> import xgboost as xgb
>>> params = {"model_type": "xgb.XGBClassifier", "eta": 0.2, "max_depth": 4}
>>> xgb_clf = xgb.XGBClassifier(eta = params['eta'], max_depth = params['max_depth'])
>>> xgb_clf.fit(xs, labels)

# Add model to project and set model
>>> tru.add_python_model("xgb", xgb_clf, train_split_name="train", train_parameters=params)
>>> tru.set_model("xgb")

# Add a segment group on which to compute fairness
>>> tru.add_segment_group("membership", {"zero": 'membership == 0', 'one': 'membership == 1'})

# Compute fairness across gender and display results.
>>> explainer = tru.get_explainer("train")
>>> explainer.compute_fairness("membership", "one", "zero")

compute_feature_contributors_to_instability(self, score_type=None, use_difference_of_means=False, wait=True)

Compute feature contributors to model instability from the base split to all comparison splits that are set in the current context. By default, instability is measured using Wasserstein Distance.

Parameters:

Name Type Description Default
score_type Optional[str]

The score type to use when computing instability. If None, uses default score type of project. Defaults to None. For a list of valid score types, see tru.list_valid_score_types.

None
use_difference_of_means bool

If True, measures instability with Difference of Means. Defaults to False.

False
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

Table of contributions per feature and comparison split.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")

# Add one or more comparison data split(s) to the current context.
>>> explainer.set_comparison_data_splits(["split2"])

# Show the features contributors to model instability for each comparison split.
>>> explainer.compute_feature_contributors_to_instability()

# Add multiple comparison data splits and re-run to see more output rows
>>> explainer.set_comparison_data_splits(["split2", "split3"])
>>> explainer.compute_feature_contributors_to_instability()

compute_feature_influences(self, start=0, stop=None, score_type=None, system_data=False, by_group=False, num_per_group=None, wait=True) inherited

Compute the QIIs/shapley-values for this explainer's currently set data split.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the project setting for "Number of default influences".

None
score_type Optional[str]

The score type to use when computing influences. If None, defaults to score type of project. Defaults to None. For a list of valid score types, see tru.list_valid_score_types.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")

# This will return "model1"'s feature influences on "split1".
>>> explainer.compute_feature_influences()

compute_feature_influences_for_data(self, pre_data, post_data=None, ys=None, score_type=None, comparison_post_data=None, num_internal_qii_samples=1000, algorithm='truera-qii') inherited

Compute the QIIs/shapley-values for the provided data.

Parameters:

Name Type Description Default
pre_data pd.DataFrame

A pandas DataFrame containing the human-readable data for which to compute influences. If post_data is not specified, pre_data is assumed to be both human- and model-readable.

required
post_data Optional[pd.DataFrame]

A pandas DataFrame containing the model-readable post-processed data that is aligned with the pre-processed data. Can be ignored if model-readable pre-processed data is provided. If providing different pre- and post-processed data, be sure the mapping between them adheres to the feature map of the data collection specified during the data collection's creation.

None
ys Optional[Union[np.ndarray, pd.Series]]

Labels for which to compute influences if required by the provided score_type. Defaults to None.

None
score_type Optional[str]

The score type to use when computing influences. If None, defaults to score type of project. Defaults to None. For a list of valid score types, see tru.list_valid_score_types.

None
comparison_post_data Optional[pd.DataFrame]

The comparison data to use when computing influences. If None, defaults to a data split of the data collection of type "all" or "train" and failing that uses the base split currently set in this explainer. Defaults to None.

None
num_internal_qii_samples int

Number of samples used internally in influence computations.

1000
algorithm str

Algorithm to use during computation. Must be one of ["truera-qii", "tree-shap-tree-path-dependent", "tree-shap-interventional", "kernel-shap"]. Defaults to "truera-qii".

'truera-qii'

Returns:

Type Description
pd.DataFrame

The QIIs/shapley-values.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")
>>> pre_data = ...
>>> post_data = ...

# This will compute feature influences on `pre_data`/`post_data`. Note that the feature mapping from
# `pre_data` to `post_data` must be the same as that expected by "model1".
>>> explainer.compute_feature_influences_for_data(
        pre_data=pre_data,
        post_data=post_data,
        score_type="probits",
        comparison_post_data=post_data
    )

compute_model_score_instability(self, score_type=None, use_difference_of_means=False, plot=False)

Compute model score instability from the base split to all comparison splits that are set in the current context. By default, instability is measured using Wasserstein Distance.

Parameters:

Name Type Description Default
score_type Optional[str]

The score type to use when computing instability. If None, defaults to score type of project. Defaults to None. For a list of valid score types, see tru.list_valid_score_types.

None
use_difference_of_means bool

If True, measures instability with Difference of Means. Defaults to False.

False
plot bool

If True, plots performances for all base and comparison splits in the current context.

False

Returns:

Type Description
Union[float, pd.DataFrame]

The model score instability. If comparison data splits are set, a pd.DataFrame of all splits and their respective score instabilities. Otherwise, a single float metric is returned.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.

# Setup the project
>>> project_name = 'test_explainer'
>>> tru.add_project(project_name, score_type='regression', input_type='tabular')
>>> tru.add_data_collection('dc')

# Create train data. y = 2*x + 1 over range 'x <= 49.0 AND x >= -50.0'
>>> xs_train = pd.DataFrame({
        "x": range(-50, 50)
    })
>>> ys_train = 2 * xs_train["x"] + 1

# Create test data. Add (seeded) random noise to segment 'x <= 49.0 AND x >= 0.0'.
>>> rng = np.random.default_rng(seed=42)
>>> noise = rng.random(50)
>>> xs_test = xs_train.copy()
>>> ys_test = ys_train.copy() + np.concatenate([np.zeros(50), noise])

# Create another split that will produce high instability. y = -2*x + 1
>>> xs_invert = xs_train.copy()
>>> ys_invert = -2 * xs_train["x"] + 1

# Add data splits to project
>>> tru.add_data_split('train', pre_data = xs_train, label_data = ys_train, split_type = "train")
>>> tru.add_data_split('test', pre_data = xs_test, label_data = ys_test, split_type = "test")
>>> tru.add_data_split('invert', pre_data = xs_invert, label_data = ys_invert, split_type = "validate")

# create xgb model
>>> import xgboost as xgb
>>> params = {"model_type": "xgb.XGBRegressor", "eta": 0.2, "max_depth": 4}
>>> xgb_reg = xgb.XGBRegressor(eta = params['eta'], max_depth = params['max_depth'])
>>> xgb_reg.fit(xs_train, ys_train)

# add model to project
>>> tru.add_python_model("xgb", xgb_reg, train_split_name="train", train_parameters=params)

# Create an explainer and set the comparison splits
>>> explainer = tru.get_explainer("train")
>>> explainer.set_comparison_data_splits(["test", "invert"])

# Denote the score_type and call the method
>>> score_type = "mean_absolute_error_for_regression"
>>> explainer.compute_model_score_instability(score_type=score_type)

compute_partial_dependencies(self, wait=True)

Get the partial dependencies for all features. Partial dependencies capture the marginal effect of a feature's value on the predicted outcome of the model.

Parameters:

Name Type Description Default
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
Tuple[Sequence[str], Mapping[str, Sequence], Mapping[str, Sequence]]

The partial dependencies described in a 3-tuple: A list of the features, a mapping from feature to the x-values in a PDP, and a mapping from feature to the y-values in a PDP.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.

# Setup the project
>>> import numpy as np
>>> import pandas as pd
>>> project_name = 'test_explainer_pdp'
>>> tru.add_project(project_name, score_type='regression', input_type='tabular')
>>> tru.add_data_collection('dc')

# Create train data. y = 2*x + 1 over range 'x <= 49.0 AND x >= -50.0'
>>> xs_train = pd.DataFrame({
    "x": range(-50, 50),
    "cat": [0]*50 + [1]*50
})

# For ys, inject noise when x >= 0
>>> noise_gain = 5
>>> rng = np.random.default_rng(seed=42)
>>> noise = noise_gain*rng.random(50)
>>> ys_train = 2 * xs_train["x"] + 1 
>>> ys_train = ys_train + np.concatenate([np.zeros(50), noise])

# Create data splits.
>>> tru.add_data_split('train', pre_data = xs_train, label_data = ys_train, split_type = "train")

# create xgb model
>>> import xgboost as xgb
>>> params = {"model_type": "xgb.XGBRegressor", "eta": 0.2, "max_depth": 4}
>>> xgb_reg = xgb.XGBRegressor(eta = params['eta'], max_depth = params['max_depth'])
>>> xgb_reg.fit(xs_train, ys_train)

# add model to project
>>> tru.add_python_model("xgb", xgb_reg, train_split_name="train", train_parameters=params)

# create explainer, compute partial dependencies (PDs)
>>> explainer = tru.get_explainer("train")
>>> pds = explainer.compute_partial_dependencies()
>>> features, xs, ys = pds

# Plot the PDs manually, accounting for numerical vs. categorical features
>>> import matplotlib.pyplot as plt
>>> for i, feature in enumerate(features):
>>>     plt.figure()
>>>     if i == 0: # numerical feature (x)
>>>         plt.plot(xs[feature], ys[feature])
>>>     if i == 1: # categorical feature (cat)
>>>         plt.bar(xs[feature], ys[feature])
>>>     plt.title(feature)

compute_performance(self, metric_type, plot=False)

Compute performance metric.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
tru.set_model("model1")
explainer = tru.get_explainer("split1")

# This will compute AUC metrics for "split1".
explainer.compute_performance("AUC")

Parameters:

Name Type Description Default
metric_type str

Name of performance metric. Must be one of the options returned by list_performance_metrics.

required
plot bool

If True, plots performances for all base and comparison splits in the current context.

False

Returns:

Type Description
Union[float, pd.DataFrame]

The performance metric. If comparison data splits are set, a pd.DataFrame of all splits and their respective performance. Otherwise, a single float metric is returned.

find_hotspots(self, num_features=1, max_num_responses=3, num_samples=100, metric_of_interest=None, metrics_to_show=None, minimum_size=50, minimum_metric_of_interest_threshold=0, size_exponent=0.25, comparison_data_split_name=None, bootstrapping_fraction=1, random_state=0, show_what_if_performance=False, use_labels=True)

Suggests high error segments for the model for the currently set data split.

Parameters:

Name Type Description Default
num_features int

Number of features to use to describe a high error segment. Defaults to 1.

1
max_num_responses int

Maximum number of high error segments to return. Defaults to 3.

3
num_samples int

Number of samples to use while attempting to find high error segments. The higher the number of samples the slower the computation, but the better the high error segments are generally. Defaults to 100.

100
metric_of_interest Optional[str]

Name specifying how segments are chosen. When None, defaults internally to either 'SEGMENT_GENERALIZED_AUC' or 'MAE' for classification or regression, respectively. Defaults to None.

None
metrics_to_show Optional[Union[str, Sequence[str]]]

Name of performance metric or list of them to include. Must be one of the options returned by list_performance_metrics. Defaults to None.

None
minimum_size int

Minimum size of a segment. Defaults to 50.

50
minimum_metric_of_interest_threshold float

Minimum difference between segment and comparison (i.e. entire split when comparison_data_split_name is not given, and segment on the comparison_data_split_name data-split otherwise). Defaults to 0.

0
size_exponent float

Exponential factor on size of segment. Should be in [0, 1]. A zero value implies the segment size has no effect. Defaults to 0.25.

0.25
comparison_data_split_name Optional[str]

Comparison data-split to use (e.g. train split for overfitting analysis). If set, we look for segments that are far more problematic in the explainer's data split than the comparison one supplied here.

None
bootstrapping_fraction float

Random fraction of points to use for analysis. Should be in (0, 1]. Defaults to 1.

1
random_state int

Random seed for two random processes: 1) selecting the features to analyze and 2) choosing points in bootstrapping. If bootstrapping_fraction < 1, then changing this parameter will introduce more 'randomness' (i.e., change the segment's values for a given feature/set of features). Otherwise, the method will always return the same values for a given feature/set of features. Defaults to 0.

0
show_what_if_performance bool

Whether to show the "what if" performance of the segment, defined as what the overall accuracy on the split would be if the segment's performance were brought up to the accuracy on the whole split. The "what if" version of a metric can only be defined if the metric can be defined per-point and averaged over. Defaults to False.

False
use_labels bool

Whether to use the labels as a feature for segmentation. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

DataFrame describing high error segments. Each row corresponds to a suggested high error segment, with the following columns: 1. segment_definition: The segment definition which can be ingested via the workspace add_segment_group function. 2. size: The number of points in this segment in total in the base data split. In the presence of a comparison split, this will also include ({base_data_split_name}). 3. size (%): The percentage of points in this segment in the base data split. In the presence of a comparison split, this will also include ({base_data_split_name}). There will also be additional columns corresponding to: A. The metric of interest along with a column for each metric in metrics_to_show. B. The "what if" metric corresponding to the metric of interest (if viable) along with each viable "what if" metric in metrics_to_show. Only displayed when show_what_if_performance is True. C. size ({comparison_data_split_name}) and size (%) ({comparison_data_split_name}): The same size and size (%) as above but for the comparison data split. Only displayed when comparison_data_split_name is provided. D. size diff (%): The absolute difference in size (%) between base and comparison data split. Only displayed when metric_of_interest is UNDER_OR_OVERSAMPLING.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
# Setup the project
>>> project_name = 'test_explainer'
>>> tru.add_project(project_name, score_type='regression', input_type='tabular')
>>> tru.add_data_collection('dc')

# Create train data. y = 2*x + 1 over range 'x <= 49.0 AND x >= -50.0'
>>> xs_train = pd.DataFrame({
        "x": range(-50, 50)
    })
>>> ys_train = 2 * xs_train["x"] + 1
# Create test data. Add (seeded) random noise to segment 'x <= 49.0 AND x >= 0.0'.
>>> rng = np.random.default_rng(seed=42)
>>> noise = rng.random(50)
>>> xs_test = xs_train.copy()
>>> ys_test = ys_train.copy() + np.concatenate([np.zeros(50), noise])

# Create data splits.
>>> tru.add_data_split('train', pre_data = xs_train, label_data = ys_train, split_type = "train")
>>> tru.add_data_split('test', pre_data = xs_test, label_data = ys_test, split_type = "test")

# create xgb model
>>> import xgboost as xgb
>>> params = {"model_type": "xgb.XGBRegressor", "eta": 0.2, "max_depth": 4}
>>> xgb_reg = xgb.XGBRegressor(eta = params['eta'], max_depth = params['max_depth'])
>>> xgb_reg.fit(xs_train, ys_train)

# add model to project
>>> tru.add_python_model("xgb", xgb_reg, train_split_name="train", train_parameters=params)

# create explainer and return high_error_segments
>>> explainer = tru.get_explainer("test")
>>> explainer.find_hotspots(
        metric_of_interest="MSE"
    )
# return high_error_segments without labels as segment feature
>>> explainer.find_hotspots(
        metric_of_interest="MSE",
        use_labels=False
    )
# return high_error_segments with multiple segment metrics 
>>> explainer.find_hotspots(
        metric_of_interest="MSE",
        metrics_to_show=["MAE"],
        use_labels=False
    )
# return high_error_segments with comparison split
>>> explainer.find_hotspots(
        metric_of_interest="MSE",
        use_labels=False,
        comparison_data_split_name="train"
    )

get_base_data_split(self) inherited

Get the base data split used by explainer.

Returns:

Type Description
str

The name of the base data split.

get_comparison_data_splits(self) inherited

Gets the comparison data splits used by the explainer.

Returns:

Type Description
Sequence[str]

The names of the comparison data splits.

get_data_collection(self) inherited

Get the data collection name used by explainer.

get_feature_names(self) inherited

Get the feature names associated with the current data-collection.

Returns:

Type Description
Sequence[str]

Feature names.

get_global_feature_importances(self, score_type=None, wait=True)

Get the global feature importances (as measured by QIIs) for this explainer's currently set data split.

Parameters:

Name Type Description Default
score_type Optional[str]

The score type to use when computing influences. If None, defaults to score type of project. Defaults to None. For a list of valid score types, see tru.list_valid_score_types.

None
wait bool

Whether to wait for the job to finish.

True

Returns:

Type Description
pd.DataFrame

The normalized global importances for each feature.

get_spline_fitter(self, start=0, stop=None) inherited

Get the spline-fitter using the provided range of points to fit splines.

Parameters:

Name Type Description Default
start int

The lower bound (inclusive) of the index of points to use during spline fitting. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to use during spline fitting. Defaults to the number of Number of default influences for the project.

None

Returns:

Type Description
SplineFitter

Spline-fitter.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
tru.set_model("model1")
explainer = tru.get_explainer("split1")

# Construct a spline fitter and use it to construct a GAM.
sf = explainer.get_spline_fitter("segment_group1", "segment1_in_segment_group1")
gam = sf.construct_gam(
    n_splines=10,
    spline_orders=3,
    constraints={"feature2": "monotonic_inc", "feature5": "monotonic_dec"}
)

# Add GAM model into TruEra.
tru.add_python_model("GAM from model1", gam)

get_xs(self, start=0, stop=None, extra_data=False, system_data=False, by_group=False, num_per_group=None) inherited

Get the inputs/data/x-values.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
extra_data bool

Include extra data columns in the response.

False
system_data bool

Include system data columns (unique ID) in the response.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The inputs/data/x-values.

get_ys(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None) inherited

Get the targets/y-values.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data columns (unique ID) in the response.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None

Returns:

Type Description
pd.DataFrame

The targets/y-values as a pd.DataFrame.

get_ys_pred(self, start=0, stop=None, system_data=False, by_group=False, num_per_group=None, wait=True) inherited

Get the model predictions.

Parameters:

Name Type Description Default
start Optional[int]

The lower bound (inclusive) of the index of points to include. Defaults to 0.

0
stop Optional[int]

The upper bound (exclusive) of the index of points to include. Defaults to None which is interpreted as the total number of points for local projects and the setting for "Number of default influences" for remote projects.

None
system_data bool

Include system data (e.g. timestamps) if available in response. Defaults to False.

False
by_group bool

For ranking projects, whether to retrieve data by group or not. Ignored for non-ranking projects. Defaults to False.

False
num_per_group Optional[int]

For ranking projects and when by_group is True, the number of points per group to return.

None
wait bool

Whether to wait for the job to finish. Defaults to True.

True

Returns:

Type Description
pd.DataFrame

The model predictions as a pd.DataFrame.

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> tru.set_model("model1")
>>> explainer = tru.get_explainer("split1")

# This will return "model1"'s predictions on "split1".
>>> explainer.get_ys_pred()

list_performance_metrics(self)

Lists all available performance metrics.

Returns:

Type Description
Sequence[str]

List of performance metric names, which can be provided to compute_performance.

plot_isp(self, feature, num=None, figsize=(700, 500), xlim=None)

Plot the influence sensitivity plot (ISP) of a specific feature.

Parameters:

Name Type Description Default
feature str

Feature to plot the ISP of.

required
num Optional[int]

Number of points to plot. Defaults to None, which is equivalent to a standard number of points used for calculations.

None
figsize Optional[Tuple[int, int]]

Size for plot in pixels. Defaults to (700, 500).

(700, 500)
xlim Optional[Tuple[int, int]]

Range for x-axis. Defaults to None, which scales to the size of the data.

None

plot_isps(self, features=None, num=None, figsize=(700, 500))

Plot the influence sensitivity plot (ISP) of a set of features.

Parameters:

Name Type Description Default
features Optional[Sequence[str]]

Features to plot the ISP of. Defaults to None, which is all features.

None
num Optional[int]

Number of points to plot. Defaults to None, which is equivalent to a standard number of points used for calculations.

None
figsize Optional[Tuple[int, int]]

Size for plot. Defaults to (21, 6).

(700, 500)

plot_pdp(self, feature, figsize=(700, 500), xlim=None)

DEPRECATED: Plot the partial dependence plot (PDP) of a specific feature.

Parameters:

Name Type Description Default
feature str

Feature to plot the PDP of.

required
figsize Optional[Tuple[int, int]]

Size for plot. Defaults to (21, 6).

(700, 500)
xlim Optional[Tuple[int, int]]

Range for x-axis. Defaults to None, which scales to the size of the data.

None

plot_pdps(self, features=None, figsize=(700, 500))

DEPRECATED: Plot the partial dependence plot (PDP) of a set of features.

Parameters:

Name Type Description Default
features Optional[Sequence[str]]

Features to plot the PDP of. Defaults to None, which is all features.

None
figsize Optional[Tuple[int, int]]

Size for plot. Defaults to (21, 6).

(700, 500)

rank_performance(self, metric_type, ascending=False)

Rank performance of all models in the data collection on the explainer's base data split. If comparison data splits are set, will also show performance of the models on them.

Parameters:

Name Type Description Default
metric_type str

Name of performance metric. Must be one of the options returned by list_performance_metrics.

required
ascending bool

If True, sort the results in ascending order. Defaults to False.

False

Returns:

Type Description
pd.DataFrame

The performance score of all models in the data collection.

set_base_data_split(self, data_split_name=None) inherited

Set the base data split to use for all operations in the explainer.

Parameters:

Name Type Description Default
data_split_name Optional[str]

Name of the data split.

None

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")

# Gets labels for data-split "split1".
>>> explainer.get_ys()

>>> explainer.set_base_data_split("split2")

# Gets labels for data-split "split2".
>>> explainer.get_ys()

set_comparison_data_splits(self, comparison_data_splits=None, use_all_data_splits=False) inherited

Sets comparison data splits to use for all operations in the explainer.

Parameters:

Name Type Description Default
comparison_data_splits Optional[Sequence[str]]

List of data split names for comparison. This is ignored if use_all_data_split is set to True. (Optional)

None
use_all_data_splits bool

If set to True, set comparison data splits as all of the data splits in the data collection except the base data split. (Optional)

False

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")
>>> explainer = tru.set_comparison_data_splits(["split2", "split3"])

# This will compute AUC metrics for "split1" and compare to "split2" and "split3".
>>> explainer.compute_performance("AUC")

set_segment(self, segment_group_name, segment_name) inherited

Sets and applies a segment filter to all explainer operations.

Parameters:

Name Type Description Default
segment_group_name str

Name of segment group under which the segment is defined.

required
segment_name str

Name of the segment.

required

Examples:

# Assuming `tru` is a `TrueraWorkspace`.
>>> explainer = tru.get_explainer("split1")

# This will return all the xs in "split1".
>>> explainer.get_xs()

>>> explainer.set_segment("segment_group1", "segment1_in_segment_group1")

# This will return only the xs in "split1" filtered to the segment "segment1_in_segment_group1".
>>> explainer.get_xs()

IngestionClient

Client for ingesting data from a variety of sources into the TruEra product.

add_credential(self, name, secret, identity=None, is_aws_iam_role=False)

Add a new credential to TruEra product. The credential is saved in a secure manner and is used to authenticate with the data source to be able to perform various operations (read, filter, sample etc.).

Parameters:

Name Type Description Default
name str

Friendly name of the credential.

required
secret str

The secret to be stored.

required
identity str

Identity portion of the secret. Not needed in all cases. Defaults to None.

None

Returns:

Type Description
Credential

Returns an object with the credential name and id. The secret is not stored in this object.

Examples:

>>> ACCESS_KEY = "access_key"
>>> SECRET_KEY = "asdf1234asdf1234"
>>> ingestion_client.add_credential(
        name="credential_1",
        secret=SECRET_KEY,
        identity=ACCESS_KEY
    )

add_data_source(self, name, uri, credentials=None, **kwargs)

Add a new data source in the system.

Parameters:

Name Type Description Default
name str

Friendly name of the data source.

required
uri str

URI describing the location of the data source. For local files this can be file:///path/to/my/file or /path/to/my/file For files stored in Azure Storage Blobs the expected path is wasb://container@account.blob.core.windows.net/blob For files stored in S3 Buckets the expected path is s3://bucket-name/file For mysql connections the expected path is mysql://database-endpoint.com:port For hive connections the expected path is hive2://database-endpoint.com:port For jdbc connections the expected path is jdbc:://database-endpoint.com:port For bigquery connections, the user is expected to pass a psuedo-uri 'bigquery:'. The connection is embedded in the json service acc key.

required
credentials Credential

Provide the credential object if the data source requires authentication to read from it. Defaults to None.

None
**format str

The format in which the file (local) or blob (AWS S3, Azure WASB etc.) are stored in. Supported formats: CSV and Parquet.

required
**column_schema Union[str, List[Tuple[str, str]]]

For providing a schema that should be respected by the data source. This can be provided in the form of a path to a JSON/YAML file containing the schema, or a list columns each represented as a tuple

required
**first_row_is_header bool

For text based delimited files (csv, tsv etc.), indicates if the first row provides header information. Defaults to True.

required
**column_delimiter str

For text based delimited files (csv, tsv etc.), provides the delimiter to separate column values. Defaults to ','.

required
**quote_character str

For text based delimited files (csv, tsv etc.), if quotes are used provide the quote character. Defaults to '"'.

required
**null_value str

For text based delimited files (csv, tsv etc.), the string that signifies null value. Defaults to 'null'.

required
**empty_value str

For text based delimited files (csv, tsv etc.), the string that signifies empty value. Defaults to '""'.

required
**date_format str

For text based delimited files (csv, tsv etc.), if any column has date time, provide the format string. Defaults to 'yyyy-MM-dd HH🇲🇲ssZZ'.

required
**account_key str

For reading from Azure Storage Blob (WASB), provide the account_key to be used to read the blob. Not required if credential object is provided.

required
**access_key_id str

For reading from a s3 bucket, provide the access key id to be used to read the blob. Not required if credential object is provided or the underlying deployment is in a role with access to the bucket.

required
**secret_access_key str

For reading from a s3 bucket, provide the secret access key to be used to read the blob. Not required if credential object is provided or the underlying deployment is in a role with access to the bucket.

required
**database_name str

For reading from MySQL database, provide the database name to use. Required for MySQL data source.

required
**table_name str

For reading from MySQL database, provide the table name to use. Required for MySQL data source.

required

Returns:

Type Description
Table

Returns a Table object which allows interaction with the attached data.

Examples:

# Adding a local file
>>> table = ingestion_client.add_data_source(
        name="local_data_1",
        uri="path/to/data.parquet"
    )

# Adding a data source from S3
>>> credentials = ingestion_client.add_credential(
        name="s3_credential", secret="...", identity="..."
    )
>>> table = ingestion_client.add_data_source(
        name="s3_data_1",
        uri="s3://some-data-bucket/data.parquet",
        credentials=credentials
    )

delete_credential(self, name)

Delete a credential in the TruEra product.

Parameters:

Name Type Description Default
name str

Friendly name of the credential.

required

Examples:

>>> ingestion_client.delete_credential("credential_1")

get_credential(self, name)

Get metadata about a credential in the TruEra product. Response does not contain the credential itself.

Parameters:

Name Type Description Default
name str

Friendly name of the credential.

required

Returns:

Type Description
dict

Dictionary containing metadata describing that credential.

Examples:

>>> credential_metadata = ingestion_client.get_credential("credential_1")

get_data_source(self, name)

Get a data source that was already created in the system.

Parameters:

Name Type Description Default
name str

The friendly name of the data source.

required

Returns:

Type Description
Table

Returns a Table object which allows interaction with the attached data.

Examples:

>>> table = ingestion_client.get_data_source("table1")

update_credential(self, name, secret, identity=None)

Update the identity and/or secret of an existing credential.

Parameters:

Name Type Description Default
name str

Friendly name of the credential.

required
secret str

The secret to be stored.

required
identity str

Identity portion of the secret. Not needed in all cases. Defaults to None.

None

Returns:

Type Description
Credential

Returns an object with the credential name and id. The secret is not stored in this object.

Examples:

>>> ACCESS_KEY = "access_key"
>>> SECRET_KEY = "asdf1234asdf1234"
>>> ingestion_client.add_credential(
        name="credential_1",
        secret=SECRET_KEY,
        identity=ACCESS_KEY
    )
>>> ingestion_client.update_credentials(
        name="credential_1",
        secret="new_secret_1234",
        identity="new_identity"
    )

Table

add_data_split(self, data_split_name, data_split_type, label_col_name=None, id_col_name=None, sample_count=5000, sample_kind='random', *, seed=None, prediction_col_name=None, pre_data_additional_skip_cols=None, model_name=None, wait=True, timeout_seconds=300, timestamp_col_name=None, score_type=None, train_baseline_model=False, **kwargs)

Ingest the Table as a split in TruEra to use in analytics.

Parameters:

Name Type Description Default
data_split_name str

Name of the data split.

required
data_split_type str

Type of the data split, options are ['all', 'train', 'test', 'validate', 'oot', 'custom']

required
label_col_name str

Name of the label/ground truth/target column in the table.

None
sample_count int

Maximum rows to use when creating the split. Defaults to 5000.

5000
sample_kind str

Specifies the strategy to use while sub-sampling the rows. Defaults to "random".

'random'
wait bool

Whether to wait for the TruEra service to complete creating the data split. Defaults to True.

True
timeout_seconds int

Timeout used when wait is set to True. Defaults to 300.

300

Returns:

Type Description
Mapping[str, str]

Returns a dictionary with data_split_name, operation_id and status of the operation.

add_extra_data(self, data_split_name, extras_col_names, id_col_name, sample_count=5000, sample_kind='random', *, timestamp_col_name=None, seed=None, wait=True, timeout_seconds=300, **kwargs)

Upload extra_data from Table to an existing split in TruEra.

Parameters:

Name Type Description Default
data_split_name str

Name of the existing data split.

required
extras_col_names Union[str, Sequence[str]]

Name(s) of the extra data column(s) in the table.

required
id_col_name str

Name of the id column used to match extra data with the corresponding data points.

required
sample_count int

Maximum rows to use when creating the split. Defaults to 5000.

5000
sample_kind str

Specifies the strategy to use while sub-sampling the rows. One of ["random", "first"].

'random'
timestamp_col_name Optional[str]

Name of the timestamp column of the extra data.

None
wait bool

Whether to wait for the TruEra service to complete creating the data split. Defaults to True.

True
timeout_seconds int

Timeout used when wait is set to True. Defaults to 300.

300

Returns:

Type Description
Mapping[str, str]

Returns a dictionary with data_split_name, operation_id and status of the operation.

add_feature_influences(self, data_split_name, feature_influence_col_names, id_col_name, model_name, *, background_split_name=None, timestamp_col_name=None, score_type=None, wait=True, timeout_seconds=300, influence_type=None, **kwargs)

Upload feature influences from Table to an existing split/model in TruEra.

Parameters:

Name Type Description Default
data_split_name str

Name of the existing data split.

required
feature_influence_col_names Sequence[str]

Name of the feature influence columns in the table.

required
id_col_name str

Name of the id column used to match predictions with the corresponding data points.

required
model_name str

Name of the model for which feature influences are computed.

required
background_split_name Optional[str]

Split name that contains the background of feature influence computation. If not provided, defaults to the default background data split of the given data collection.

None
timestamp_col_name Optional[str]

Name of the timestamp column of the predictions.

None
score_type Optional[str]

String name of score type (QoI) for prediction column.

None
wait bool

Whether to wait for the TruEra service to complete creating the data split. Defaults to True.

True
timeout_seconds int

Timeout used when wait is set to True. Defaults to 300.

300
influence_type Optional[str]

Influence algorithm used to generate influences. If influence type of project is set to "truera-qii", assumes that explanations are generated using truera-qii. If influence type of project is set to "shap", then influence_type must be passed in as one of ["tree-shap-tree-path-dependent", "tree-shap-interventional", "kernel-shap"].

None

Returns:

Type Description
Mapping[str, str]

Returns a dictionary with data_split_name, operation_id and status of the operation.

add_labels(self, data_split_name, label_col_name, id_col_name, sample_count=5000, sample_kind='random', *, timestamp_col_name=None, seed=None, wait=True, timeout_seconds=300, model_name=None, **kwargs)

Upload labels from Table to an existing split in TruEra.

Parameters:

Name Type Description Default
data_split_name str

Name of the existing data split.

required
label_col_name str

Name of the label/ground truth/target column in the table.

required
id_col_name str

Name of the id column used to match labels with the corresponding data points.

required
sample_count int

Maximum rows to use when creating the split. Defaults to 5000.

5000
sample_kind str

Specifies the strategy to use while sub-sampling the rows. One of ["random", "first"].

'random'
timestamp_col_name Optional[str]

Name of the timestamp column of the labels.

None
wait bool

Whether to wait for the TruEra service to complete creating the data split. Defaults to True.

True
timeout_seconds int

Timeout used when wait is set to True. Defaults to 300.

300

Returns:

Type Description
Mapping[str, str]

Returns a dictionary with data_split_name, operation_id and status of the operation.

add_predictions(self, data_split_name, prediction_col_name, id_col_name, model_name, *, timestamp_col_name=None, score_type=None, wait=True, timeout_seconds=300, **kwargs)

Upload predictions from Table to an existing split/model in TruEra.

Parameters:

Name Type Description Default
data_split_name str

Name of the existing data split.

required
prediction_col_name str

Name of the prediction column in the table.

required
id_col_name str

Name of the id column used to match predictions with the corresponding data points.

required
timestamp_col_name Optional[str]

Name of the timestamp column of the predictions.

None
score_type Optional[str]

String name of score type (QoI) for prediction column.

None
wait bool

Whether to wait for the TruEra service to complete creating the data split. Defaults to True.

True
timeout_seconds int

Timeout used when wait is set to True. Defaults to 300.

300

Returns:

Type Description
Mapping[str, str]

Returns a dictionary with data_split_name, operation_id and status of the operation.

append_to_data_split(self, data_split_name, id_col_name, *, sample_count=5000, sample_kind='random', seed=None, label_col_name=None, prediction_col_name=None, wait=True, timeout_seconds=300, timestamp_col_name=None, model_name=None, score_type=None, **kwargs)

Ingest the Table into an existing split in TruEra to use in analytics. All columns of the data frame will be ingested into pre_data except for label / prediction columns if specified.

Parameters:

Name Type Description Default
data_split_name str

Name of the data split.

required
id_col_name str

Name of the column to use for unique ID.

required
sample_count int

Maximum rows to use when creating the split. Defaults to 5000.

5000
sample_kind str

Specifies the strategy to use while sub-sampling the rows. Options are "random" and "first". It is not recommended to use "first" as it may result in a non-uniform sampling. Defaults to "random".

'random'
label_col_name str

Name of the label/ground truth/target column in the table.

None
prediction_col_name str

Name of the prediction column in the table.

None
timestamp_col_name str

Name of the timestamp column (if using).

None
seed int

Seed for reproducing the same ingestion - defaults to a random seed.

None
wait bool

Whether to wait for the TruEra service to complete creating the data split. Defaults to True.

True
timeout_seconds int

Timeout used when wait is set to True. Defaults to 300.

300

Returns:

Type Description
Mapping[str, str]

Returns a dictionary with data_split_name, operation_id and status of the operation.

filter(self, expression)

[Alpha] Filter a table by providing a boolean expression.

Parameters:

Name Type Description Default
expression str

The expression to filter the table. Simple SQL expressions are supported: = : filters for equality, ex: amount = 1000 != : filters for inequality, ex: amount != 1000 < : filters for less-than, ex: amount < 1000 <= : filters for less-than-or-equal, ex: amount <= 1000 > : filters for greater-than, ex: amount > 1000 >= : filters for great-than-or-equal, ex: amount >= 1000 NOT: filters records if the inner condition is not true, ex: NOT(amount >= 1000) AND: filters records if both the conditions are true, ex: (amount >= 1000) AND (state = 'WA') OR : filters records if any of the two conditions is true, ex: (amount >= 1000) OR (state = 'WA') String literals should be within quotes (''), numeric literals should not have quotes (''). Left side of a binary expression should be a column name (without quotes), right side of an expression should be a literal. For example, (amount < salary) is not a valid expression, as both left and right side of the expression are column-names.

required

Exceptions:

Type Description
ValueError

Raised if provided expression is None or empty.

Returns:

Type Description
Table

Returns a table which points to the filtered rows.

get_sample_rows(self, count=10, *, wait=True, timeout_seconds=300)

Get sampled rows from the table.

Parameters:

Name Type Description Default
count int

The number of rows to sample. Maximum allowed is 2000. Defaults to 100.

10
wait bool

If set to true, the client will wait until timeout to get data from the service. This is useful when the data is still being fetched or filtered. Defaults to True.

True
timeout_seconds int

Timeout used when wait is set to True. Defaults to 300.

300

Returns:

Type Description
pd.DataFrame

Returns a pandas DataFrame containing the sampled rows.

merge_dataframes_and_create_column_spec(id_col_name, timestamp_col_name=None, pre_data=None, post_data=None, predictions=None, labels=None, extra_data=None, feature_influences=None, feature_influence_suffix='_truera-qii_influence')

Helper function to merge multiple DataFrames into one and generate a ColumnSpec

Parameters:

Name Type Description Default
id_col_name str

Id column name.

required
timestamp_col_name Optional[str]

Timestamp column name. Defaults to None.

None
pre_data Optional[pd.DataFrame]

DataFrame corresponding to pre data. Defaults to None.

None
post_data Optional[pd.DataFrame]

DataFrame corresponding to post data. Defaults to None.

None
predictions Optional[pd.DataFrame]

DataFrame corresponding to predictions. Defaults to None.

None
labels Optional[pd.DataFrame]

DataFrame corresponding to labels. Defaults to None.

None
extra_data Optional[pd.DataFrame]

DataFrame corresponding to extra data. Defaults to None.

None
feature_influences Optional[pd.DataFrame]

DataFrame correspondnig to feature influences. Defaults to None.

None
feature_influence_suffix Optional[str ]

Suffix to append to column names of feature influences in order to prevent duplicate name issues. Defaults to FEATURE_INFLUENCE_SUFFIX_TRUERA_QII.

'_truera-qii_influence'

Returns:

Type Description
Tuple[pd.DataFrame, ColumnSpec]

A tuple consisting of the merged DataFrame and corresponding ColumnSpec

Segment

apply(self, data)

Applies the filter associated with this segment to the provided data.

Parameters:

Name Type Description Default
data DataFrame

Data to apply segment on.

required

Returns:

Type Description
np.ndarray

Boolean array of same length of data, indicating whether each row meets the filter requirements

pretty_print(self)

Print out the filter associated with this segment.

SegmentGroup

get_segments(self)

Returns all segments present in the given segment group.

Returns:

Type Description
Mapping[str, Segment]

Map from each segment name to its corresponding Segment object.

pretty_print(self)

Print out this segmentation and its associated segments.

SplineFitter

__init__(self, xs, ys, qiis) special

Construct a spline fitter.

Parameters:

Name Type Description Default
xs pd.DataFrame

x-values.

required
ys Union[pd.DataFrame, np.ndarray]

y-values.

required
qiis pd.DataFrame

QII/influences/shapley-values.

required

construct_gam(self, n_splines=10, spline_orders=3, constraints=None, outer_model=LogisticRegression())

Construct a GAM based off QII splines.

Parameters:

Name Type Description Default
n_splines int

Number of splines per feature. Defaults to 10.

10
spline_orders int

Order of splines. Defaults to 3.

3
constraints Union[str, Mapping[str, str]]

If a single str, then the constraints for all features. Otherwise a dict from feature to constraints. All constraints must be one of [None, "monotonic_inc", "monotonic_dec"]. Defaults to None.

None
outer_model Any

Model to combine splines. This must work with sklearn.pipeline.Pipeline. Defaults to LogisticRegression().

LogisticRegression()

Returns:

Type Description
Pipeline

GAM model.

fit_spline(self, feature, n_splines=10, spline_order=1, constraints=None)

Compute a spline for a single feature.

Parameters:

Name Type Description Default
feature str

Feature to compute splines for.

required
n_splines int

Number of splines per feature. Defaults to 10.

10
spline_order int

Order of splines. Defaults to 3.

1
constraints str

The constraints for the spline --- must be one of [None, "monotonic_inc", "monotonic_dec"]. Defaults to None.

None

Returns:

Type Description
Callable[[Union[pd.Series, np.ndarray]], np.ndarray]

Spline.

plot_isp(self, feature, figsize=(21, 6))

Plot the influence sensitivity plot (ISP) of a specific feature along with the associated spline.

Parameters:

Name Type Description Default
feature str

Feature to plot the ISP of.

required
figsize Tuple[int, int]

Size for plot. Defaults to (21, 6).

(21, 6)

plot_isps(self, features=None, figsize=(21, 6))

Plot the influence sensitivity plot (ISP) of a set of features along with the associated spline.

Parameters:

Name Type Description Default
features Sequence[str]

Features to plot the ISP of. Defaults to None, which is all features.

None
figsize Tuple[int, int]

Size for plot. Defaults to (21, 6).

(21, 6)

Tester (ABC)

add_fairness_test(self, test_name, metric='DISPARATE_IMPACT_RATIO', *, data_split_names=None, data_split_name_regex=None, all_protected_segments=False, protected_segments=None, comparison_segments=None, data_collection_names=None, all_data_collections=False, warn_if_less_than=None, warn_if_greater_than=None, warn_if_within=None, warn_if_outside=None, fail_if_less_than=None, fail_if_greater_than=None, fail_if_within=None, fail_if_outside=None, description=None, overwrite=False)

Add a fairness test to the current data collection in context. To set warning condition, please provide one of [warn_if_less_than, warn_if_greater_than, warn_if_within, warn_if_outside]. Similarly, to set fail condition please provide one of [fail_if_less_than, fail_if_greater_than, fail_if_within, fail_if_outside].

Parameters:

Name Type Description Default
test_name str

The name of the test.

required
metric str

Fairness metric for the test. Must be one of the options returned by list_fairness_metrics.

'DISPARATE_IMPACT_RATIO'
data_split_names Optional[Sequence[str]]

List of the data splits that we want to use for the test.

None
all_protected_segments bool

If set to True, the test will apply to all protected segments (including future ones). Defaults to False which means the test will only apply to the protected segments specified in protected_segments.

False
protected_segments Optional[Sequence[Tuple[str, str]]]

List of (segment_group_name, segment_name) tuple to be used as protected segments.

None
comparison_segments Optional[Sequence[Tuple[str, str]]]

List of (segment_group_name, segment_name) tuple to be used as comparison segments. Defaults to None which means the complement of the protected segment.

None
data_collection_names Optional[Sequence[str]]

List of the data collections for the test. If not specified, the test will only apply to the current data collection in context.

None
all_data_collections bool

If set to True, the test will apply to all data collections (including future ones). Defaults to False which means the test will only apply to the current data collection in context (if data_collection_names is not specified).

False
warn_if_less_than Optional[float]

Warn if score is less than the value specified in this argument.

None
warn_if_greater_than Optional[float]

Warn if score is greater than the value specified in this argument.

None
warn_if_within Optional[Tuple[float, float]]

Warn if value[0] < score < value[1].

None
warn_if_outside Optional[Tuple[float, float]]

Warn if score < value[0] OR score > value[1].

None
fail_if_less_than Optional[float]

Fail if score is less than the value specified in this argument.

None
fail_if_greater_than Optional[float]

Fail if score is greater than the value specified in this argument.

None
fail_if_within Optional[Tuple[float, float]]

Fail if value[0] < score < value[1].

None
fail_if_outside Optional[Tuple[float, float]]

Fail if score < value[0] OR score > value[1].

None
description Optional[str]

Text description of the test.

None
overwrite bool

If set to True, will overwrite the thresholds for existing test specified under the given data_split_name, segment, and metric. Defaults to False.

False

Examples:

# Explicitly specifying comparison segment
>>> tru.tester.add_fairness_test(
    test_name="Fairness Test",
    data_split_names=["split1_name", "split2_name"],
    protected_segments=[("segment_group_name", "protected_segment_name")],
    comparison_segments=[("segment_group_name", "comparison_segment_name")],
    comparison_segment_name=<comparison segment name>,
    metric="DISPARATE_IMPACT_RATIO",
    warn_if_outside=[0.8, 1.25],
    fail_if_outside=[0.5, 2]
)

# Not specifying comparison segment means the comparison segment is the complement of protected segment
# will be used as comparison
>>> tru.tester.add_fairness_test(
    test_name="Fairness Test",
    data_split_names=["split1_name", "split2_name"],
    protected_segments=[("segment_group_name", "protected_segment_name")],
    metric="DISPARATE_IMPACT_RATIO",
    warn_if_outside=[0.9, 1.15],
    fail_if_outside=[0.8, 1.25]
)

add_feature_importance_test(self, test_name, *, data_split_names=None, data_split_name_regex=None, min_importance_value, background_split_name=None, score_type=None, segments=None, data_collection_names=None, warn_if_greater_than=None, fail_if_greater_than=None, description=None, overwrite=False)

Add a feature importance test to the current data collection in context. To set warning condition, please provide warn_if_greater_than. Similarly, to set fail condition please provide fail_if_greater_than.

Parameters:

Name Type Description Default
test_name str

The name of the test.

required
data_split_names Optional[Sequence[str]]

List of the data splits that we want to use for the test.

None
data_split_name_regex Optional[str]

Regex of the data split name that we want to use for the test (future data splits that match this naming pattern will automatically be included in the test).

None
min_importance_value float

Minimum global importance value of a feature.

required
background_split_name Optional[str]

The name of the data split to be used as background data for computing feature influences. If None, this value will be inferred from the project settings. Defaults to None.

None
score_type Optional[str]

The score type to use when computing influences. If None, this value will be inferred from the project settings. Defaults to None. For a list of valid score types, see list_valid_score_types.

None
segments Optional[Tuple[str, str]]

List of (segment_group_name, segment_name) tuple to be used as filter. A test will be defined on each of the given segment filters.

None
warn_if_greater_than Optional[float]

Warn if more than this number of features have global importance values lower than min_importance_value.

None
fail_if_greater_than Optional[float]

Fail if more than this number of features have global importance values lower than min_importance_value.

None
description Optional[str]

Text description of the test.

None
overwrite bool

If set to True, will overwrite the thresholds for existing test specified under the given data_split_name and segment. Defaults to False.

False

Examples:

>>> tru.tester.add_feature_importance_test(
    test_name="Feature Importance Test",
    data_split_names=["split1_name", "split2_name"],
    min_importance_value=0.01,
    background_split_name="background split name",
    score_type=<score_type>, # (e.g., "regression", or "logits"/"probits"
                            # for the classification project)
    warn_if_greater_than=5, # warn if number of features with global importance values lower than `min_importance_value` is > 5
    fail_if_greater_than=10
)

add_performance_test(self, test_name, metric, *, data_split_names=None, data_split_name_regex=None, data_collection_names=None, all_data_collections=False, segments=None, warn_if_less_than=None, warn_if_greater_than=None, warn_if_within=None, warn_if_outside=None, warn_threshold_type='ABSOLUTE', fail_if_less_than=None, fail_if_greater_than=None, fail_if_within=None, fail_if_outside=None, fail_threshold_type='ABSOLUTE', reference_split_name=None, reference_model_name=None, description=None, overwrite=False)

Add a performance test group to the current data collection in context. To set warning condition, please provide one of [warn_if_less_than, warn_if_greater_than, warn_if_within, warn_if_outside]. Similarly, to set fail condition please provide one of [fail_if_less_than, fail_if_greater_than, fail_if_within, fail_if_outside].

Parameters:

Name Type Description Default
test_name str

The name of the test.

required
metric str

Performance metric for the test. Must be one of the options returned by list_performance_metrics.

required
data_split_names Optional[Sequence[str]]

List of the data splits that we want to use for the test.

None
data_split_name_regex Optional[str]

Regex of the data split name that we want to use for the test (future data splits that match this naming pattern will automatically be included in the test).

None
data_collection_names Optional[Sequence[str]]

List of the data collections for the test. If not specified, the test will only apply to the current data collection in context.

None
all_data_collections bool

If set to True, the test will apply to all data collections (including future ones). Defaults to False which means the test will only apply to the current data collection in context (if data_collection_names is not specified).

False
segments Optional[Sequence[Tuple[str, str]]]

List of (segment_group_name, segment_name) tuple to be used as filter. A test will be defined on each of the given segment filters.

None
warn_if_less_than Optional[float]

Warn if score is less than the value specified in this argument.

None
warn_if_greater_than Optional[float]

Warn if score is greater than the value specified in this argument.

None
warn_if_within Optional[Tuple[float, float]]

Warn if value[0] < score < value[1].

None
warn_if_outside Optional[Tuple[float, float]]

Warn if score < value[0] OR score > value[1].

None
warn_threshold_type str

Must be one of ["ABSOLUTE", "RELATIVE"]. Describe whether the warning threshold is defined as absolute value or relative to split in reference_split_name. If it's relative, the effective threshold is score_of_reference_split * (1 + value). Defaults to "ABSOLUTE".

'ABSOLUTE'
fail_if_less_than Optional[float]

Fail if score is less than the value specified in this argument.

None
fail_if_greater_than Optional[float]

Fail if score is greater than the value specified in this argument.

None
fail_if_within Optional[Tuple[float, float]]

Fail if value[0] < score < value[1].

None
fail_if_outside Optional[Tuple[float, float]]

Fail if score < value[0] OR score > value[1].

None
fail_threshold_type str

Must be one of ["ABSOLUTE", "RELATIVE"]. Describe whether the fail threshold is defined as absolute value or relative to split in reference_split_name. If it's relative, the effective threshold is score_of_reference_split * (1 + value). Defaults to "ABSOLUTE".

'ABSOLUTE'
reference_split_name Optional[str]

Name of the reference split used for the "RELATIVE" threshold type. If not specified and reference_model_name is also not provided, the relative threshold will be calculated with respect to each models' train split (for models whose train split is not specified, then those will be treated as if no thresholds were specified).

None
reference_model_name Optional[str]

Name of the reference model used for the "RELATIVE" threshold type.

None
description Optional[str]

Text description of the test.

None
overwrite bool

If set to True, will overwrite the thresholds for existing test specified under the given data_split_name, segment, and metric. Defaults to False.

False

Examples of adding performance test with absolute threshold:

# Performance test on multiple data splits with a single value threshold
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 1",
    data_split_names=["split1_name", "split2_name"],
    metric="CLASSIFICATION_ACCURACY",
    warn_if_less_than=0.85,
    fail_if_less_than=0.82
)

# Alternative, we can also specify data split name using regex
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 2",
    data_split_name_regex=".*-California", # this test will be run on all data splits where the name contains "-California"
    all_data_collections=True, # this test will be applicable to all data collections
    metric="CLASSIFICATION_ACCURACY",
    warn_if_less_than=0.85,
    fail_if_less_than=0.82
)

# Performance test using a segment with a single value threshold
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 3",
    data_split_names=["split1_name", "split2_name"],
    segment_group_name="segment_group_name",
    segment_name="segment_name",
    metric="FALSE_POSITIVE_RATE",
    warn_if_greater_than=0.05,
    fail_if_greater_than=0.1
)

# Performance test with a range threshold
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 4",
    data_split_names=["split1_name", "split2_name"],
    metric="FALSE_NEGATIVE_RATE",
    warn_if_outside=(0.05, 0.1),
    fail_if_outside=(0.02, 0.15)
)

Examples of adding performance test with relative threshold:

# Explicitly specifying the reference split of a RELATIVE threshold
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 5",
    data_split_names=["split1_name", "split2_name"],
    metric="CLASSIFICATION_ACCURACY",
    warn_if_less_than=-0.05, # warn if accuracy of split < (1 + -0.05) * accuracy of reference split
    warn_threshold_type="RELATIVE",
    fail_if_less_than=-0.08,
    fail_threshold_type="RELATIVE",
    reference_split_name="reference_split_name"
)

# Not explicitly specifying the reference split on a RELATIVE threshold means
# the reference split is each model's train split
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 6",
    data_split_names=["split1_name", "split2_name"],
    metric="FALSE_POSITIVE_RATE",
    warn_if_greater_than=0.02,
    warn_threshold_type="RELATIVE",
    fail_if_greater_than=0.021,
    fail_threshold_type="RELATIVE"
)

# RELATIVE test using reference model instead of reference split
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 7",
    data_split_names=["split1_name", "split2_name"],
    metric="CLASSIFICATION_ACCURACY",
    warn_if_less_than=0,
    warn_threshold_type="RELATIVE",
    fail_if_less_than=-0.01,
    fail_threshold_type="RELATIVE",
    reference_model_name="reference_model_name"

# RELATIVE test using both reference model and reference split
>>> tru.tester.add_performance_test(
    test_name="Accuracy Test 8",
    data_split_names=["split1_name", "split2_name"],
    metric="CLASSIFICATION_ACCURACY",
    warn_if_less_than=0,
    warn_threshold_type="RELATIVE",
    fail_if_less_than=-0.01,
    fail_threshold_type="RELATIVE",
    reference_model_name="reference_model_name",
    reference_split_name="reference_split_name"
)

add_stability_test(self, test_name, metric='DIFFERENCE_OF_MEAN', *, comparison_data_split_names=None, comparison_data_split_name_regex=None, base_data_split_name=None, data_collection_names=None, all_data_collections=False, segments=None, warn_if_less_than=None, warn_if_greater_than=None, warn_if_within=None, warn_if_outside=None, fail_if_less_than=None, fail_if_greater_than=None, fail_if_within=None, fail_if_outside=None, description=None, overwrite=False)

Add a stability test to the current data collection in context. To set warning condition, please provide one of [warn_if_less_than, warn_if_greater_than, warn_if_within, warn_if_outside]. Similarly, to set fail condition please provide one of [fail_if_less_than, fail_if_greater_than, fail_if_within, fail_if_outside].

Parameters:

Name Type Description Default
test_name str

The name of the test.

required
metric str

Stability metric for the test. Must be one ["WASSERSTEIN", "DIFFERENCE_OF_MEAN", "POPULATION_STABILITY_INDEX"]

'DIFFERENCE_OF_MEAN'
comparison_data_split_names Optional[Sequence[str]]

List of the data splits that we want to use for the test.

None
comparison_data_split_name_regex Optional[str]

Regex of the data split name that we want to use for the test (future data splits that match this naming pattern will automatically be included in the test).

None
base_data_split_name Optional[str]

Name of the reference data split to use as the comparison baseline for the test. If None, will be the model's train split.

None
data_collection_names Optional[Sequence[str]]

List of the data collections for the test. If not specified, the test will only apply to the current data collection in context.

None
all_data_collections bool

If set to True, the test will apply to all data collections (including future ones). Defaults to False which means the test will only apply to the current data collection in context (if data_collection_names is not specified).

False
segments Optional[Sequence[Tuple[str, str]]]

List of (segment_group_name, segment_name) tuple to be used as filter. A test will be defined on each of the given segment filters.

None
warn_if_less_than Optional[float]

Warn if score is less than the value specified in this argument.

None
warn_if_greater_than Optional[float]

Warn if score is greater than the value specified in this argument.

None
warn_if_within Optional[Tuple[float, float]]

Warn if value[0] < score < value[1].

None
warn_if_outside Optional[Tuple[float, float]]

Warn if score < value[0] OR score > value[1].

None
fail_if_less_than Optional[float]

Fail if score is less than the value specified in this argument.

None
fail_if_greater_than Optional[float]

Fail if score is greater than the value specified in this argument.

None
fail_if_within Optional[Tuple[float, float]]

Fail if value[0] < score < value[1].

None
fail_if_outside Optional[Tuple[float, float]]

Fail if score < value[0] OR score > value[1].

None
description Optional[str]

Text description of the test.

None
overwrite bool

If set to True, will overwrite the thresholds for existing test specified under the given comparison_data_split_name, segment, and metric. Defaults to False.

False

Examples:

>>> tru.tester.add_stability_test(
    test_name="Stability Test",
    comparison_data_split_names=["split1_name", "split2_name"],
    base_data_split_name="reference_split_name",
    metric="DIFFERENCE_OF_MEAN",
    warn_if_outside=[-1, 1],
    fail_if_outside=[-2, 2]
)

delete_tests(self, test_name=None, test_type=None, data_split_name=None, segment_group_name=None, segment_name=None, metric=None)

Delete tests.

Parameters:

Name Type Description Default
test_name Optional[str]

Only delete tests with the given name.

None
test_type Optional[str]

Only delete tests of this type. Must be one of ["performance", "stability", "fairness"] or None. If None, delete all test types. Defaults to None.

None
data_split_name Optional[str]

Only delete tests associated with this data split. Defaults to None.

None
segment_group_name Optional[str]

Only delete tests associated with this segment group. Defaults to None.

None
segment_name Optional[str]

Only delete tests associated with this segment. Defaults to None.

None
metric Optional[str]

Only delete tests associated with this metric. Defaults to None.

None

get_model_leaderboard(self, sort_by='performance', wait=True)

Get the summary of test outcomes for all models in the data collection.

Parameters:

Name Type Description Default
sort_by str

Rank models according to the test type specified in this arg (models with the fewest test failures will be at the top). Must be one of ["performance", "stability", "fairness"]. Defaults to "performance".

'performance'
wait bool

Whether to wait for test results to finish computing. Defaults to True.

True

Returns:

Type Description
ModelTestLeaderboard

A ModelTestLeaderboard object containing the summary of test outcomes for all models in the data collection. On Jupyter notebooks, this object will be displayed as a nicely formatted HTML table. This object also has a pretty_print as well as as_json and as_dict representation.

get_model_test_results(self, data_split_name=None, comparison_models=None, test_types=None, wait=True)

Get the test results for the model in context.

Parameters:

Name Type Description Default
data_split_name Optional[str]

If provided, filters to the tests associated with this split.

None
comparison_models Optional[Sequence[str]]

If provided, compare the test results against this list of models.

None
test_types Optional[Sequence[str]]

If provided, filter to only the given test-types. Must be a subset of ["performance", "stability", "fairness"] or None (which defaults to all). Defaults to None.

None
wait bool

Whether to wait for test results to finish computing.

True

Returns:

Type Description
ModelTestResults

A ModelTestResults object containing the test results for the model in context. On Jupyter notebooks, this object will be displayed as a nicely formatted HTML table. This object also has a pretty_print as well as as_json and as_dict representation.

get_model_tests(self, data_split_name=None)

Get the details of all the model tests in the current data collection or the model tests associated with the given data split.

Parameters:

Name Type Description Default
data_split_name Optional[str]

If provided, filters to the tests associated with this split.

None

Returns:

Type Description
ModelTestDetails

A ModelTestDetails object containing the details for each test that has been created. On Jupyter notebooks, this object will be displayed as a nicely formatted HTML table. This object also has a pretty_print as well as as_json and as_dict representation.