Reading Data from Data Stores

TruEra supports a range data stores, from remote databases to data warehouses to a file or folder in a blob store. Currently, supported sources include:

• Azure Storage Blobs (WASB)
• AWS Storage Blobs (S3)
• BigQuery Tables

TruEra's Python SDK also supports upload and manipulation of data local to the caller's machine. This is useful for larger local inputs, as well as for learning the programming model without having any data in cloud storage.

Data Ingestion Basics

TruEra's remote data ingestion functionality provides a lightweight, interactive framework with automated tools supporting three primary operations — data sourcing, filtering, and output — each involving tables.

Placed into context, the stages of remote data ingestion involve:

1. Configuring the credentials needed, if any, to access the source (see add_credential()).
2. Adding/identiftying the data source containing the Table object (see add_data_source()).
3. Adding the data in the table in order to create a data split (see add_data()).

Before going further, let's quickly review the basic constructs:

TABLES

A table is a logical set of rows. A row is an ordered collection of fields. Tables can be of any size, comprise zero or more rows, and have a schema. They can come directly from a data store or be derived from another table. A table exists in one of three states — currently available, still processing, or failed. A table in a failed state throws an error. A table cannot be used by TruEra analytics — diagnostic, monitoring, and reports — until it is used to create a split.

DATA SOURCES

Data read from a data source creates a table. A data source can be any supported input type. Adding a data source creates a new table referenced by ID or the name given to the data source. A table created from a data source becomes a root table, unmodified by a filter or any other operation.

When reading from a data store requires credentials, these can be provided to TruEra either when reading is initiated or created ahead of time for the particular read. Storage of credentials can be configured to use either vault or an encryption key during TruEra platform setup.

FILTERS

A filter is applied to a table to return a new table limited to rows meeting the filtering condition. For example, filtering Table-A for rows where Column-1 < 5 returns Table-B containing all rows in A for which the filtering condition (Column-1 < 5) is true.

For the list of TruEra-supported SQL expressions for filtering, click here.

OUTPUT

A table ready for use by TruEra can be used to create a split. An operation ID is returned to query the status of the filtering operation. Options are also available for handling this synchronously.

To summarize, the best-practice flow for remote data ingestion entails:

1. Creating a root table from your data source.
2. Applying filters to the table and/or making queries about its rows/schema to formulate filters.
3. Creating one or more splits from a filtered table.

The specific process steps for remote data ingestion vary between the CLI, Python SDK, and the TruEra Web App. Each is explained next.

Remote Ingestion from a Data Store using the Python SDK

The basic ingestion classes used to create and filter tables, then generate splits, comprise:

INGESTION CLIENT

This class contains most of the functionality for creating tables and materializing splits from them. Ingestion client is returned from a call to get_ingestion_client() in the TruEra workspace. The complete list of functions can be found in the Python client reference. The ingestion client also supports creating and deleting credentials.

TABLE

This class represents the table itself in TruEra. Table objects support calls to filtering, getting the schema, and fetching sample rows returned in a data frame. See the Python client reference for complete details on supported Table class functions.

CREDENTIAL

This class is returned by a call to add_credential() and contains a Credential object containing a name and identity correlating which TruEra projects are permitted to use the credential to access the data source.

To ingest remote data:

Create an initial root table using TrueraWorkspace.add_data_source(). You can optionally provide a Credential object, when necessary. With the resulting Table object, you can filter() and get_sample_rows(). Use the TrueraWorkspace.add_data() method to add a data split.

Click Next below to contunue.

SQL Expressions for Filtering✕

A Boolean operator logically compares an operand to a literal producing a true/false result. The operand is always a column name (without quotation marks), is always on the left side of the expression, and represents the value of the table cell at coordinates row:col. A literal is a constant/unchanging value and is always on the right side of an expression. String literals must be enclosed in quotation marks ("). Do not use quotation marks for numeric literals.

For example, the expression amount > salary is invalid and will throw an exception because both the left and right side of the operator are column names.

Operator	Comparison	Example Filter Expression
=	operand EQUALS literal	amount = 1000
!=	operand NOT EQUAL to literal	amount != 1000
<	operand LESS THAN literal	amount < 1000
≤	operand LESS THAN OR EQUAL to literal	amount <= 1000
>	operand GREATER THAN literal	amount > 1000
≥	operand GREATER THAN OR EQUAL to literal	amount >= 1000
AND	operand in both expressions must be TRUE	(amount >= 1000) AND (state = "WA")
OR	operand in either or both expressions is TRUE	(amount >= 1000) OR (state = "WA")
NOT	reverses logical value of combined expressions	NOT((state = "WA") OR (state = "CA"))