Skip to main content
Version: 0.16.9

How to connect to data on S3 using Pandas

This guide will help you connect to your data stored on AWS S3 using Pandas. This will allow you to ValidateThe act of applying an Expectation Suite to a Batch. and explore your data.

Prerequisites: This how-to guide assumes you have:

Steps​

1. Choose how to run the code in this guide​

Get an environment to run the code in this guide. Please choose an option below.

If you use the Great Expectations CLICommand Line Interface, run this command to automatically generate a pre-configured Jupyter Notebook. Then you can follow along in the YAML-based workflow below:

great_expectations datasource new

2. Instantiate your project's DataContext​

Import these necessary packages and modules.

import great_expectations as gx
from great_expectations.core.batch import Batch, BatchRequest, RuntimeBatchRequest
from great_expectations.core.yaml_handler import YAMLHandler

yaml = YAMLHandler()

Load your DataContext into memory using the get_context() method.

context = gx.get_context()

3. Configure your Datasource​

Using this example configuration, add in your S3 bucket and path to a directory that contains some of your data:

datasource = context.sources.add_or_update_pandas_s3(
    name="s3_datasource", bucket="taxi-data-sample-test"
)

In the example, we have added a Datasource that connects to data in S3 using a Pandas dataframe. The name of the new datasource is s3_datasource and it refers to a S3 bucket named taxi-data-sample-test.

4. Save the Datasource configuration to your DataContext​

Save the configuration into your DataContext by using the add_datasource() function.

context.add_datasource(**yaml.load(datasource_yaml))

5. Test your new Datasource​

Verify your new DatasourceProvides a standard API for accessing and interacting with data from a wide variety of source systems. by loading data from it into a ValidatorUsed to run an Expectation Suite against data. using a Batch RequestProvided to a Datasource in order to create a Batch..

We will use the BatchRequest configured in the previous step.

context.add_or_update_expectation_suite(expectation_suite_name="test_suite")
validator = context.get_validator(
    batch_request=request, expectation_suite_name="test_suite"
)

print(validator.head())

🚀🚀 Congratulations! 🚀🚀 You successfully connected Great Expectations with your data.

Additional Notes​

To view the full scripts used in this page, see them on GitHub:

Next Steps​

Now that you've connected to your data, you'll want to work on these core skills: