Working with Python

Overview

Dewey makes it easy to work with large datasets in Python using DuckDB, Polars, and other lightweight tools. These examples demonstrate how to load data efficiently, query Parquet files at scale, filter before downloading, and run fast analyses with minimal setup.

You can get started in seconds with simple, copy-and-paste snippets designed to help you explore, aggregate, and visualize your data reliably.

Download

See Data Access for full details and a step-by-step guide on downloading data, or use the brief code snippets provided here.

Run the download through your Terminal with:

pip install deweypy

python -m deweypy --api-key <YOUR_API_KEY> speedy-download <prj_YOUR_PROJECT_ID>

Or run the download through your Python notebook by adding ! to the commands:

!pip install deweypy

!python -m deweypy --api-key <YOUR_API_KEY> speedy-download <prj_YOUR_PROJECT_ID>

Or see the DuckDB tutorial for filtering and querying just the data you need before downloading.

Load Locally

If you already have data files downloaded on your computer (either via direct browser download or with the Dewey Client in Terminal/Python), then the following guide walks through loading, filtering, and working with that data in Python using DuckDB.

Parquet Files

DuckDB can read the dataset metadata, apply filters, and download only the required partitions or rows, load files into a DataFrame, and combine multiple files into an analysis-ready dataset.

import duckdb
import pandas as pd

folder_path = r"<YOUR_FILE_PATH>"  
# example r"C:/Users/Documents/Dewey/dewey-downloads/rental-data-united-states"

con = duckdb.connect()

# Run SQL → DuckDB Relation → Convert to pandas DataFrame
df = con.execute(
    f"SELECT * FROM read_parquet('{folder_path}/*.parquet')"
).df()

print(df.shape)
df.head

Option to select only the columns you need or add filters:

df = con.execute(
    f"""
    SELECT 
{column 1},
{column 1},
{column 1}
    FROM read_parquet('{folder_path}/*.parquet')
    WHERE {numeric_column} > {threshold_value}
      AND {string_column} = '{category_name}'
      AND {date_column} >= '{start_date}'
    """
).df()

CSV Files

import duckdb
import pandas as pd

folder_path = r"<YOUR_FILE_PATH>"  
# example: r"C:/Users/Documents/Dewey/dewey-downloads/rental-data-united-states"

con = duckdb.connect()

df = con.execute(
    f"SELECT * FROM read_csv_auto('{folder_path}/*.csv.gz')"
).df()

print(df.shape)
df.head()

You can also select and filter only what you need by specifying in the con.execute() function.

Data Exploration & Visualization

The following section provides some starter Exploratory Data Analysis (EDA) tools you can run immediately after loading a dataset into Python. These commands help you inspect schema, evaluate variables, and visualize distributions in order to get a sense of the datasets shape and quality. 

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# -------------------------
# DIMENSIONS
# -------------------------
print("Shape:", df.shape)

# -------------------------
# STRUCTURE & SAMPLE
# -------------------------
df.info()
print(df.head())

# -------------------------
# SUMMARY STATS
# -------------------------
print(df.describe(include="all"))

# -------------------------
# MISSING VALUES
# -------------------------
print(df.isna().sum())

# -------------------------
# UNIQUE VALUES
# -------------------------
print(df.nunique())

# -------------------------
# NUMERIC SUBSET
# -------------------------
num = df.select_dtypes(include=[np.number])

# -------------------------
# CORRELATION
# -------------------------
corr = num.corr()
print(corr)

# -------------------------
# CORRELATION HEATMAP
# -------------------------
plt.figure(figsize=(10, 8))
sns.heatmap(corr, cmap="RdBu_r", center=0, linewidths=.5)
plt.title("Correlation Heatmap")
plt.show()

# -------------------------
# NUMERIC HISTOGRAM FACETS
# -------------------------
num_melted = num.melt(var_name="variable", value_name="value")

g = sns.FacetGrid(num_melted, col="variable", col_wrap=4, sharex=False, sharey=False)
g.map(plt.hist, "value", bins=30, edgecolor="black")
plt.show()