Working with Notebooks

This page covers how Dewey notebooks work recommended habits for efficient data exploration, evaluation, and analysis.

How Dewey notebooks work

Dewey notebooks are hosted JupyterLab, Marimo, and RStudio environments that run inside your Project, with your saved datasets. There's no local setup and no download step.

• Scoped to your project. A notebook can read the datasets in its project, and only those. In Evaluate subscription plans, that's one project with up to 3 datasets.

• Ready to go. Python 3.14 with dewey, duckdb, pandas, polars, pyarrow, and matplotlib come preloaded. scipy, statsmodels, pyfixest, and seaborn are installed too. RStudio includes the tidyverse essentials, arrow, duckdb, and fixest.

• Data is read in place. Datasets stay in Dewey's storage. Loading one gives you a lazy handle, and queries read only the columns and partitions they touch.

• Files persist, sessions don't. Notebooks pause when idle. Your files are kept, but the kernel restarts, so rerun your cells when you come back.

The recommended flow

Four habits make a big difference to how quickly you get an answer to "will this dataset fit my research question?" First, customize and filter the dataset down before saving it to a Project, then in your Notebook, use lazy loading and DuckDB to gradually expand your scope.

1. Customize before you add

Why it matters: your notebook works on whatever you add to your project. Add a full national dataset for a question about three cities and two years, and every query scans data you'll never use.

Customizing a dataset on its Dewey page (picking only the places, dates, and columns you need) before you add it to your project means:

• Faster results. Queries finish in seconds instead of minutes.

• Fewer errors. Smaller data is far less likely to hit memory limits.

• More of your 30 days spent evaluating, not waiting.

Add datasets before you launch. Datasets added while a notebook is running aren't available until you restart it.

Load with dewey

Why it matters: dewey is how the notebook reads your project's data. It's preinstalled and already authenticated, so there are no API keys, credentials, or file paths to manage.

This build of dewey is specific to the notebook environment. It reads directly from Dewey's storage and works only inside your project's notebooks.

1. See what's in your project

import dewey

dewey.datasets()           # every dataset in this project
dewey.search("patterns")   # filter that list by name

The Name column is what you pass to the other calls. Names aren't case-sensitive. If you mistype one, the error suggests close matches.

2. Check the columns before loading anything

dewey.schema("[Dataset name]")

This reads metadata only, so it's instant on a dataset of any size.

3. Load a dataset

ds = dewey.load("[Dataset name]")

dewey.load() returns a handle, not a dataframe. Nothing has been read yet. From the handle, choose how you want to work:

Data is read only when you ask for a result, for example with .limit(10).df().

Working with several datasets? Each loaded dataset holds its own query connection. Use with so it's released when you're done:

with dewey.load("[Dataset name]") as ds:
    preview = ds.rel.limit(10).df()

In R (RStudio): dewey_datasets(), dewey_search(), dewey_schema(), and dewey_load() work the same way. dewey_load() returns a lazy table for dplyr; call collect() when you want the result.

Filter in DuckDB first

Why it matters: many Dewey datasets are far larger than notebook memory, so pulling one straight into pandas is slow at best and fails at worst. ds.rel is a lazy DuckDB relation over the dataset's Parquet files. Filters, projections, and aggregations push down to the scan, so DuckDB reads only the row groups and columns your query needs, and spills to disk if it has to.

The rule: do your filtering and summarizing on ds.rel, and convert to pandas or Polars only at the end, once the result is small.

ds = dewey.load("[Dataset name]")

# Filter rows and keep only the columns you need
ca = ds.rel.filter("state = 'CA'").project("date, city, visits")

# Summarize, then sort
by_city = ca.aggregate("city, sum(visits) AS total_visits", "city").order("total_visits DESC")

# Only now turn the small result into a dataframe
by_city.limit(20).df()

Column names here are examples. Use ds.schema to see yours.

Prefer SQL? Run a full query against the relation:

ds.rel.query("t", """
    SELECT city, sum(visits) AS total_visits
    FROM t
    WHERE state = 'CA'
    GROUP BY city
    ORDER BY total_visits DESC
    LIMIT 20
""").df()

Start small, then widen

Why it matters: your first few queries are about learning the data, not answering the question. Running them on a small slice gives you answers in seconds, so you can spot a missing field, an unexpected format, or a coverage gap early. Then you can run the real check across everything with confidence. Over a 30-day evaluation, that adds up.

1. Check the columns first. dewey.schema("[Dataset name]") costs nothing. Confirm that the fields your question needs exist before you read any data.

2. Load one slice. Most datasets are split into partitions, often by month. Load a single partition to work on a fraction of the data:

ds = dewey.load("[Dataset name]")
ds.resolved.partition_columns       # for example ('date_range_start',)

one_month = dewey.load("[Dataset name]", partitions=["2024-01-01"])

When a dataset has one partition column, you can pass just the value. Otherwise, use "column=value".

3. Look before you count. Preview rows, then check the things that would rule the dataset out:

one_month.rel.limit(10).df()                       # what does a row look like?

one_month.rel.aggregate("""
    count(*)                      AS rows,
    count(DISTINCT city)          AS cities,
    min(date)                     AS first_date,
    max(date)                     AS last_date,
    count(*) - count(visits)      AS missing_visits
""").df()

4. Widen. Once the query does what you expect, drop the partitions= filter, or add more values, and run the same code across the full range. If the full run is slow, that usually means a filter or column selection can move earlier in the query.

More help

• Python notebook guide: a full worked example, from loading to joining datasets. [Github link]

• R notebook guide: the same workflow in RStudio. [Github link]

• Discussions: ask other researchers about a dataset's coverage and quirks. [link]