Customize and access

Now that you've confirmed the dataset is right for your research, it's time to start accessing the dataset for analysis.

Accessing data

To get started, select the Get Data button in the upper-right corner of a dataset page.

❗️

Note data provider requirements

In some cases, Unique Data Provider Requirements will appear before the access options. Be sure to review these thoroughly before continuing to ensure your research project is compliant with these terms.

Customizing data

In many cases, you'll have the option to first customize a dataset before downloading it. This is especially useful when working with large datasets - it lets you filter them down to only the data you need so that, ultimately, the dataset you download is smaller and easier to work with.

The Columns section will show each column header in the dataset, and provide you with the ability to remove columns you do not need. The Filters section will include any attributes that you can use to filter. Not all attributes will appear in the Filters section, but some common ones are region, NAICS, brand, etc.

Row-based requirements

At times, especially for large datasets, you will have the option to customize a dataset, but only if you reduce the row count to a certain percentage of the total product size. You will see the row count requirements at the top of the Customize Data modal. A progress bar will be displayed at the bottom of the screen, which will turn green when the filtered amount is equal to or below the required amount. Reducing the dataset below that threshold will enable you to continue with the data export.

In many cases, if you can't find filters for you, or you'd like to do the querying on your own system, you can still download the entire dataset without a filter applied.

📘

Why can't I filter on every attribute?

This is an example of a data-engineering decision aimed at reducing compute costs so we can keep our service reasonably priced. Please contact the Dewey team if you run into in issue with the filtering limits.

Reducing columns

To help you reduce the size of your export, one additional feature option is the capability to remove extraneous columns. Use the drop-down on the left to select the attributes you'd like in your export.

Note that this does not impact any row-based requirements on the dataset.

Downloading data

Once you are done customizing a dataset, or you've skipped that step altogether, you can download the data. Choose to either download a CSV or use the Bulk API access option. See the step-by-step guide for access options.

💡

Why are some options not available?

As the size of the dataset changes, the available download options change. If you need further support using the API, check out our API tutorial page.

Data refreshes

A data refresh typically means that new data is added to the existing dataset. Several datasets on Dewey are updated at a fixed cadence. You can review the Refresh Schedule in the data highlights section on the dataset page by the 🔁 symbol.

👍

I need consistent data for my research, will refreshes impact this?

No, scheduled data refreshes do not affect historical data. If a reinstatement is deemed necessary by one of our partners, we'll be sure to notify you ahead of time so you time to make the proper adjustments.

❗️

Note static files do not refresh

Some of the access options will instantly download a static file to your local machine. If you select these options, you won't dynamically access updates to the data that occur during refreshes, so be sure to check back to get the latest data, or include it in your script.

Managing "Saved" Datasets

Datasets that you have saved and accessed can be managed in the Saved menu on the left side of the platform interface.

From this menu, you can review the datasets you have accessed, and details such as data provider, date of last refreshed, last access date, and you can access the dataset with the Get data button.

You can remove datasets from your Saved datasets by selecting ... and selecting 🗑️Remove.