Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Open Data 101
A quick guide to help you get started using NYC Open Data and understand what you're looking at.
Is this the data you're looking for?
You’ve found a public dataset! That’s great, but it’s worth spending a few minutes examining the data before you start using it. If you’re looking at data for the first time, it can be confusing to know where to begin, especially since the information is all over the landing page, or about page, of the dataset.
This information can be really helpful, but if you don’t know where to look, it can also be overwhelming. This guide will show you some of the main places to look on the about page to help you determine if this dataset will actually help answer your questions.
We don’t always recommend reading the about page from top to bottom. So join us for a little tour of the order we find most useful.
Note: Videos on this page demonstrate the steps described in the text. They have no sound.
Where to look first
The first things you see on the about page are:
- The title of the dataset
- A description of it
- The agency it belongs to
- When it was last updated
Some datasets are updated daily, some may not be updated for years. If your work requires recent data, it’s important to check the date it was updated.
What's in this dataset?
The next place to look to understand a dataset is the what’s in this dataset section. It’s located just below the about this dataset section, which we’ll get to next.
The what’s in this dataset section tells you how many rows and columns make up a dataset. Knowing the size of a dataset is useful because some datasets have too many rows to work with in standard tools like Excel. Excel has a limit of one million rows, which many of our datasets exceed.
Then you can see how the dataset is aggregated in the each row is a… entry. This tells you the basic unit of a dataset. For example, in the Restaurant Inspections dataset, each row is a citation, not a restaurant. This is because restaurants can be inspected multiple times over the years, so if you want to use this to make a list of restaurants in an area, you will have duplicates.
Below this, you can see information about each column in the dataset, including a short description of the column and the type of data it contains.
Data dictionaries
After looking at what’s in this dataset, scroll back up to the about this dataset section. Here you can see how frequently the dataset is updated and when it was originally created. You can also download the data dictionary, which is listed under attachments.
The data dictionary is the main source of documentation about a dataset. It’s the most useful place to look when you have general questions about a dataset and how to use it. Two main tabs of the spreadsheet to look at are:
- The dataset information tab
- The column information tab
The dataset information tab has more information than what’s included on the about page. There are sections on why and how the data is collected, which can give you more insight into the role of this data for the agency. There is also information on how the data can be used, which may link to related datasets, as well as some of the known limitations of the data.
The column information tab goes into more detail about each of the columns in the dataset. In addition to the column name and description, there is information about what is expected and any limitations for each column, which can be helpful if you spot a potential error. Agencies sometimes add notes if they have more they want to tell you about the column.
If you notice a missing data dictionary or key information that’s not included, contact us.
The changelog
Next, scroll to the very bottom of the about page. You may see a dataset changelog. This isn’t required, and agencies have to opt into this feature on a dataset-by-dataset basis. If it has been enabled, you can use it to see how often the data has been updated and download previous versions of the dataset.
Note, there is no way to download only a portion of the archive (it will download the complete file). Please be mindful of the size of the current dataset, as you can use that as a gauge for the size of the archive file.
Featured content
If you scroll back up, you may notice featured content on some about pages. If the Open Data Team knows about any interesting projects or reports that use this data, we try to highlight it here.
These projects can give you a sense of the different ways the data can be used. They can provide inspiration for your own work or give you another way to explore the data!
If you’ve used Open Data to create an interesting project, you can submit it to be featured on our site too!
Previewing the data
That’s everything on the main tab of the about page. If you want to get a quick look at the data, select the data tab at the top of the screen. Here, you can see a few rows at a glance to get a sense of what’s in this dataset to help you decide if this is what you’re looking for.
If you want to play with the data a little more before you download it, you can learn about filtering and exporting subsets of the data.