This article describes how to access data in Colorado Information Marketplace (CIM), the State of Colorado’s official open data portal and the data resource for Go Code Colorado.
More on CIM
The Colorado Information Marketplace (CIM) is a platform provided by Socrata and licensed by Colorado’s Statewide Internet Portal Authority (SIPA) on behalf of the Governor’s Office of Information Technology and the Chief Data Officer who sponsors and administers CIM. CIM is provided by Socrata as software-as-a-service (SaaS). CIM can be found here: https://data.colorado.gov/.
There are three primary methods to access data in CIM.
- Through CIM’s web application
- Through the Socrata Open Data API (SODA)
- In Microsoft Excel using an OData link
Access Through the Web Application
CIM’s web application enables you to search and browse datasets and views. You can browse datasets by view type (e.g., “datasets”, “maps”), by category (e.g., “demographics”), or by topic (e.g., “Go Code Colorado”). CIM contains multiple public datasets, but those datasets published specifically for Go Code Colorado will be “tagged” with both the “BIC” and “Go Code Colorado” topic keywords.
When you’ve found a dataset you’re interested in, you can use CIM web application to view the dataset itself. You can browse datasets in several views:
- as a table
- as a rich list
- as single records
CIM’s web application provides several other features for exploring datasets:
- Visualize (charts, map, etc.)
- View metadata
You can export the datasets in several formats:
You could then import the datasets into your own data architecture/platform.
Access through the Socrata Open Data API (SODA)
The Socrata Open Data API allows you to programatically access open data resources from CIM. Click the link below and for an example on how queries are structured in SODA.
http://soda.demo.socrata.com/resource/earthquakes.json?$where=magnitude > 5.5
The SODA API documentation has been broken down into three user categories: App Developers; Data Publishers; Data Consumers.
APP Developers: Locating Open Data and APIs
Once you’re on CIM, scroll down to the data catalog and use the search box and browse filters to find datasets that interest you. Every dataset is accessible via the SODA API.
Finding your API endpoint
Every Socrata open dataset has a built-in SODA API. But how you find the API endpoint can vary a bit.
If you’re on a Socrata dataset, identifiable by the colorful buttons at the upper right, click on Export and then API and you’ll find the API endpoint under API Access Endpoint. Copy that and save it for later.
Building simple filters and queries
Filtering data via a SODA API is fairly straightforward. There are two primary mechanisms you can use to filter data: Simple Filters and SoQL Queries
Filtering data is very straightforward. SODA APIs are self-describing – the schema and contents of the dataset itself determines how you can query it. Any field within the data can be used as a filter, simply by appending it to the API endpoint as a GET parameter.
Additional filters can be added, and the filters will be ANDed together. Read the detailed documentation on Filtering Datasets for more info.
The ”Socrata Query Language” (SoQL) is a simple, SQL-like query language specifically designed for making it easy to work with data on the web. The language is both powerful and easy to learn, and everything works via GET parameters. Many different functions are available via SoQL. Read the detailed documentation on SoQL Queries for more info.
For performance, SODA APIs are paged, and return a maximum of 1000 records per page. So, to request subsequent pages, you’ll need to use the $limit and $offset parameters to request more data. The $limit parameter chooses how many records to return per page, and $offset tells the API on what record to start returning data. Read the detailed documentation on Paging for more info.
Throttling and Applications Tokens
You can make a certain number of requests without an application token, but they come from a shared pool and you’re eventually going to get cut off.
If you want more requests, register for an application token and your application will be granted up to 1000 requests per rolling hour period. If you need even more than that, special exceptions are made by request. Use the Help! tab on the right of this page to file a trouble ticket.
Read the detailed documentation on Paging for more info.
SODA Features Summary
SODA provides an open, standards-based, RESTful application programming interface to access datasets in the CIM.
Every hosted dataset is readily and uniformly accessible. You can read the data, but also perform more advanced operations such as searching and filtering. Combine datasets with other Web services to create new mashups and applications. Socrata Open Data API (SODA) specifications were released to the Open Data community under a Creative Commons license
SODA is supported by a developer site http://beta.dev.socrata.com/.
Access in Microsoft Excel through an OData Link
CIM provides an OData link for every hosted dataset. With OData, you can create a spreadsheet in Microsoft Excel to analyze or visualize a dataset. By using an OData link, the data will be refreshable. You can use the OData link with Microsoft Excel 2010, Microsoft Excel 2013 or Microsoft Power Query.