****************************************************************************************************
___ ___ _ ___ _
/___\ _ __ ___ _ __ / \ __ _ | |_ __ _ / __\ _ __ __ _ __ __| | ___ _ __
// //| '_ \ / _ \| '_ \ / /\ / / _` || __| / _` | / / | '__| / _` |\ \ /\ / /| | / _ \| '__|
/ \_// | |_) || __/| | | | / /_// | (_| || |_ | (_| |/ /___ | | | (_| | \ V V / | || __/| |
\___/ | .__/ \___||_| |_|/___,' \__,_| \__| \__,_|\____/ |_| \__,_| \_/\_/ |_| \___||_|
|_| - v2.3.0
****************************************************************************************************
A flexible tool to crawl and normalize data from open data portals into your projects.
Report a Bug
·
Request a Feature
Table of Contents
Open Data Crawler is a tool designed to extract datasets, and optionally their metadata, from open data and statistics portals. The community can contribute by adding support for new portals or implementing additional features.
- Download datasets from open data or statistics portals
- Retrieve metadata from resources
- Filter datasets by data type
- Filter datasets by topic or category
- datos.gob.es
- CKAN
- Zenodo * (request limit ≈ 1.39 req/sec ~ 5000 req/hour)
- GBIF
- datos.madrid.es
- data.europa.eu
- INE (Instituto Nacional de Estadística) *
- Socrata *
- Eurostat *
- World Bank Data Catalog *
- OpenDataSoft *
* Works with restrictions or download limitations
See the open issues for a full list of proposed features and known issues.
To set up the project locally, follow these steps:
- Python 3.12+ on Linux
- Optional CKAN and Zenodo tokens can be configured in
[CKAN]and[zenodo]sections. Public endpoints do not receive an empty bearer token. - Copy
opendatacrawler/config.example.inito a private configuration file and setODC_CONFIG=/absolute/path/config.ini. A localopendatacrawler/config.iniis also read whenODC_CONFIGis unset; credentials are excluded from built distributions. - Runtime defaults can also be configured in
opendatacrawler/config.iniusing sections such as[defaults],[datosgobes],[zenodo]or[ine].- Example for partial tabular downloads:
[defaults] partial_dataset_rows = 100 partial_dataset_sample_mode = first ; partial_dataset_random_seed = 1
- Example for partial tabular downloads:
Important
Command-line arguments take priority over config.ini, and config.ini takes priority over the built-in defaults.
- Clone the repo
git clone https://github.com/aberenguerpas/opendatacrawler.git- Install the package from the project root
pip install .Using this tool is very simple: you only need to specify the URL of the data portal, and the tool will automatically detect the portal type and start downloading the data.
For a full list of available commands, run:
opendatacrawler -hopendatacrawler -d https://datos.gob.es
Note
In datos.gob.es, this first refreshes or reuses the local catalog index under <output>/datos.gob.es/.cache/, and then processes packages from that local index.
opendatacrawler -d https://datos.gob.es -t xls csv
Note
The INE crawler uses JSON by default and also supports -t csv, -t xlsx/-t xls, and -t px, treating xls as xlsx. Any other extension is rejected for INE. Each selected format is downloaded as a separate distribution.
opendatacrawler -d https://datos.gob.es -c turismo transporte
Note
-c is not supported by INE as it does not expose categories through this crawler.
opendatacrawler -d https://datos.gob.es -id a01002820-academias-andaluzas
opendatacrawler -d https://datos.gob.es -p /my/example/path/
opendatacrawler -d https://datos.gob.es -n 10
opendatacrawler -d https://datos.gob.es -nr 1
opendatacrawler -d https://datos.gob.es -nd
opendatacrawler -d https://data.europa.eu --country es
Note
--country is only supported by data.europa.eu.
opendatacrawler -d https://datos.gob.es -id a01002820-academias-andaluzas -replace
opendatacrawler -d https://datos.gob.es --skip-updates-check
opendatacrawler -d https://datos.gob.es -rd
Caution
-rd deletes the local data and logs for the selected domain before starting the crawl.
opendatacrawler -d https://datos.gob.es --save-raw-data
Caution
--save-raw-data stores the original portal payload inside the generated metadata files. This is useful for debugging and traceability, but it increases metadata size and write time.
opendatacrawler -d https://datos.gob.es --no-extract-schema
Store partial local copies when downloading CSV/TSV datasets using the sampling settings defined in config.ini:
[defaults]
partial_dataset_rows = 100
partial_dataset_sample_mode = first
; partial_dataset_random_seed = 1opendatacrawler -d https://datos.gob.es -t csv tsv --partial-datasetImportant
--partial-dataset enables the feature for the current run. The number of retained data rows is read from config.ini (partial_dataset_rows), and partial_dataset_sample_mode controls whether the crawler keeps the first rows or a random sample after full download and processing. If partial_dataset_random_seed is set, the random sample is reproducible across runs.
Contributions are what make the open source community such an amazing place to learn, share, and build together. Any contribution is greatly appreciated.
If you have a suggestion, improvement, or want to add support for a new site or portal, feel free to fork the repository and open a pull request. You can also open an issue using the enhancement label.
And don't forget to give the project a star! Thanks for your support! 🌟
- Fork the repository
- Create a feature branch
git checkout -b feature/AmazingFeature- Commit your changes
git commit -m 'Add some AmazingFeature'- Push to the branch
git push origin feature/AmazingFeature- Open a Pull Request
- Create a file named
<PortalName>crawler.pyinsideopendatacrawler/portals/(e.g.examplecrawler.py). - Create a class
<PortalName>Crawlerwith this constructor:
class ExampleCrawler:
def __init__(self, odcrawler):
self.odcrawler = odcrawler
# Optional: limit requests per second (useful if the portal enforces rate limits)
# self.odcrawler.init_rate_limit(reqs_per_sec=<RequestPerSecondLimit>)
# Optional: authentication token if the portal requires it
# self.token = utils.AUTH_TOKENS.get(<PortalName>)Note
If the portal requires authentication, make sure to define its token in the config.ini file (see Requirements)
- Implement the required methods:
-
get_package_list(self):
Returns the list of dataset or package identifiers from the portal's API. Each identifier represents a dataset entry (i.e., a collection of one or more downloadable resources), which will be fetched individually in the next step. -
parse_resource(self, data, base_name):
Normalizes a single resource object (distribution, file, link...) from the dataset into the project's internal resource schema. -
get_package(self, dataset_id, metadata_file_name):
Retrieves and normalizes the full metadata of a package of datasets given its identifier (including its associated resources usingparse_resource()).To keep consistency across crawlers, it's recommended to structure the resource list as follows:
if distributions: self.odcrawler.init_and_parse_resources(metadata, distributions)
- Use helper functions from the
opendatacrawler/utils/package and built-in logging:
- Network calls:
self.odcrawler.make_request(url, self.odcrawler.user_agent, headers=...)
Note
Reuse the returned agent value and close every response. Catalog failures must raise CatalogError; never return partial IDs as a successful enumeration.
-
File helpers:
utils.generate_short_filename(),utils.get_mime_and_ext(),utils.get_extension_mime()... -
Field extraction helpers:
utils.extract_multilang_field(),utils.extract_mapped_field()For more advanced usage of
utils.extract_mapped_field()with fallback logic and JSON mappings, see Define new mapping files.
- Return metadata using the normalized keys defined by the project, based on the DCAT standard. Each package of datasets must return a dictionary containing at least:
-
identifier: the package's ID from the portal. -
fileName: the normalized name for saving the metadata file (provided toget_package()). -
resources: a dictionary keyed byfileName, populated byinit_and_parse_resources().It is also strongly recommended to include the following for debugging and traceability:
requestURL: the exact URL used to fetch the metadata.accessURL: the public-facing URL where a user would normally access the package.
Each resource in the resources dictionary must contain:
downloadURL: the direct link to download the file.fileName: the filename, including its extension, to use when saving the resource locally.mediaType: the MIME type of the resource, which should be used to guess the proper file extension viautils.get_extension_mime().
Note
You don't need to include all possible DCAT fields, just the ones available in the portal. However, the structure of the returned metadata must remain consistent.
- Register portal detection:
Update the functiondetect_dms()inodcrawler.pyto detect your portal and return the correct crawler instance.
You can also contribute to the project by defining your own JSON mapping files inside the resources/ folder, following the naming pattern <portal>_<field>_map.json (lowercase) and then loading these mappings using load_resource() from opendatacrawler.utils, and assign them to a constant using the format <PORTALNAME>CRAWLER_<FIELDNAME>_MAP (uppercase).
These are used by utils.extract_mapped_field() to normalize raw values (e.g. URIs) into human-readable text, and to avoid unnecessary requests for known or static entities (such as publishers, themes, spatial areas, etc.).
In the case of needing a new field map for the field theme in the crawler ExampleCrawler, where the field contains URIs that should be normalized to an object with a human-readable name and an identifier:
- Create a file named
resources/example_theme_map.jsonwith the desired content. Example:
{
"https://example.org/theme/health-care": {
"name": "Health Care",
"identifier": "HC12345"
}
}- Load the map using
load_resource()fromopendatacrawler.utils, providing a fallback value for unmapped entries:
EXAMPLECRAWLER_THEME_MAP = load_resource(
"example_theme_map.json",
fallback_value={"name": "FALLBACK_VALUE", "identifier": None}
)- Use
utils.extract_mapped_field()in your crawler code to normalize a value:
utils.extract_mapped_field("https://example.org/theme/energy-efficiency", EXAMPLECRAWLER_THEME_MAP)- If the input exists in the map, the mapped value will be returned.
Otherwise, the fallback logic is applied:
- For URLs (as in the example below), the last segment is extracted, hyphens are replaced with spaces, and the result is converted to title case (e.g., "energy-efficiency" becomes "Energy Efficiency").
- For non-URLs, the raw value is used as-is.
The fallback result is then inserted into the "FALLBACK_VALUE" placeholder defined in the fallback template.
Final result:
[{ "name": "Energy Efficiency", "identifier": None }]Distributed under the MIT License. See LICENSE for more information.
-
Paula Margarita García-Tapia Mateo
✉️ paula.garciatapia@ua.es -
Alberto Berenguer Pastor
✉️ alberto.berenguer@ua.es