Skip to content

About

A tool for bulk download of datasets from different data sources.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

95 Commits

Folders and files

Repository files navigation

Contributors Forks Stargazers Issues MIT License


****************************************************************************************************
   ___                         ___         _             ___                         _             
  /___\ _ __    ___  _ __     /   \  __ _ | |_   __ _   / __\ _ __   __ _ __      __| |  ___  _ __ 
 //  //| '_ \  / _ \| '_ \   / /\ / / _` || __| / _` | / /   | '__| / _` |\ \ /\ / /| | / _ \| '__|
/ \_// | |_) ||  __/| | | | / /_// | (_| || |_ | (_| |/ /___ | |   | (_| | \ V  V / | ||  __/| |   
\___/  | .__/  \___||_| |_|/___,'   \__,_| \__| \__,_|\____/ |_|    \__,_|  \_/\_/  |_| \___||_|   
       |_|                                                                              - v2.3.0   
****************************************************************************************************

A flexible tool to crawl and normalize data from open data portals into your projects.
Report a Bug · Request a Feature

Table of Contents
  1. About the Project
  2. Getting Started
  3. Usage
  4. Contributing
  5. License
  6. Contact

About The Project

Open Data Crawler is a tool designed to extract datasets, and optionally their metadata, from open data and statistics portals. The community can contribute by adding support for new portals or implementing additional features.

Key Features

  • Download datasets from open data or statistics portals
  • Retrieve metadata from resources
  • Filter datasets by data type
  • Filter datasets by topic or category

Currently Supported Portals and Sites

* Works with restrictions or download limitations

See the open issues for a full list of proposed features and known issues.

(back to top)


Getting Started

To set up the project locally, follow these steps:

Requirements

  • Python 3.12+ on Linux
  • Optional CKAN and Zenodo tokens can be configured in [CKAN] and [zenodo] sections. Public endpoints do not receive an empty bearer token.
  • Copy opendatacrawler/config.example.ini to a private configuration file and set ODC_CONFIG=/absolute/path/config.ini. A local opendatacrawler/config.ini is also read when ODC_CONFIG is unset; credentials are excluded from built distributions.
  • Runtime defaults can also be configured in opendatacrawler/config.ini using sections such as [defaults], [datosgobes], [zenodo] or [ine].
    • Example for partial tabular downloads:
      [defaults]
      partial_dataset_rows = 100
      partial_dataset_sample_mode = first
      ; partial_dataset_random_seed = 1

Important

Command-line arguments take priority over config.ini, and config.ini takes priority over the built-in defaults.

Installation

  1. Clone the repo
git clone https://github.com/aberenguerpas/opendatacrawler.git
  1. Install the package from the project root
pip install .

(back to top)


Usage

Using this tool is very simple: you only need to specify the URL of the data portal, and the tool will automatically detect the portal type and start downloading the data.

For a full list of available commands, run:

opendatacrawler -h

Examples

Download all data from a portal:

opendatacrawler -d https://datos.gob.es

Note

In datos.gob.es, this first refreshes or reuses the local catalog index under <output>/datos.gob.es/.cache/, and then processes packages from that local index.

Download specific format data (e.g., 'xls' and 'csv'):

opendatacrawler -d https://datos.gob.es -t xls csv

Note

The INE crawler uses JSON by default and also supports -t csv, -t xlsx/-t xls, and -t px, treating xls as xlsx. Any other extension is rejected for INE. Each selected format is downloaded as a separate distribution.

Download specific categories (e.g., 'turismo' and 'transporte'):

opendatacrawler -d https://datos.gob.es -c turismo transporte

Note

-c is not supported by INE as it does not expose categories through this crawler.

Download a specific dataset by ID:

opendatacrawler -d https://datos.gob.es -id a01002820-academias-andaluzas

Save the downloaded data in a custom folder:

opendatacrawler -d https://datos.gob.es -p /my/example/path/

Limit how many packages are processed:

opendatacrawler -d https://datos.gob.es -n 10

Limit how many resources are downloaded per package:

opendatacrawler -d https://datos.gob.es -nr 1

Save only metadata without downloading dataset files:

opendatacrawler -d https://datos.gob.es -nd

Download data from a specific country in data.europa.eu:

opendatacrawler -d https://data.europa.eu --country es

Note

--country is only supported by data.europa.eu.

Force re-download of specific datasets:

opendatacrawler -d https://datos.gob.es -id a01002820-academias-andaluzas -replace

Resume without checking whether already-downloaded packages changed remotely:

opendatacrawler -d https://datos.gob.es --skip-updates-check

Reset the local data and logs for one domain before crawling:

opendatacrawler -d https://datos.gob.es -rd

Caution

-rd deletes the local data and logs for the selected domain before starting the crawl.

Save original raw metadata from the portal:

opendatacrawler -d https://datos.gob.es --save-raw-data

Caution

--save-raw-data stores the original portal payload inside the generated metadata files. This is useful for debugging and traceability, but it increases metadata size and write time.

Disable schema extraction for CSV/TSV files:

opendatacrawler -d https://datos.gob.es --no-extract-schema

Store partial local copies when downloading CSV/TSV datasets using the sampling settings defined in config.ini:

[defaults]
partial_dataset_rows = 100
partial_dataset_sample_mode = first
; partial_dataset_random_seed = 1
opendatacrawler -d https://datos.gob.es -t csv tsv --partial-dataset

Important

--partial-dataset enables the feature for the current run. The number of retained data rows is read from config.ini (partial_dataset_rows), and partial_dataset_sample_mode controls whether the crawler keeps the first rows or a random sample after full download and processing. If partial_dataset_random_seed is set, the random sample is reproducible across runs.

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, share, and build together. Any contribution is greatly appreciated.

If you have a suggestion, improvement, or want to add support for a new site or portal, feel free to fork the repository and open a pull request. You can also open an issue using the enhancement label.

And don't forget to give the project a star! Thanks for your support! 🌟

How to Contribute

  1. Fork the repository
  2. Create a feature branch
  git checkout -b feature/AmazingFeature
  1. Commit your changes
  git commit -m 'Add some AmazingFeature'
  1. Push to the branch
  git push origin feature/AmazingFeature
  1. Open a Pull Request

Add Support for a New Portal

  1. Create a file named <PortalName>crawler.py inside opendatacrawler/portals/ (e.g. examplecrawler.py).
  2. Create a class <PortalName>Crawler with this constructor:
class ExampleCrawler:
    def __init__(self, odcrawler):
      self.odcrawler = odcrawler

      # Optional: limit requests per second (useful if the portal enforces rate limits)
      # self.odcrawler.init_rate_limit(reqs_per_sec=<RequestPerSecondLimit>)


      # Optional: authentication token if the portal requires it
      # self.token = utils.AUTH_TOKENS.get(<PortalName>)

Note

If the portal requires authentication, make sure to define its token in the config.ini file (see Requirements)

  1. Implement the required methods:
  • get_package_list(self):
    Returns the list of dataset or package identifiers from the portal's API. Each identifier represents a dataset entry (i.e., a collection of one or more downloadable resources), which will be fetched individually in the next step.

  • parse_resource(self, data, base_name):
    Normalizes a single resource object (distribution, file, link...) from the dataset into the project's internal resource schema.

  • get_package(self, dataset_id, metadata_file_name):
    Retrieves and normalizes the full metadata of a package of datasets given its identifier (including its associated resources using parse_resource()).

    To keep consistency across crawlers, it's recommended to structure the resource list as follows:

    if distributions:
        self.odcrawler.init_and_parse_resources(metadata, distributions)
  1. Use helper functions from the opendatacrawler/utils/ package and built-in logging:
  • Network calls: self.odcrawler.make_request(url, self.odcrawler.user_agent, headers=...)

Note

Reuse the returned agent value and close every response. Catalog failures must raise CatalogError; never return partial IDs as a successful enumeration.

  • File helpers: utils.generate_short_filename(), utils.get_mime_and_ext(), utils.get_extension_mime()...

  • Field extraction helpers: utils.extract_multilang_field(), utils.extract_mapped_field()

    For more advanced usage of utils.extract_mapped_field() with fallback logic and JSON mappings, see Define new mapping files.

  1. Return metadata using the normalized keys defined by the project, based on the DCAT standard. Each package of datasets must return a dictionary containing at least:
  • identifier: the package's ID from the portal.

  • fileName: the normalized name for saving the metadata file (provided to get_package()).

  • resources: a dictionary keyed by fileName, populated by init_and_parse_resources().

    It is also strongly recommended to include the following for debugging and traceability:

    • requestURL: the exact URL used to fetch the metadata.
    • accessURL: the public-facing URL where a user would normally access the package.

Each resource in the resources dictionary must contain:

  • downloadURL: the direct link to download the file.
  • fileName: the filename, including its extension, to use when saving the resource locally.
  • mediaType: the MIME type of the resource, which should be used to guess the proper file extension via utils.get_extension_mime().

Note

You don't need to include all possible DCAT fields, just the ones available in the portal. However, the structure of the returned metadata must remain consistent.

  1. Register portal detection:
    Update the function detect_dms() in odcrawler.py to detect your portal and return the correct crawler instance.

Define New Mapping Files

You can also contribute to the project by defining your own JSON mapping files inside the resources/ folder, following the naming pattern <portal>_<field>_map.json (lowercase) and then loading these mappings using load_resource() from opendatacrawler.utils, and assign them to a constant using the format <PORTALNAME>CRAWLER_<FIELDNAME>_MAP (uppercase).

These are used by utils.extract_mapped_field() to normalize raw values (e.g. URIs) into human-readable text, and to avoid unnecessary requests for known or static entities (such as publishers, themes, spatial areas, etc.).

Example of using a mapping file

In the case of needing a new field map for the field theme in the crawler ExampleCrawler, where the field contains URIs that should be normalized to an object with a human-readable name and an identifier:

  1. Create a file named resources/example_theme_map.json with the desired content. Example:
{
  "https://example.org/theme/health-care": {
    "name": "Health Care",
    "identifier": "HC12345"
  }
}
  1. Load the map using load_resource() from opendatacrawler.utils, providing a fallback value for unmapped entries:
EXAMPLECRAWLER_THEME_MAP = load_resource(
    "example_theme_map.json",
    fallback_value={"name": "FALLBACK_VALUE", "identifier": None}
)
  1. Use utils.extract_mapped_field() in your crawler code to normalize a value:
utils.extract_mapped_field("https://example.org/theme/energy-efficiency", EXAMPLECRAWLER_THEME_MAP)
  1. If the input exists in the map, the mapped value will be returned.

Otherwise, the fallback logic is applied:

  • For URLs (as in the example below), the last segment is extracted, hyphens are replaced with spaces, and the result is converted to title case (e.g., "energy-efficiency" becomes "Energy Efficiency").
  • For non-URLs, the raw value is used as-is.

The fallback result is then inserted into the "FALLBACK_VALUE" placeholder defined in the fallback template.

Final result:

[{ "name": "Energy Efficiency", "identifier": None }]

(back to top)


License

Distributed under the MIT License. See LICENSE for more information.

(back to top)


Contact

(back to top)

About

A tool for bulk download of datasets from different data sources.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages