From 6ce4283971f73072f8fe62f10518bb2295846175 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 2 Jul 2026 10:42:15 -0700 Subject: [PATCH 01/16] Redo statvar examples --- custom_dc/custom_data.md | 55 ++++++++++++++++++++++------------------ 1 file changed, 30 insertions(+), 25 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index ec96439c4..afcd909d0 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -176,28 +176,32 @@ Nodes in the Data Commons knowledge graph are defined in Metadata Content Format You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container. -Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. +Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. ``` -Node: dcid:who/Adult_curr_cig_smokers +Node: dcid:who/Percent_Smokers_Adults typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" populationType: dcid:Person -measuredProperty: dcid:percent +measuredProperty: dcid:cigaretteSmoking +statType: dcid:percent +measurementDenominator: dcid:Count_Person -Node: dcid:who/Adult_curr_cig_smokers_female +Node: dcid:who/Percent_Smokers_Adult_Females typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" -populationType: dcid:Person -measuredProperty: dcid:percent -gender: dcid:Female +populationType: dcid:Female +measuredProperty: dcid:cigaretteSmoking +statType: dcid:percent +measurementDenominator: dcid:Count_Person_Female -Node: dcid:who/Adult_curr_cig_smokers_male +Node: dcid:who/Percent_Smokers_Adult_Males typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" -populationType: dcid:Person -measuredProperty: dcid:percent -gender: dcid:Male +populationType: dcid:Male +measuredProperty: dcid:cigaretteSmoking +statType: dcid:percent +measurementDenominator: dcid:Count_Person_Male ``` The order of nodes and fields within nodes does not matter. @@ -206,8 +210,8 @@ The following fields are always required: > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. - `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. - `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. -- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. -- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the `percent` of persons being measured. +- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person`, `dcid:Female` and `dcid:Male`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. +- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoking` of persons, females, and males, being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: @@ -219,11 +223,11 @@ Note that all fields that reference another node in the graph must be prefixed b The following fields are optional: - `description`: A more detailed textual description of the variable. -- `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, `varianceValue`, `marginOfError`, `stdErr` and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +- `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. - `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. - `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. -Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our example, there is one constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. +Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we could have used a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. {: #naming} #### Variable DCID naming conventions @@ -345,16 +349,17 @@ Here is an example of some real-world data from the WHO on the prevalance of smo ```csv SERIES,GEOGRAPHY,TIME_PERIOD,OBS_VALUE -dcs:who/Adult_curr_cig_smokers_female,dcid:country/AFG,2019,1.2 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/AFG,2019,13.4 -dcs:who/Adult_curr_cig_smokers,dcid:country/AFG,2019,7.5 -dcs:who/Adult_curr_cig_smokers_female,dcid:country/AGO,2016,1.8 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/AGO,2016,14.3 -dcs:who/Adult_curr_cig_smokers_female,dcid:country/ALB,2018,4.5 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/ALB,2018,35.7 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/ARE,2018,11.1 -dcs:who/Adult_curr_cig_smoking_female,dcid:country/ARE,2018,1.6 -dcs:who/Adult_curr_cig_smokers,dcid:country/ARE,2018,6.3 +dcs:who/Percent_Smokers_Adult_Females,dcid:country/AFG,2019,1.2 +dcs:who/Percent_Smokers_Adult_Males,dcid:country/AFG,2019,13.4 +dcs:who/Percent_Smokers_Adults,dcid:country/AFG,2019,7.5 +dcs:who/Percent_Smokers_Adult_Females,dcid:country/AGO,2016,1.8 +dcs:who/Percent_Smokers_Adult_Males,dcid:country/AGO,2016,14.3 +dcs:who/Percent_Smokers_Adult_Females,dcid:country/ALB,2018,4.5 +dcs:who/Percent_Smokers_Adult_Males,dcid:country/ALB,2018,35.7 +dcs:who/Percent_Smokers_Adult_Males,dcid:country/ARE,2018,11.1 +dcs:who/Percent_Smokers_Adult_Females,dcid:country/ARE,2018,1.6 +dcs:who/Percent_Smokers_Adults,dcid:country/ARE,2018,6.3 +... ``` In this case, the columns need to be mapped to the expected columns listed above; see below for details. From 50048e141d02f95b2254315d8bba76ef1457ce28 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 2 Jul 2026 10:48:46 -0700 Subject: [PATCH 02/16] Fix name --- custom_dc/custom_data.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index afcd909d0..e118863e5 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -183,7 +183,7 @@ Node: dcid:who/Percent_Smokers_Adults typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" populationType: dcid:Person -measuredProperty: dcid:cigaretteSmoking +measuredProperty: dcid:cigaretteSmoker statType: dcid:percent measurementDenominator: dcid:Count_Person @@ -191,7 +191,7 @@ Node: dcid:who/Percent_Smokers_Adult_Females typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" populationType: dcid:Female -measuredProperty: dcid:cigaretteSmoking +measuredProperty: dcid:cigaretteSmoker statType: dcid:percent measurementDenominator: dcid:Count_Person_Female @@ -199,7 +199,7 @@ Node: dcid:who/Percent_Smokers_Adult_Males typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Male -measuredProperty: dcid:cigaretteSmoking +measuredProperty: dcid:cigaretteSmoker statType: dcid:percent measurementDenominator: dcid:Count_Person_Male ``` @@ -211,7 +211,7 @@ The following fields are always required: - `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. - `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. - `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person`, `dcid:Female` and `dcid:Male`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. -- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoking` of persons, females, and males, being measured. +- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: From 44c933b25bfd3bb98c3cd2353c42ff02154e5bd6 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 2 Jul 2026 11:21:27 -0700 Subject: [PATCH 03/16] Fix population type --- custom_dc/custom_data.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index e118863e5..a1a1919df 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -190,7 +190,8 @@ measurementDenominator: dcid:Count_Person Node: dcid:who/Percent_Smokers_Adult_Females typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" -populationType: dcid:Female +populationType: dcid:Person +gender: dcid:Female measuredProperty: dcid:cigaretteSmoker statType: dcid:percent measurementDenominator: dcid:Count_Person_Female @@ -198,7 +199,8 @@ measurementDenominator: dcid:Count_Person_Female Node: dcid:who/Percent_Smokers_Adult_Males typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" -populationType: dcid:Male +populationType: dcid:Person +gender: dcid:Male measuredProperty: dcid:cigaretteSmoker statType: dcid:percent measurementDenominator: dcid:Count_Person_Male @@ -210,7 +212,7 @@ The following fields are always required: > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. - `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. - `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. -- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person`, `dcid:Female` and `dcid:Male`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. +- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. - `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: @@ -227,7 +229,7 @@ The following fields are optional: - `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. - `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. -Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we could have used a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. +Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. {: #naming} #### Variable DCID naming conventions From 2c53d1c8d52c860eeb02d607cbf08ba789cba0b4 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 2 Jul 2026 11:23:38 -0700 Subject: [PATCH 04/16] Fix typo --- custom_dc/custom_data.md | 159 ++++++++++++++++++++++----------------- 1 file changed, 90 insertions(+), 69 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index a1a1919df..3f8f44451 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -6,6 +6,7 @@ parent: Build your own Data Commons --- {:.no_toc} + # Prepare and load your own data This page shows you how to format and load your own custom data into your local instance. This is step 2 of the [recommended workflow](/custom_dc/index.html#workflow). @@ -21,13 +22,14 @@ Custom Data Commons requires that you provide your data in a specific schema, fo At a high level, you need to provide the following: -- If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. -- All observations data must be in CSV format, using the schema described later. -- You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. +* If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. +* All observations data must be in CSV format, using the schema described later. +* You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. If you need to define new entities, please see [Define custom entities](custom_entities.md) for details. {: #dir} + ### Files and directory structure You can have as many CSV and MCF files as you like, and they can be in multiple subdirectories (with an additional [configuration option](#subdirs)). There must only be one JSON config file, in the top-level input directory. For example: @@ -43,26 +45,28 @@ my_data/ ├── datafile3.csv └── datafile4.csv ``` + The top-level directory (e.g. `my_data`) can live anywhere in the file system; you will specify the full path to it when you [configure your input directory](#env). When you set up your files in Google Cloud Storage using the Terraform script, it will automatically create a top-level directory in your bucket called `input`. -The following sections walk you through the process of setting up your data. +The following sections walk you through the process of setting up your data. ## Prerequisite steps The following sections describe the high-level conceptual work you need to do before starting to write your data and config files. {: entities} + ### Step 0.1: Determine whether you need new entities or entity types Data Commons is optimized to support aggregations of data at geographical levels, such as city, state, country, and so on. If your data is aggregated by place, these are supported as entities out of the box. If, however, you want to aggregate data for entities that are _not_ places, then you may need to define new entities, and possibly even entity types. -In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. +In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. #### Entities and entity types Schema.org and the base Data Commons knowledge graph define entity types for just about everything in the world. An _entity type_ is a high-level concept, and is derived directly from a [`Class`](https://datacommons.org/browser/Class){: target="_blank"} type. Non-place entities are of two types: -- The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. -- The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. +* The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. +* The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. It is rare that you would need to create a new entity type, unless you are working in a highly specialized domain. An _entity_ is an instance of an entity type. For example, for `PublicSchool`, base Data Commons has many U.S. schools in its knowledge graph, such as [`nces/010162001665`](https://datacommons.org/browser/nces/010162001665){: target="_blank"} (Adams Elementary School) or [`nces/010039000201`](https://datacommons.org/browser/nces/010039000201){: target="_blank"} (Wylam Elementary School). Base Data Commons contains thousands of places and other entities, but it's possible that it does not have specific entities that you need. For example, it has about 100 instances of `Company`, but you may want data for other companies besides those. As another example, let's say your organization wants to collect (possibly private) data about different divisions or departments of your org; in this case you would need to define entities for them. @@ -70,35 +74,40 @@ An _entity_ is an instance of an entity type. For example, for `PublicSchool`, b > **Note:** You should always reuse existing entity types and entities from base Data Commons rather than re-defining them. This way, you get all the properties already defined for those entities and all their linked nodes, and can more easily join with base data if needed. {: #search} + #### Search for an existing entity / entity type -Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. +Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. To search using the REST APIs: -1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. -1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: +1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: 
https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=ENTITY_TYPE&property=<-typeOf
_ENTITY_TYPE_ is the DCID you've obtained in the previous step, such as `Hospital` or `PublicSchool`. For example: + ``` https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=PublicSchool&property=<-typeOf ``` + 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. To search using the Python APIs: 1. Start your Python interactive environment and [create a client for the base Data Commons](/api/python/v2/index.html). -1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. 1. Use the `fetch_property_values` method to find all the instances of the type: 
client.node.fetch_property_values(node_dcids="ENTITY_TYPE", properties="typeOf", out=False)
_ENTITY_TYPE_ is the DCID you've obtained in the previous step. For example: + ``` client.node.fetch_property_values(node_dcids="PublicSchool", properties="typeOf", out=False) ``` + 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. ### Step 0.2: Identify your statistical variables @@ -125,12 +134,12 @@ If you do need to define new variables, they must follow a certain model. The va | San Jose | 2023 | private | secondary | 100 | The measure here is a simple count; the set of things is "schools"; and the constraints are the type and levels of the schools, namely "public", "private", "elementary", "middle" and "secondary". All of these things must be encoded as separate variables. Therefore, although the _properties_ of school type and school level may already be defined in the Data Commons knowledge graph (or you may need to define them), they _cannot_ be present as columns in the CSV files that you store in Data Commons. Instead, you must create separate "count" variables to represent each case. In our example, you would actually need 6 different variables: -- `Count_School_Public_Elementary` -- `Count_School_Public_Middle` -- `Count_School_Public_Secondary` -- `Count_School_Private_Elementary` -- `Count_School_Private_Middle` -- `Count_School_Private_Secondary` +* `Count_School_Public_Elementary` +* `Count_School_Public_Middle` +* `Count_School_Public_Secondary` +* `Count_School_Private_Elementary` +* `Count_School_Private_Middle` +* `Count_School_Private_Secondary` If you wanted totals or subtotals of combinations, you would need to create additional variables for these as well. @@ -159,24 +168,25 @@ Data Commons uses a schema that is called "variable-per-row". This means that ev The names and order of the columns aren't important, as you can map them to the expected columns in the JSON file. However, the city and variable names must be existing DCIDs. If such DCIDs don't already exist in the base Data Commons, you must provide definitions of them in MCF files. -> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. +> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. ## Prepare your data In this section, we will walk you through a concrete example of how to go about setting up your MCF, CSV, and JSON files. {: #mcf} + ### Step 1: Define statistical variables in MCF If you are only reusing existing variables, you can skip this step entirely. -Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. +Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. > **Note:** You cannot "override" a variable definition by changing the value of existing fields. If you need to override the values of existing fields, you should create a new variable, with a new DCID. You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container. -Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. +Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. ``` Node: dcid:who/Percent_Smokers_Adults @@ -205,50 +215,52 @@ measuredProperty: dcid:cigaretteSmoker statType: dcid:percent measurementDenominator: dcid:Count_Person_Male ``` + The order of nodes and fields within nodes does not matter. The following fields are always required: -- `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. +* `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. -- `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. -- `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. -- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. -- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. +* `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. +* `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. +* `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. +* `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevalance of smoking, represented as a property called `cigaretteSmoker` of persons being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: + * Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: ![domain incudes](/assets/images/custom_dc/customdc_screenshot9.png){: width="800"} - - Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. + * Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:`, which are interchangeable. All fields that do not reference another node must be in quotation marks. The following fields are optional: -- `description`: A more detailed textual description of the variable. -- `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. -- `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. -- `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. +* `description`: A more detailed textual description of the variable. +* `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +* `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. +* `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. {: #naming} + #### Variable DCID naming conventions -- Variable DCIDs should be in PascalCase with underscores between properties. -- For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: +* Variable DCIDs should be in PascalCase with underscores between properties. +* For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: _`statType_measuredProperty_populationType_constraintValue1_constraintValue2`_ Example: `GrowthRate_Amount_EconomicActivity_GrossDomesticProduction` -- If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` -- For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: - - `Annual_Average_RetailPrice_Electricity` - - `Annual_Average_Wage` -- For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: - - `Count_Death_Female_AsAFractionOf_Count_Person_Female` - - `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` -- Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. +* If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` +* For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: + * `Annual_Average_RetailPrice_Electricity` + * `Annual_Average_Wage` +* For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: + * `Count_Death_Female_AsAFractionOf_Count_Person_Female` + * `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` +* Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. ### Step 2 (optional): Define a statistical variable group {#statvar-group} @@ -275,12 +287,13 @@ name: "WHO" specializationOf: dcid:dc/g/Root ``` + You can define as many statistical variable group nodes as you like. Each must include the following fields: -- `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. -- `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. -- `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. -- `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: +* `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. +* `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. +* `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. +* `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: ``` Node: dcid:who/g/WHO @@ -324,6 +337,7 @@ memberOf: dcid:MyVariables ``` {: #exp_csv} + ### Step 3: Prepare the CSV observation files CSV files contain the following columns using the following headings: @@ -333,19 +347,19 @@ CSV files contain the following columns using the following headings: The columns can be in any order, and you can specify custom names for the headings and use the `columnMappings` field in the JSON file to map them accordingly (see below for details). These columns are required: -- `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. -- `variable`: The DCID of an existing variable or the node you have defined in the MCF -- `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. -- `value`: See [Observation values](#obs) for valid values of this column. +* `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. +* `variable`: The DCID of an existing variable or the node you have defined in the MCF +* `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. +* `value`: See [Observation values](#obs) for valid values of this column. > **Note:** The type of the entities in a single file should be unique; do not mix multiple entity types in the same CSV file. For example, if you have observations for cities and counties, put all the city data in one CSV file and all the county data in another one. These columns are optional, and allow you to specify additional per-observation properties: -- [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. -- [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. -- [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. -- [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. +* [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. +* [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. +* [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. +* [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. Here is an example of some real-world data from the WHO on the prevalance of smoking in adult populations, broken down by sex, in the correct CSV format: @@ -369,18 +383,19 @@ In this case, the columns need to be mapped to the expected columns listed above #### Observation values {#obs} Here are the rules for observation values: -- Variable values must be numeric. Do not include any special characters such as `*` or `#`. -- Zeros are accepted and recorded. -- For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. -- Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. +* Variable values must be numeric. Do not include any special characters such as `*` or `#`. +* Zeros are accepted and recorded. +* For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. +* Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. {: #json} + ### Step 4: Write the JSON config file You must define a `config.json` in the top-level directory where your CSV files are located. You need to provide these specifications: -- The input files location and entity type -- The sources and provenances of the data -- Column mappings, if you are using custom names for the column headings +* The input files location and entity type +* The sources and provenances of the data +* Column mappings, if you are using custom names for the column headings Here is an example of how the config file would look for the CSV file we defined above. More details are below. @@ -411,18 +426,19 @@ Here is an example of how the config file would look for the CSV file we defined ``` The following fields are required: -- `input_files`: - - `format` must be `variablePerRow` - - `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. +* `input_files`: + * `format` must be `variablePerRow` + * `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. The following is optional: -- `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. +* `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. Note that you don't specify your MCF files as input files; the Data Commons importer will identify them automatically. The other fields are explained in the [Data config file specification reference](config.md). {: #loadlocal} + ## Load local custom data The following procedures show you how to load and serve your custom data locally. @@ -430,11 +446,12 @@ The following procedures show you how to load and serve your custom data locally To load data in Google Cloud, see instead [Load data in Google Cloud](/custom_dc/deploy_cloud.html) for procedures. {: #env} + ### Configure environment variables Edit the `env.list` file you created [previously](/custom_dc/quickstart.html#env-vars) as follows: -- Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. -- Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. +* Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. +* Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. ### Start the Docker containers with local custom data {#docker-data} @@ -465,7 +482,7 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + @@ -473,6 +490,7 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script > **Note:** Any time you make changes to the CSV or JSON files and want to reload the data, you need to restart both containers. {:.no_toc} + #### (Optional) Start the data management container in schema update mode {#schema-update-mode} If you have tried to start a container, and have received a `SQL check failed` error, this indicates that a database schema update is needed. You need to restart the data management container, and you can specify an additional, optional, flag. This mode updates the database schema without re-importing data or re-building natural language embeddings. This is the quickest way to resolve a SQL check failed error during services container startup. @@ -503,12 +521,13 @@ If you have tried to start a container, and have received a `SQL check failed` e -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + {: #verify} + ### Verify your data If the servers have started up without errors, check to ensure that your data is showing up as expected. @@ -516,6 +535,7 @@ If the servers have started up without errors, check to ensure that your data is 1. Verify statistical variables: go to the [Statistical Variable Explorer](https://localhost:8080/tools/statvar){: target="_blank"} to verify that your statistical variables are showing up correctly. You should see something like this: ![](/assets/images/custom_dc/customdc_screenshot11.png){: width="400"} + 1. Click on a variable name to get more information on the right panel. 1. Verify that your observations are loaded: Click on an **Example Place** link to open the detailed page for that place. Scroll to the bottom, where you should see a timeline graph of observations for the selected place. 1. Verify natural-language querying: go to the [Search page](https://localhost:8080/tools/explore){: target="_blank"} and enter a query related to your data. You should get relevant graphs using your data. @@ -536,6 +556,7 @@ At the prompt, enter SQL queries. For example, for the sample OECD data, this qu ```shell sqlite> select * from observations limit 10; ``` + returns output like this: ```shell @@ -550,4 +571,4 @@ country/BEL|average_annual_wage|2005|55662.21541|c/p/1 To exit the sqlite shell, press `Ctrl-D`. - \ No newline at end of file + From 906d7947e2a5802fcded6c48e145ce959a229eb2 Mon Sep 17 00:00:00 2001 From: kmoscoe <165203920+kmoscoe@users.noreply.github.com> Date: Mon, 6 Jul 2026 13:58:17 -0700 Subject: [PATCH 05/16] Remove nav entry for Python V1 API and redirect (#718) * Add description of orderedFacets fields * Fix date types * Remove nav entry for V1 Python API * Revert "Remove nav entry for V1 Python API" This reverts commit 57bbf7fb32e39861ed625f01fd2813c81e73a1af. * Undelete file and make it a redirect * Fix URL * Change redirect to redirect from and delete file. --- api/python/api_key.md | 28 ---------------------------- api/python/index.md | 11 ----------- api/python/v2/index.md | 2 ++ 3 files changed, 2 insertions(+), 39 deletions(-) delete mode 100644 api/python/api_key.md delete mode 100644 api/python/index.md diff --git a/api/python/api_key.md b/api/python/api_key.md deleted file mode 100644 index 99331cb45..000000000 --- a/api/python/api_key.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -layout: default -title: Set API Key -parent: Python -grand_parent: API -published: false ---- - -# Sets the API key (Optional) - -## `datacommons.set_api_key(api_key)` - -Sets an environment variable `"DC_API_KEY"` to given `api_key`. - -An API key can be provided to -the API after importing the library, or set as an environment variable -`"DC_API_KEY"`. - -For more details about how to get an API key and provide it to the Python -Client API, please visit the [Python library setup guide](/api/python/) -for more details. - -Setting the API key is optional. Data Commons *does not charge* users, but uses the -API key for understanding API usage. - -**Arguments** - -* `api_key (str)` - The API key. diff --git a/api/python/index.md b/api/python/index.md deleted file mode 100644 index 0dd24819a..000000000 --- a/api/python/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -layout: default -title: Python (V1) -nav_order: 30 -parent: API - Query data programmatically -has_children: true ---- - -# Data Commons Python API - -> This version of the Data Commons Python API is deprecated. Please migrate your applications to [V2](/api/python/v2). For help on translating your requests, see the [Migration guide](/api/python/v2/migration.html). \ No newline at end of file diff --git a/api/python/v2/index.md b/api/python/v2/index.md index 47fd4ddf2..288bb2b1c 100644 --- a/api/python/v2/index.md +++ b/api/python/v2/index.md @@ -5,6 +5,8 @@ nav_order: 1 parent: API - Query data programmatically has_children: true published: true +redirect_from: + /api/python/index --- {:.no_toc} From 1987c1b6d6100537e21753fa000572744ae87151 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Wed, 8 Jul 2026 10:55:13 -0700 Subject: [PATCH 06/16] Rename variables to conform to conventions --- custom_dc/custom_data.md | 12 +++++----- custom_dc/custom_entities.md | 46 ++++++++++++++++++------------------ 2 files changed, 29 insertions(+), 29 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index 3f8f44451..2d1de5c0c 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -269,15 +269,15 @@ By default, existing variables are shown in the Statistical Variable Explorer in Here is an example that defines a single group node with the heading "WHO" and assigns all 3 statistical variables to the same group. ``` -Node: dcid:who/Adult_curr_cig_smokers +Node: dcid:who/Percent_Smokers_Adult ... memberOf: dcid:who/g/WHO -Node: dcid:who/Adult_curr_cig_smokers_female +Node: dcid:who/Percent_Smokers_Adult_Female ... memberOf:dcid:who/g/WHO -Node: dcid:who/Adult_curr_cig_smokers_male +Node: dcid:who/Percent_Smokers_Adult_Male ... memberOf: dcid:who/g/WHO @@ -310,15 +310,15 @@ You can define as many statistical variable group nodes as you like. Each must i You can also assign a variable to as many group nodes as you like: simply specify a comma-separated list of group DCIDs in the `memberOf`. For example, to assign the 3 variables to both groups: ``` -Node: dcid:who/Adult_curr_cig_smokers +Node: dcid:who/Percent_Smokers_Adult ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking -Node: dcid:who/Adult_curr_cig_smokers_female +Node: dcid:who/Percent_Smokers_Adult_Female ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking -Node: dcid:who/Adult_curr_cig_smokers_male +Node: dcid:who/Percent_Smokers_Adult_Male ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking ``` diff --git a/custom_dc/custom_entities.md b/custom_dc/custom_entities.md index 6f332238f..ab6f781d3 100644 --- a/custom_dc/custom_entities.md +++ b/custom_dc/custom_entities.md @@ -131,12 +131,12 @@ The other fields are explained in the [Data config file specification reference] If you are providing observations for the non-place entities, the observations must be in a separate file. You'll need a different CSV file for each entity type for which you are providing observations. For example, let's say you've already defined in MCF the following variables that measure weekly hospital capacity: -* `total_count_staffed_beds` -* `count_staffed_adult_beds` -* `count_staffed_inpatient_icu_beds` -* `count_staffed_adult_inpatient_icu_beds` -* `count_staffed_inpatient_icu_beds_occupied` -* `count_staffed_adult_icu_beds_occupied` +* `Count_StaffedBeds` +* `Count_StaffedBeds_Adult` +* `Count_StaffedBeds_Inpatient_ICU` +* `Count_StaffedBeds_Adult_Inpatient_ICU` +* `Count_StaffedBedsOccupied_Inpatient_ICU` +* `Count_StaffedBedsOccupied_Adult_ICU_beds` Aside: Note that the thing being measured here is "beds". There is an existing [Bed](https://datacommons.org/browser/Bed) class in Data Commons. So when defining such variables, you would specify `schema:bed` as the `populationType`. @@ -144,23 +144,23 @@ Just like for place entities, you provide observations for these variables in a ```csv entity,date,variable,value -20001,2023-01-27,count_staffed_adult_beds,1048 -20001,2023-01-27,count_staffed_adult_icu_beds_occupied,146 -20001,2023-01-27,count_staffed_adult_inpatient_icu_beds,146 -20001,2023-01-27,count_staffed_inpatient_icu_beds,264 -20001,2023-01-27,count_staffed_inpatient_icu_beds_occupied,264 -20001,2023-01-27,total_count_staffed_beds,1262 -20017,2023-01-27,count_staffed_adult_beds,0 -20017,2023-01-27,count_staffed_adult_icu_beds_occupied,0 -20017,2023-01-27,count_staffed_adult_inpatient_icu_beds, -20017,2023-01-27,count_staffed_inpatient_icu_beds, -20017,2023-01-27,count_staffed_inpatient_icu_beds_occupied,0 -21301,2023-01-27,count_staffed_adult_beds,780 -21301,2023-01-27,count_staffed_adult_icu_beds_occupied,62 -21301,2023-01-27,count_staffed_adult_inpatient_icu_beds,62 -21301,2023-01-27,count_staffed_inpatient_icu_beds,101 -21301,2023-01-27,count_staffed_inpatient_icu_beds_occupied,66 -21301,2023-01-27,total_count_staffed_beds,836 +20001,2023-01-27,Count_StaffedBeds,1048 +20001,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU_beds,146 +20001,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,146 +20001,2023-01-27,Count_StaffedBeds_Inpatient_ICU,264 +20001,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,264 +20001,2023-01-27,Count_StaffedBeds,1262 +20017,2023-01-27,Count_StaffedBeds_Adult,0 +20017,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU_beds,0 +20017,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU, +20017,2023-01-27,Count_StaffedBeds_Inpatient_ICU, +20017,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,0 +21301,2023-01-27,Count_StaffedBeds_Adult,780 +21301,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU_beds,62 +21301,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,62 +21301,2023-01-27,Count_StaffedBeds_Inpatient_ICU,101 +21301,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,66 +21301,2023-01-27,Count_StaffedBeds,836 ... ``` We could also have added an `observationPeriod` column, which would be set to `P7D` for all rows. From f1724282cb42ea1bc93712065d29083bf3584e04 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Wed, 8 Jul 2026 11:38:18 -0700 Subject: [PATCH 07/16] Revert "Fix typo" This reverts commit 2c53d1c8d52c860eeb02d607cbf08ba789cba0b4. --- custom_dc/custom_data.md | 159 +++++++++++++++++---------------------- 1 file changed, 69 insertions(+), 90 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index 2d1de5c0c..8569787ab 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -6,7 +6,6 @@ parent: Build your own Data Commons --- {:.no_toc} - # Prepare and load your own data This page shows you how to format and load your own custom data into your local instance. This is step 2 of the [recommended workflow](/custom_dc/index.html#workflow). @@ -22,14 +21,13 @@ Custom Data Commons requires that you provide your data in a specific schema, fo At a high level, you need to provide the following: -* If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. -* All observations data must be in CSV format, using the schema described later. -* You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. +- If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. +- All observations data must be in CSV format, using the schema described later. +- You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. If you need to define new entities, please see [Define custom entities](custom_entities.md) for details. {: #dir} - ### Files and directory structure You can have as many CSV and MCF files as you like, and they can be in multiple subdirectories (with an additional [configuration option](#subdirs)). There must only be one JSON config file, in the top-level input directory. For example: @@ -45,28 +43,26 @@ my_data/ ├── datafile3.csv └── datafile4.csv ``` - The top-level directory (e.g. `my_data`) can live anywhere in the file system; you will specify the full path to it when you [configure your input directory](#env). When you set up your files in Google Cloud Storage using the Terraform script, it will automatically create a top-level directory in your bucket called `input`. -The following sections walk you through the process of setting up your data. +The following sections walk you through the process of setting up your data. ## Prerequisite steps The following sections describe the high-level conceptual work you need to do before starting to write your data and config files. {: entities} - ### Step 0.1: Determine whether you need new entities or entity types Data Commons is optimized to support aggregations of data at geographical levels, such as city, state, country, and so on. If your data is aggregated by place, these are supported as entities out of the box. If, however, you want to aggregate data for entities that are _not_ places, then you may need to define new entities, and possibly even entity types. -In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. +In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. #### Entities and entity types Schema.org and the base Data Commons knowledge graph define entity types for just about everything in the world. An _entity type_ is a high-level concept, and is derived directly from a [`Class`](https://datacommons.org/browser/Class){: target="_blank"} type. Non-place entities are of two types: -* The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. -* The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. +- The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. +- The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. It is rare that you would need to create a new entity type, unless you are working in a highly specialized domain. An _entity_ is an instance of an entity type. For example, for `PublicSchool`, base Data Commons has many U.S. schools in its knowledge graph, such as [`nces/010162001665`](https://datacommons.org/browser/nces/010162001665){: target="_blank"} (Adams Elementary School) or [`nces/010039000201`](https://datacommons.org/browser/nces/010039000201){: target="_blank"} (Wylam Elementary School). Base Data Commons contains thousands of places and other entities, but it's possible that it does not have specific entities that you need. For example, it has about 100 instances of `Company`, but you may want data for other companies besides those. As another example, let's say your organization wants to collect (possibly private) data about different divisions or departments of your org; in this case you would need to define entities for them. @@ -74,40 +70,35 @@ An _entity_ is an instance of an entity type. For example, for `PublicSchool`, b > **Note:** You should always reuse existing entity types and entities from base Data Commons rather than re-defining them. This way, you get all the properties already defined for those entities and all their linked nodes, and can more easily join with base data if needed. {: #search} - #### Search for an existing entity / entity type -Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. +Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. To search using the REST APIs: -1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. -1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: +1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: 
https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=ENTITY_TYPE&property=<-typeOf
_ENTITY_TYPE_ is the DCID you've obtained in the previous step, such as `Hospital` or `PublicSchool`. For example: - ``` https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=PublicSchool&property=<-typeOf ``` - 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. To search using the Python APIs: 1. Start your Python interactive environment and [create a client for the base Data Commons](/api/python/v2/index.html). -1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. 1. Use the `fetch_property_values` method to find all the instances of the type: 
client.node.fetch_property_values(node_dcids="ENTITY_TYPE", properties="typeOf", out=False)
_ENTITY_TYPE_ is the DCID you've obtained in the previous step. For example: - ``` client.node.fetch_property_values(node_dcids="PublicSchool", properties="typeOf", out=False) ``` - 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. ### Step 0.2: Identify your statistical variables @@ -134,12 +125,12 @@ If you do need to define new variables, they must follow a certain model. The va | San Jose | 2023 | private | secondary | 100 | The measure here is a simple count; the set of things is "schools"; and the constraints are the type and levels of the schools, namely "public", "private", "elementary", "middle" and "secondary". All of these things must be encoded as separate variables. Therefore, although the _properties_ of school type and school level may already be defined in the Data Commons knowledge graph (or you may need to define them), they _cannot_ be present as columns in the CSV files that you store in Data Commons. Instead, you must create separate "count" variables to represent each case. In our example, you would actually need 6 different variables: -* `Count_School_Public_Elementary` -* `Count_School_Public_Middle` -* `Count_School_Public_Secondary` -* `Count_School_Private_Elementary` -* `Count_School_Private_Middle` -* `Count_School_Private_Secondary` +- `Count_School_Public_Elementary` +- `Count_School_Public_Middle` +- `Count_School_Public_Secondary` +- `Count_School_Private_Elementary` +- `Count_School_Private_Middle` +- `Count_School_Private_Secondary` If you wanted totals or subtotals of combinations, you would need to create additional variables for these as well. @@ -168,25 +159,24 @@ Data Commons uses a schema that is called "variable-per-row". This means that ev The names and order of the columns aren't important, as you can map them to the expected columns in the JSON file. However, the city and variable names must be existing DCIDs. If such DCIDs don't already exist in the base Data Commons, you must provide definitions of them in MCF files. -> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. +> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. ## Prepare your data In this section, we will walk you through a concrete example of how to go about setting up your MCF, CSV, and JSON files. {: #mcf} - ### Step 1: Define statistical variables in MCF If you are only reusing existing variables, you can skip this step entirely. -Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. +Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. > **Note:** You cannot "override" a variable definition by changing the value of existing fields. If you need to override the values of existing fields, you should create a new variable, with a new DCID. You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container. -Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. +Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. ``` Node: dcid:who/Percent_Smokers_Adults @@ -215,52 +205,50 @@ measuredProperty: dcid:cigaretteSmoker statType: dcid:percent measurementDenominator: dcid:Count_Person_Male ``` - The order of nodes and fields within nodes does not matter. The following fields are always required: -* `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. +- `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. -* `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. -* `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. -* `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. -* `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevalance of smoking, represented as a property called `cigaretteSmoker` of persons being measured. +- `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. +- `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. +- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. +- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - * Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: + - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: ![domain incudes](/assets/images/custom_dc/customdc_screenshot9.png){: width="800"} - * Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. + - Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:`, which are interchangeable. All fields that do not reference another node must be in quotation marks. The following fields are optional: -* `description`: A more detailed textual description of the variable. -* `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. -* `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. -* `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. +- `description`: A more detailed textual description of the variable. +- `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +- `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. +- `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. {: #naming} - #### Variable DCID naming conventions -* Variable DCIDs should be in PascalCase with underscores between properties. -* For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: +- Variable DCIDs should be in PascalCase with underscores between properties. +- For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: _`statType_measuredProperty_populationType_constraintValue1_constraintValue2`_ Example: `GrowthRate_Amount_EconomicActivity_GrossDomesticProduction` -* If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` -* For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: - * `Annual_Average_RetailPrice_Electricity` - * `Annual_Average_Wage` -* For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: - * `Count_Death_Female_AsAFractionOf_Count_Person_Female` - * `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` -* Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. +- If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` +- For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: + - `Annual_Average_RetailPrice_Electricity` + - `Annual_Average_Wage` +- For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: + - `Count_Death_Female_AsAFractionOf_Count_Person_Female` + - `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` +- Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. ### Step 2 (optional): Define a statistical variable group {#statvar-group} @@ -287,13 +275,12 @@ name: "WHO" specializationOf: dcid:dc/g/Root ``` - You can define as many statistical variable group nodes as you like. Each must include the following fields: -* `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. -* `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. -* `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. -* `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: +- `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. +- `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. +- `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. +- `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: ``` Node: dcid:who/g/WHO @@ -337,7 +324,6 @@ memberOf: dcid:MyVariables ``` {: #exp_csv} - ### Step 3: Prepare the CSV observation files CSV files contain the following columns using the following headings: @@ -347,19 +333,19 @@ CSV files contain the following columns using the following headings: The columns can be in any order, and you can specify custom names for the headings and use the `columnMappings` field in the JSON file to map them accordingly (see below for details). These columns are required: -* `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. -* `variable`: The DCID of an existing variable or the node you have defined in the MCF -* `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. -* `value`: See [Observation values](#obs) for valid values of this column. +- `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. +- `variable`: The DCID of an existing variable or the node you have defined in the MCF +- `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. +- `value`: See [Observation values](#obs) for valid values of this column. > **Note:** The type of the entities in a single file should be unique; do not mix multiple entity types in the same CSV file. For example, if you have observations for cities and counties, put all the city data in one CSV file and all the county data in another one. These columns are optional, and allow you to specify additional per-observation properties: -* [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. -* [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. -* [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. -* [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. +- [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. +- [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. +- [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. +- [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. Here is an example of some real-world data from the WHO on the prevalance of smoking in adult populations, broken down by sex, in the correct CSV format: @@ -383,19 +369,18 @@ In this case, the columns need to be mapped to the expected columns listed above #### Observation values {#obs} Here are the rules for observation values: -* Variable values must be numeric. Do not include any special characters such as `*` or `#`. -* Zeros are accepted and recorded. -* For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. -* Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. +- Variable values must be numeric. Do not include any special characters such as `*` or `#`. +- Zeros are accepted and recorded. +- For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. +- Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. {: #json} - ### Step 4: Write the JSON config file You must define a `config.json` in the top-level directory where your CSV files are located. You need to provide these specifications: -* The input files location and entity type -* The sources and provenances of the data -* Column mappings, if you are using custom names for the column headings +- The input files location and entity type +- The sources and provenances of the data +- Column mappings, if you are using custom names for the column headings Here is an example of how the config file would look for the CSV file we defined above. More details are below. @@ -426,19 +411,18 @@ Here is an example of how the config file would look for the CSV file we defined ``` The following fields are required: -* `input_files`: - * `format` must be `variablePerRow` - * `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. +- `input_files`: + - `format` must be `variablePerRow` + - `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. The following is optional: -* `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. +- `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. Note that you don't specify your MCF files as input files; the Data Commons importer will identify them automatically. The other fields are explained in the [Data config file specification reference](config.md). {: #loadlocal} - ## Load local custom data The following procedures show you how to load and serve your custom data locally. @@ -446,12 +430,11 @@ The following procedures show you how to load and serve your custom data locally To load data in Google Cloud, see instead [Load data in Google Cloud](/custom_dc/deploy_cloud.html) for procedures. {: #env} - ### Configure environment variables Edit the `env.list` file you created [previously](/custom_dc/quickstart.html#env-vars) as follows: -* Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. -* Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. +- Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. +- Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. ### Start the Docker containers with local custom data {#docker-data} @@ -482,7 +465,7 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + @@ -490,7 +473,6 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script > **Note:** Any time you make changes to the CSV or JSON files and want to reload the data, you need to restart both containers. {:.no_toc} - #### (Optional) Start the data management container in schema update mode {#schema-update-mode} If you have tried to start a container, and have received a `SQL check failed` error, this indicates that a database schema update is needed. You need to restart the data management container, and you can specify an additional, optional, flag. This mode updates the database schema without re-importing data or re-building natural language embeddings. This is the quickest way to resolve a SQL check failed error during services container startup. @@ -521,13 +503,12 @@ If you have tried to start a container, and have received a `SQL check failed` e -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + {: #verify} - ### Verify your data If the servers have started up without errors, check to ensure that your data is showing up as expected. @@ -535,7 +516,6 @@ If the servers have started up without errors, check to ensure that your data is 1. Verify statistical variables: go to the [Statistical Variable Explorer](https://localhost:8080/tools/statvar){: target="_blank"} to verify that your statistical variables are showing up correctly. You should see something like this: ![](/assets/images/custom_dc/customdc_screenshot11.png){: width="400"} - 1. Click on a variable name to get more information on the right panel. 1. Verify that your observations are loaded: Click on an **Example Place** link to open the detailed page for that place. Scroll to the bottom, where you should see a timeline graph of observations for the selected place. 1. Verify natural-language querying: go to the [Search page](https://localhost:8080/tools/explore){: target="_blank"} and enter a query related to your data. You should get relevant graphs using your data. @@ -556,7 +536,6 @@ At the prompt, enter SQL queries. For example, for the sample OECD data, this qu ```shell sqlite> select * from observations limit 10; ``` - returns output like this: ```shell @@ -571,4 +550,4 @@ country/BEL|average_annual_wage|2005|55662.21541|c/p/1 To exit the sqlite shell, press `Ctrl-D`. - + \ No newline at end of file From e69ced26121c4de6d8580b51f353cbe2b30a0f78 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Wed, 8 Jul 2026 11:43:57 -0700 Subject: [PATCH 08/16] fix capitalization --- custom_dc/custom_data.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index 8569787ab..51556f6c1 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -184,7 +184,7 @@ typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" populationType: dcid:Person measuredProperty: dcid:cigaretteSmoker -statType: dcid:percent +statType: dcid:Percent measurementDenominator: dcid:Count_Person Node: dcid:who/Percent_Smokers_Adult_Females @@ -193,7 +193,7 @@ name: "Prevalence of current cigarette smoking among adults (%) [Female]" populationType: dcid:Person gender: dcid:Female measuredProperty: dcid:cigaretteSmoker -statType: dcid:percent +statType: dcid:Percent measurementDenominator: dcid:Count_Person_Female Node: dcid:who/Percent_Smokers_Adult_Males @@ -202,7 +202,7 @@ name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Person gender: dcid:Male measuredProperty: dcid:cigaretteSmoker -statType: dcid:percent +statType: dcid:Percent measurementDenominator: dcid:Count_Person_Male ``` The order of nodes and fields within nodes does not matter. From d8834b884e9912e21a0511b4e5d10dc95b8e90e0 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Wed, 8 Jul 2026 11:50:37 -0700 Subject: [PATCH 09/16] Fix more names --- custom_dc/custom_data.md | 26 +++++++++++++------------- custom_dc/custom_entities.md | 28 ++++++++++++++-------------- 2 files changed, 27 insertions(+), 27 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index 51556f6c1..76a959098 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -179,7 +179,7 @@ You can define your statistical variables in a single MCF file, or split them in Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. ``` -Node: dcid:who/Percent_Smokers_Adults +Node: dcid:who/Percent_Smokers_Adult typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" populationType: dcid:Person @@ -187,7 +187,7 @@ measuredProperty: dcid:cigaretteSmoker statType: dcid:Percent measurementDenominator: dcid:Count_Person -Node: dcid:who/Percent_Smokers_Adult_Females +Node: dcid:who/Percent_Smokers_Adult_Female typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" populationType: dcid:Person @@ -196,7 +196,7 @@ measuredProperty: dcid:cigaretteSmoker statType: dcid:Percent measurementDenominator: dcid:Count_Person_Female -Node: dcid:who/Percent_Smokers_Adult_Males +Node: dcid:who/Percent_Smokers_Adult_Male typeOf: dcid:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Person @@ -351,16 +351,16 @@ Here is an example of some real-world data from the WHO on the prevalance of smo ```csv SERIES,GEOGRAPHY,TIME_PERIOD,OBS_VALUE -dcs:who/Percent_Smokers_Adult_Females,dcid:country/AFG,2019,1.2 -dcs:who/Percent_Smokers_Adult_Males,dcid:country/AFG,2019,13.4 -dcs:who/Percent_Smokers_Adults,dcid:country/AFG,2019,7.5 -dcs:who/Percent_Smokers_Adult_Females,dcid:country/AGO,2016,1.8 -dcs:who/Percent_Smokers_Adult_Males,dcid:country/AGO,2016,14.3 -dcs:who/Percent_Smokers_Adult_Females,dcid:country/ALB,2018,4.5 -dcs:who/Percent_Smokers_Adult_Males,dcid:country/ALB,2018,35.7 -dcs:who/Percent_Smokers_Adult_Males,dcid:country/ARE,2018,11.1 -dcs:who/Percent_Smokers_Adult_Females,dcid:country/ARE,2018,1.6 -dcs:who/Percent_Smokers_Adults,dcid:country/ARE,2018,6.3 +dcs:who/Percent_Smokers_Adult_Female,dcid:country/AFG,2019,1.2 +dcs:who/Percent_Smokers_Adult_Male,dcid:country/AFG,2019,13.4 +dcs:who/Percent_Smokers_Adult,dcid:country/AFG,2019,7.5 +dcs:who/Percent_Smokers_Adult_Female,dcid:country/AGO,2016,1.8 +dcs:who/Percent_Smokers_Adult_Male,dcid:country/AGO,2016,14.3 +dcs:who/Percent_Smokers_Adult_Female,dcid:country/ALB,2018,4.5 +dcs:who/Percent_Smokers_Adult_Male,dcid:country/ALB,2018,35.7 +dcs:who/Percent_Smokers_Adult_Male,dcid:country/ARE,2018,11.1 +dcs:who/Percent_Smokers_Adult_Female,dcid:country/ARE,2018,1.6 +dcs:who/Percent_Smokers_Adult,dcid:country/ARE,2018,6.3 ... ``` diff --git a/custom_dc/custom_entities.md b/custom_dc/custom_entities.md index ab6f781d3..88d9ffc94 100644 --- a/custom_dc/custom_entities.md +++ b/custom_dc/custom_entities.md @@ -145,18 +145,18 @@ Just like for place entities, you provide observations for these variables in a ```csv entity,date,variable,value 20001,2023-01-27,Count_StaffedBeds,1048 -20001,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU_beds,146 +20001,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,146 20001,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,146 20001,2023-01-27,Count_StaffedBeds_Inpatient_ICU,264 20001,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,264 20001,2023-01-27,Count_StaffedBeds,1262 20017,2023-01-27,Count_StaffedBeds_Adult,0 -20017,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU_beds,0 +20017,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,0 20017,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU, 20017,2023-01-27,Count_StaffedBeds_Inpatient_ICU, 20017,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,0 21301,2023-01-27,Count_StaffedBeds_Adult,780 -21301,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU_beds,62 +21301,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,62 21301,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,62 21301,2023-01-27,Count_StaffedBeds_Inpatient_ICU,101 21301,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,66 @@ -241,17 +241,17 @@ Then, if desired, you could provide aggregated observations for each hospital ty ```csv entity,date,variable,value -ShortTermHospital,2023-01-27,count_staffed_adult_beds,... -ShortTermHospital,2023-01-27,count_staffed_adult_icu_beds_occupied,... -ShortTermHospital,2023-01-27,count_staffed_adult_inpatient_icu_beds,... -ShortTermHospital,2023-01-27,count_staffed_inpatient_icu_beds,... -ShortTermHospital,2023-01-27,count_staffed_inpatient_icu_beds_occupied,... -ShortTermHospital,2023-01-27,total_count_staffed_beds,... -LongTermHospital,2023-01-27,count_staffed_adult_beds,... -LongTermHospital,2023-01-27,count_staffed_adult_icu_beds_occupied,... -LongTermHospital,2023-01-27,count_staffed_adult_inpatient_icu_beds... -LongTermHospital,2023-01-27,count_staffed_inpatient_icu_beds... -LongTermHospital,2023-01-27,count_staffed_inpatient_icu_beds_occupied,... +ShortTermHospital,2023-01-27,Count_StaffedBeds_Adult... +ShortTermHospital,2023-01-27,Count_StaffedBedsOccuped_Adult_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBeds_Inpatient_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBeds,... +LongTermHospital,2023-01-27,Count_StaffedBeds_Adult,... +LongTermHospital,2023-01-27,Count_StaffedBedsOccuped_Adult_ICU,... +LongTermHospital,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU... +LongTermHospital,2023-01-27,Count_StaffedBeds_Inpatient_ICU... +LongTermHospital,2023-01-27,ount_StaffedBedsOccupied_Inpatient_ICU,... ... ``` From c7a1fd863b595199f0fcf42f96e69aa1d21860fb Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Wed, 8 Jul 2026 13:17:26 -0700 Subject: [PATCH 10/16] Fix prefixes --- custom_dc/custom_data.md | 58 +++++++++++++++++++++------------------- 1 file changed, 31 insertions(+), 27 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index 76a959098..ae752b782 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -180,37 +180,37 @@ Here's an example of defining some statistical variables representing data in a ``` Node: dcid:who/Percent_Smokers_Adult -typeOf: dcid:StatisticalVariable +typeOf: dcs:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" -populationType: dcid:Person +populationType: schema:Person measuredProperty: dcid:cigaretteSmoker -statType: dcid:Percent -measurementDenominator: dcid:Count_Person +statType: dcs:Percent +measurementDenominator: dcs:Count_Person Node: dcid:who/Percent_Smokers_Adult_Female -typeOf: dcid:StatisticalVariable +typeOf: dcs:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" -populationType: dcid:Person -gender: dcid:Female +populationType: schema:Person +gender: dcs:Female measuredProperty: dcid:cigaretteSmoker -statType: dcid:Percent -measurementDenominator: dcid:Count_Person_Female +statType: dcs:Percent +measurementDenominator: dcs:Count_Person_Female Node: dcid:who/Percent_Smokers_Adult_Male -typeOf: dcid:StatisticalVariable +typeOf: dcs:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Person -gender: dcid:Male +gender: dcs:Male measuredProperty: dcid:cigaretteSmoker -statType: dcid:Percent -measurementDenominator: dcid:Count_Person_Male +statType: dcs:Percent +measurementDenominator: dcs:Count_Person_Male ``` The order of nodes and fields within nodes does not matter. The following fields are always required: -- `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. +- `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. It must be preceded by the prefix `dcid:`. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. -- `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. +- `typeOf`: In the case of statistical variable, this is always `dcs:StatisticalVariable`. - `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. - `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. - `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. @@ -221,11 +221,15 @@ The following fields are always required: - Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. -Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:`, which are interchangeable. All fields that do not reference another node must be in quotation marks. +Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:` or `schema:`. Use the following guidelines to determine which to use: +- If the node exists in [schema.org](https://schema.org/docs/schemas.html){: target="_blank"} (you can look them up in the **Term Finder**), use `schema`. +- If the node exists in the core Data Commons schema, [dcschema.mcf](https://github.com/datacommonsorg/schema/blob/main/core/dcschema.mcf){: target="blank"}, use `dcs`. +- Otherwise, use `dcid` for all others. +All fields that do not reference another node must be in quotation marks. The following fields are optional: - `description`: A more detailed textual description of the variable. -- `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +- `statType`: By default, if not specified, this is `dcis:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. - `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. - `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. @@ -351,16 +355,16 @@ Here is an example of some real-world data from the WHO on the prevalance of smo ```csv SERIES,GEOGRAPHY,TIME_PERIOD,OBS_VALUE -dcs:who/Percent_Smokers_Adult_Female,dcid:country/AFG,2019,1.2 -dcs:who/Percent_Smokers_Adult_Male,dcid:country/AFG,2019,13.4 -dcs:who/Percent_Smokers_Adult,dcid:country/AFG,2019,7.5 -dcs:who/Percent_Smokers_Adult_Female,dcid:country/AGO,2016,1.8 -dcs:who/Percent_Smokers_Adult_Male,dcid:country/AGO,2016,14.3 -dcs:who/Percent_Smokers_Adult_Female,dcid:country/ALB,2018,4.5 -dcs:who/Percent_Smokers_Adult_Male,dcid:country/ALB,2018,35.7 -dcs:who/Percent_Smokers_Adult_Male,dcid:country/ARE,2018,11.1 -dcs:who/Percent_Smokers_Adult_Female,dcid:country/ARE,2018,1.6 -dcs:who/Percent_Smokers_Adult,dcid:country/ARE,2018,6.3 +dcid:who/Percent_Smokers_Adult_Female,dcid:country/AFG,2019,1.2 +dcid:who/Percent_Smokers_Adult_Male,dcid:country/AFG,2019,13.4 +dcid:who/Percent_Smokers_Adult,dcid:country/AFG,2019,7.5 +dcid:who/Percent_Smokers_Adult_Female,dcid:country/AGO,2016,1.8 +dcid:who/Percent_Smokers_Adult_Male,dcid:country/AGO,2016,14.3 +dcid:who/Percent_Smokers_Adult_Female,dcid:country/ALB,2018,4.5 +dcid:who/Percent_Smokers_Adult_Male,dcid:country/ALB,2018,35.7 +dcid:who/Percent_Smokers_Adult_Male,dcid:country/ARE,2018,11.1 +dcid:who/Percent_Smokers_Adult_Female,dcid:country/ARE,2018,1.6 +dcid:who/Percent_Smokers_Adult,dcid:country/ARE,2018,6.3 ... ``` From 900afe9889f5ad269c3e63a3125bfd34e94c7737 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 9 Jul 2026 08:34:22 -0700 Subject: [PATCH 11/16] Changes from Caro --- custom_dc/custom_data.md | 42 ++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index ae752b782..cc232152a 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -179,7 +179,7 @@ You can define your statistical variables in a single MCF file, or split them in Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. ``` -Node: dcid:who/Percent_Smokers_Adult +Node: dcid:who/Percent_AdultSmokers typeOf: dcs:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" populationType: schema:Person @@ -187,7 +187,7 @@ measuredProperty: dcid:cigaretteSmoker statType: dcs:Percent measurementDenominator: dcs:Count_Person -Node: dcid:who/Percent_Smokers_Adult_Female +Node: dcid:who/Percent_AdultSmokers_Female typeOf: dcs:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" populationType: schema:Person @@ -196,7 +196,7 @@ measuredProperty: dcid:cigaretteSmoker statType: dcs:Percent measurementDenominator: dcs:Count_Person_Female -Node: dcid:who/Percent_Smokers_Adult_Male +Node: dcid:who/Percent_AdultSmokers_Male typeOf: dcs:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Person @@ -212,7 +212,7 @@ The following fields are always required: > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. - `typeOf`: In the case of statistical variable, this is always `dcs:StatisticalVariable`. - `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. -- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. +- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `schema:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. - `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: @@ -229,7 +229,7 @@ All fields that do not reference another node must be in quotation marks. The following fields are optional: - `description`: A more detailed textual description of the variable. -- `statType`: By default, if not specified, this is `dcis:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +- `statType`: By default, if not specified, this is `dcs:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. - `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. - `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. @@ -261,15 +261,15 @@ By default, existing variables are shown in the Statistical Variable Explorer in Here is an example that defines a single group node with the heading "WHO" and assigns all 3 statistical variables to the same group. ``` -Node: dcid:who/Percent_Smokers_Adult +Node: dcid:who/Percent_AdultSmokers ... memberOf: dcid:who/g/WHO -Node: dcid:who/Percent_Smokers_Adult_Female +Node: dcid:who/Percent_AdultSmokers_Female ... memberOf:dcid:who/g/WHO -Node: dcid:who/Percent_Smokers_Adult_Male +Node: dcid:who/Percent_AdultSmokers_Male ... memberOf: dcid:who/g/WHO @@ -301,15 +301,15 @@ You can define as many statistical variable group nodes as you like. Each must i You can also assign a variable to as many group nodes as you like: simply specify a comma-separated list of group DCIDs in the `memberOf`. For example, to assign the 3 variables to both groups: ``` -Node: dcid:who/Percent_Smokers_Adult +Node: dcid:who/Percent_AdultSmokers ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking -Node: dcid:who/Percent_Smokers_Adult_Female +Node: dcid:who/Percent_AdultSmokers_Female ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking -Node: dcid:who/Percent_Smokers_Adult_Male +Node: dcid:who/Percent_AdultSmokers_Male ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking ``` @@ -355,16 +355,16 @@ Here is an example of some real-world data from the WHO on the prevalance of smo ```csv SERIES,GEOGRAPHY,TIME_PERIOD,OBS_VALUE -dcid:who/Percent_Smokers_Adult_Female,dcid:country/AFG,2019,1.2 -dcid:who/Percent_Smokers_Adult_Male,dcid:country/AFG,2019,13.4 -dcid:who/Percent_Smokers_Adult,dcid:country/AFG,2019,7.5 -dcid:who/Percent_Smokers_Adult_Female,dcid:country/AGO,2016,1.8 -dcid:who/Percent_Smokers_Adult_Male,dcid:country/AGO,2016,14.3 -dcid:who/Percent_Smokers_Adult_Female,dcid:country/ALB,2018,4.5 -dcid:who/Percent_Smokers_Adult_Male,dcid:country/ALB,2018,35.7 -dcid:who/Percent_Smokers_Adult_Male,dcid:country/ARE,2018,11.1 -dcid:who/Percent_Smokers_Adult_Female,dcid:country/ARE,2018,1.6 -dcid:who/Percent_Smokers_Adult,dcid:country/ARE,2018,6.3 +dcid:who/Percent_AdultSmokers_Female,dcid:country/AFG,2019,1.2 +dcid:who/Percent_AdultSmokers_Male,dcid:country/AFG,2019,13.4 +dcid:who/Percent_AdultSmokers,dcid:country/AFG,2019,7.5 +dcid:who/Percent_AdultSmokers_Female,dcid:country/AGO,2016,1.8 +dcid:who/Percent_AdultSmokers_Male,dcid:country/AGO,2016,14.3 +dcid:who/Percent_AdultSmokers_Female,dcid:country/ALB,2018,4.5 +dcid:who/Percent_AdultSmokers_Male,dcid:country/ALB,2018,35.7 +dcid:who/Percent_AdultSmokers_Male,dcid:country/ARE,2018,11.1 +dcid:who/Percent_AdultSmokers_Female,dcid:country/ARE,2018,1.6 +dcid:who/Percent_AdultSmokers,dcid:country/ARE,2018,6.3 ... ``` From fe15ba74603b2a14af73dfebefd93fa2d831c0aa Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 9 Jul 2026 08:37:21 -0700 Subject: [PATCH 12/16] changes from Caro --- custom_dc/custom_data.md | 173 ++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 76 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index cc232152a..6fb042ebe 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -6,6 +6,7 @@ parent: Build your own Data Commons --- {:.no_toc} + # Prepare and load your own data This page shows you how to format and load your own custom data into your local instance. This is step 2 of the [recommended workflow](/custom_dc/index.html#workflow). @@ -21,13 +22,14 @@ Custom Data Commons requires that you provide your data in a specific schema, fo At a high level, you need to provide the following: -- If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. -- All observations data must be in CSV format, using the schema described later. -- You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. +* If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. +* All observations data must be in CSV format, using the schema described later. +* You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. If you need to define new entities, please see [Define custom entities](custom_entities.md) for details. {: #dir} + ### Files and directory structure You can have as many CSV and MCF files as you like, and they can be in multiple subdirectories (with an additional [configuration option](#subdirs)). There must only be one JSON config file, in the top-level input directory. For example: @@ -43,26 +45,28 @@ my_data/ ├── datafile3.csv └── datafile4.csv ``` + The top-level directory (e.g. `my_data`) can live anywhere in the file system; you will specify the full path to it when you [configure your input directory](#env). When you set up your files in Google Cloud Storage using the Terraform script, it will automatically create a top-level directory in your bucket called `input`. -The following sections walk you through the process of setting up your data. +The following sections walk you through the process of setting up your data. ## Prerequisite steps The following sections describe the high-level conceptual work you need to do before starting to write your data and config files. {: entities} + ### Step 0.1: Determine whether you need new entities or entity types Data Commons is optimized to support aggregations of data at geographical levels, such as city, state, country, and so on. If your data is aggregated by place, these are supported as entities out of the box. If, however, you want to aggregate data for entities that are _not_ places, then you may need to define new entities, and possibly even entity types. -In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. +In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. #### Entities and entity types Schema.org and the base Data Commons knowledge graph define entity types for just about everything in the world. An _entity type_ is a high-level concept, and is derived directly from a [`Class`](https://datacommons.org/browser/Class){: target="_blank"} type. Non-place entities are of two types: -- The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. -- The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. +* The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. +* The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. It is rare that you would need to create a new entity type, unless you are working in a highly specialized domain. An _entity_ is an instance of an entity type. For example, for `PublicSchool`, base Data Commons has many U.S. schools in its knowledge graph, such as [`nces/010162001665`](https://datacommons.org/browser/nces/010162001665){: target="_blank"} (Adams Elementary School) or [`nces/010039000201`](https://datacommons.org/browser/nces/010039000201){: target="_blank"} (Wylam Elementary School). Base Data Commons contains thousands of places and other entities, but it's possible that it does not have specific entities that you need. For example, it has about 100 instances of `Company`, but you may want data for other companies besides those. As another example, let's say your organization wants to collect (possibly private) data about different divisions or departments of your org; in this case you would need to define entities for them. @@ -70,35 +74,40 @@ An _entity_ is an instance of an entity type. For example, for `PublicSchool`, b > **Note:** You should always reuse existing entity types and entities from base Data Commons rather than re-defining them. This way, you get all the properties already defined for those entities and all their linked nodes, and can more easily join with base data if needed. {: #search} + #### Search for an existing entity / entity type -Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. +Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. To search using the REST APIs: -1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. -1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: +1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: 
https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=ENTITY_TYPE&property=<-typeOf
_ENTITY_TYPE_ is the DCID you've obtained in the previous step, such as `Hospital` or `PublicSchool`. For example: + ``` https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=PublicSchool&property=<-typeOf ``` + 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. To search using the Python APIs: 1. Start your Python interactive environment and [create a client for the base Data Commons](/api/python/v2/index.html). -1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. 1. Use the `fetch_property_values` method to find all the instances of the type: 
client.node.fetch_property_values(node_dcids="ENTITY_TYPE", properties="typeOf", out=False)
_ENTITY_TYPE_ is the DCID you've obtained in the previous step. For example: + ``` client.node.fetch_property_values(node_dcids="PublicSchool", properties="typeOf", out=False) ``` + 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. ### Step 0.2: Identify your statistical variables @@ -125,12 +134,12 @@ If you do need to define new variables, they must follow a certain model. The va | San Jose | 2023 | private | secondary | 100 | The measure here is a simple count; the set of things is "schools"; and the constraints are the type and levels of the schools, namely "public", "private", "elementary", "middle" and "secondary". All of these things must be encoded as separate variables. Therefore, although the _properties_ of school type and school level may already be defined in the Data Commons knowledge graph (or you may need to define them), they _cannot_ be present as columns in the CSV files that you store in Data Commons. Instead, you must create separate "count" variables to represent each case. In our example, you would actually need 6 different variables: -- `Count_School_Public_Elementary` -- `Count_School_Public_Middle` -- `Count_School_Public_Secondary` -- `Count_School_Private_Elementary` -- `Count_School_Private_Middle` -- `Count_School_Private_Secondary` +* `Count_School_Public_Elementary` +* `Count_School_Public_Middle` +* `Count_School_Public_Secondary` +* `Count_School_Private_Elementary` +* `Count_School_Private_Middle` +* `Count_School_Private_Secondary` If you wanted totals or subtotals of combinations, you would need to create additional variables for these as well. @@ -159,28 +168,29 @@ Data Commons uses a schema that is called "variable-per-row". This means that ev The names and order of the columns aren't important, as you can map them to the expected columns in the JSON file. However, the city and variable names must be existing DCIDs. If such DCIDs don't already exist in the base Data Commons, you must provide definitions of them in MCF files. -> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. +> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. ## Prepare your data In this section, we will walk you through a concrete example of how to go about setting up your MCF, CSV, and JSON files. {: #mcf} + ### Step 1: Define statistical variables in MCF If you are only reusing existing variables, you can skip this step entirely. -Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. +Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. > **Note:** You cannot "override" a variable definition by changing the value of existing fields. If you need to override the values of existing fields, you should create a new variable, with a new DCID. You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container. -Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. +Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. ``` Node: dcid:who/Percent_AdultSmokers -typeOf: dcs:StatisticalVariable +typeOf: schema:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" populationType: schema:Person measuredProperty: dcid:cigaretteSmoker @@ -188,7 +198,7 @@ statType: dcs:Percent measurementDenominator: dcs:Count_Person Node: dcid:who/Percent_AdultSmokers_Female -typeOf: dcs:StatisticalVariable +typeOf: schema:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" populationType: schema:Person gender: dcs:Female @@ -197,7 +207,7 @@ statType: dcs:Percent measurementDenominator: dcs:Count_Person_Female Node: dcid:who/Percent_AdultSmokers_Male -typeOf: dcs:StatisticalVariable +typeOf: schema:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Person gender: dcs:Male @@ -205,54 +215,56 @@ measuredProperty: dcid:cigaretteSmoker statType: dcs:Percent measurementDenominator: dcs:Count_Person_Male ``` + The order of nodes and fields within nodes does not matter. The following fields are always required: -- `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. It must be preceded by the prefix `dcid:`. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. +* `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. It must be preceded by the prefix `dcid:`. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. -- `typeOf`: In the case of statistical variable, this is always `dcs:StatisticalVariable`. -- `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. -- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `schema:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. -- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. +* `typeOf`: In the case of statistical variable, this is always `schema:StatisticalVariable`. +* `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. +* `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `schema:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. +* `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: + * Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: ![domain incudes](/assets/images/custom_dc/customdc_screenshot9.png){: width="800"} - - Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. + * Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:` or `schema:`. Use the following guidelines to determine which to use: -- If the node exists in [schema.org](https://schema.org/docs/schemas.html){: target="_blank"} (you can look them up in the **Term Finder**), use `schema`. -- If the node exists in the core Data Commons schema, [dcschema.mcf](https://github.com/datacommonsorg/schema/blob/main/core/dcschema.mcf){: target="blank"}, use `dcs`. -- Otherwise, use `dcid` for all others. +* If the node exists in [schema.org](https://schema.org/docs/schemas.html){: target="_blank"} (you can look them up in the **Term Finder**), use `schema`. +* If the node exists in the core Data Commons schema, [dcschema.mcf](https://github.com/datacommonsorg/schema/blob/main/core/dcschema.mcf){: target="blank"}, use `dcs`. +* Otherwise, use `dcid` for all others. All fields that do not reference another node must be in quotation marks. The following fields are optional: -- `description`: A more detailed textual description of the variable. -- `statType`: By default, if not specified, this is `dcs:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. -- `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. -- `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. +* `description`: A more detailed textual description of the variable. +* `statType`: By default, if not specified, this is `dcs:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +* `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. +* `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. {: #naming} + #### Variable DCID naming conventions -- Variable DCIDs should be in PascalCase with underscores between properties. -- For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: +* Variable DCIDs should be in PascalCase with underscores between properties. +* For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: _`statType_measuredProperty_populationType_constraintValue1_constraintValue2`_ Example: `GrowthRate_Amount_EconomicActivity_GrossDomesticProduction` -- If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` -- For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: - - `Annual_Average_RetailPrice_Electricity` - - `Annual_Average_Wage` -- For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: - - `Count_Death_Female_AsAFractionOf_Count_Person_Female` - - `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` -- Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. +* If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` +* For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: + * `Annual_Average_RetailPrice_Electricity` + * `Annual_Average_Wage` +* For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: + * `Count_Death_Female_AsAFractionOf_Count_Person_Female` + * `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` +* Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. ### Step 2 (optional): Define a statistical variable group {#statvar-group} @@ -279,12 +291,13 @@ name: "WHO" specializationOf: dcid:dc/g/Root ``` + You can define as many statistical variable group nodes as you like. Each must include the following fields: -- `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. -- `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. -- `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. -- `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: +* `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. +* `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. +* `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. +* `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: ``` Node: dcid:who/g/WHO @@ -323,11 +336,12 @@ name: "My variables" specializationOf: dcid:dc/g/Root Node: dcid:GenderIncomeInequality_Person_15OrMoreYears_WithIncome -typeOf: dcs:StatisticalVariable +typeOf: schema:StatisticalVariable memberOf: dcid:MyVariables ``` {: #exp_csv} + ### Step 3: Prepare the CSV observation files CSV files contain the following columns using the following headings: @@ -337,19 +351,19 @@ CSV files contain the following columns using the following headings: The columns can be in any order, and you can specify custom names for the headings and use the `columnMappings` field in the JSON file to map them accordingly (see below for details). These columns are required: -- `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. -- `variable`: The DCID of an existing variable or the node you have defined in the MCF -- `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. -- `value`: See [Observation values](#obs) for valid values of this column. +* `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. +* `variable`: The DCID of an existing variable or the node you have defined in the MCF +* `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. +* `value`: See [Observation values](#obs) for valid values of this column. > **Note:** The type of the entities in a single file should be unique; do not mix multiple entity types in the same CSV file. For example, if you have observations for cities and counties, put all the city data in one CSV file and all the county data in another one. These columns are optional, and allow you to specify additional per-observation properties: -- [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. -- [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. -- [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. -- [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. +* [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. +* [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. +* [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. +* [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. Here is an example of some real-world data from the WHO on the prevalance of smoking in adult populations, broken down by sex, in the correct CSV format: @@ -373,18 +387,19 @@ In this case, the columns need to be mapped to the expected columns listed above #### Observation values {#obs} Here are the rules for observation values: -- Variable values must be numeric. Do not include any special characters such as `*` or `#`. -- Zeros are accepted and recorded. -- For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. -- Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. +* Variable values must be numeric. Do not include any special characters such as `*` or `#`. +* Zeros are accepted and recorded. +* For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. +* Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. {: #json} + ### Step 4: Write the JSON config file You must define a `config.json` in the top-level directory where your CSV files are located. You need to provide these specifications: -- The input files location and entity type -- The sources and provenances of the data -- Column mappings, if you are using custom names for the column headings +* The input files location and entity type +* The sources and provenances of the data +* Column mappings, if you are using custom names for the column headings Here is an example of how the config file would look for the CSV file we defined above. More details are below. @@ -415,18 +430,19 @@ Here is an example of how the config file would look for the CSV file we defined ``` The following fields are required: -- `input_files`: - - `format` must be `variablePerRow` - - `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. +* `input_files`: + * `format` must be `variablePerRow` + * `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. The following is optional: -- `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. +* `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. Note that you don't specify your MCF files as input files; the Data Commons importer will identify them automatically. The other fields are explained in the [Data config file specification reference](config.md). {: #loadlocal} + ## Load local custom data The following procedures show you how to load and serve your custom data locally. @@ -434,11 +450,12 @@ The following procedures show you how to load and serve your custom data locally To load data in Google Cloud, see instead [Load data in Google Cloud](/custom_dc/deploy_cloud.html) for procedures. {: #env} + ### Configure environment variables Edit the `env.list` file you created [previously](/custom_dc/quickstart.html#env-vars) as follows: -- Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. -- Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. +* Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. +* Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. ### Start the Docker containers with local custom data {#docker-data} @@ -469,7 +486,7 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + @@ -477,6 +494,7 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script > **Note:** Any time you make changes to the CSV or JSON files and want to reload the data, you need to restart both containers. {:.no_toc} + #### (Optional) Start the data management container in schema update mode {#schema-update-mode} If you have tried to start a container, and have received a `SQL check failed` error, this indicates that a database schema update is needed. You need to restart the data management container, and you can specify an additional, optional, flag. This mode updates the database schema without re-importing data or re-building natural language embeddings. This is the quickest way to resolve a SQL check failed error during services container startup. @@ -507,12 +525,13 @@ If you have tried to start a container, and have received a `SQL check failed` e -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + {: #verify} + ### Verify your data If the servers have started up without errors, check to ensure that your data is showing up as expected. @@ -520,6 +539,7 @@ If the servers have started up without errors, check to ensure that your data is 1. Verify statistical variables: go to the [Statistical Variable Explorer](https://localhost:8080/tools/statvar){: target="_blank"} to verify that your statistical variables are showing up correctly. You should see something like this: ![](/assets/images/custom_dc/customdc_screenshot11.png){: width="400"} + 1. Click on a variable name to get more information on the right panel. 1. Verify that your observations are loaded: Click on an **Example Place** link to open the detailed page for that place. Scroll to the bottom, where you should see a timeline graph of observations for the selected place. 1. Verify natural-language querying: go to the [Search page](https://localhost:8080/tools/explore){: target="_blank"} and enter a query related to your data. You should get relevant graphs using your data. @@ -540,6 +560,7 @@ At the prompt, enter SQL queries. For example, for the sample OECD data, this qu ```shell sqlite> select * from observations limit 10; ``` + returns output like this: ```shell @@ -554,4 +575,4 @@ country/BEL|average_annual_wage|2005|55662.21541|c/p/1 To exit the sqlite shell, press `Ctrl-D`. - \ No newline at end of file + From 2038c761c68ebded375f23d3806355b9f094cc87 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 9 Jul 2026 08:40:46 -0700 Subject: [PATCH 13/16] Regenerate llms-full --- llms-full.txt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/llms-full.txt b/llms-full.txt index a00bd4042..17869bf93 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -35,7 +35,7 @@ {'title': 'Data coverage', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/datasets/index.md', 'desc': 'Overview of datasets in Data Commons and per-country coverage.'} {'title': 'Place types', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/place_types.md', 'desc': 'Reference for place types in Data Commons.'} {'title': 'Get support', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/support.md', 'desc': 'Support channels and feedback paths.'} -{'title': 'Contribute to Data Commons', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/contributing.md', 'desc': {}} +{'title': 'Contribute to Data Commons', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/contributing.md', 'desc': 'Contribute data, code, or documentation to Data Commons.'} {'title': 'Web Components', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/api/web_components/index.md', 'desc': 'Embed Data Commons charts in your own website.'} {'title': 'Map chart', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/api/web_components/map.md', 'desc': {}} {'title': 'Bar chart', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/api/web_components/bar.md', 'desc': {}} @@ -5275,6 +5275,8 @@ nav_order: 1 parent: API - Query data programmatically has_children: true published: true +redirect_from: + /api/python/index --- {:.no_toc} @@ -14988,7 +14990,7 @@ If you are using a Custom Data Commons instance, make sure to indicate that the ## Email support forum -For technical questions that you can't find answers to in this documentation, you can email support@datacommons.org.404: Not Found--- +For technical questions that you can't find answers to in this documentation, you can email support@datacommons.org.404: Not Found--- layout: default title: Embed data and visualizations in your own website nav_order: 60 From 2ce21fc185f55f2893a8bf0423c6ec4db958f5cb Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 9 Jul 2026 08:43:21 -0700 Subject: [PATCH 14/16] Revert "Fix typo" This reverts commit 2c53d1c8d52c860eeb02d607cbf08ba789cba0b4. --- custom_dc/custom_data.md | 141 +++++++++++++++++---------------------- 1 file changed, 60 insertions(+), 81 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index 6fb042ebe..0230cb6b1 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -6,7 +6,6 @@ parent: Build your own Data Commons --- {:.no_toc} - # Prepare and load your own data This page shows you how to format and load your own custom data into your local instance. This is step 2 of the [recommended workflow](/custom_dc/index.html#workflow). @@ -22,14 +21,13 @@ Custom Data Commons requires that you provide your data in a specific schema, fo At a high level, you need to provide the following: -* If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. -* All observations data must be in CSV format, using the schema described later. -* You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. +- If you need to define your own statistical variables (metrics), you need to provide [MCF (Meta Content Framework)](https://en.wikipedia.org/wiki/Meta_Content_Framework){: target="_blank"} files. +- All observations data must be in CSV format, using the schema described later. +- You must also provide a JSON configuration file, named `config.json`, that specifies how to map and resolve the CSV contents to the Data Commons schema knowledge graph. The contents of the JSON file are described below. If you need to define new entities, please see [Define custom entities](custom_entities.md) for details. {: #dir} - ### Files and directory structure You can have as many CSV and MCF files as you like, and they can be in multiple subdirectories (with an additional [configuration option](#subdirs)). There must only be one JSON config file, in the top-level input directory. For example: @@ -45,28 +43,26 @@ my_data/ ├── datafile3.csv └── datafile4.csv ``` - The top-level directory (e.g. `my_data`) can live anywhere in the file system; you will specify the full path to it when you [configure your input directory](#env). When you set up your files in Google Cloud Storage using the Terraform script, it will automatically create a top-level directory in your bucket called `input`. -The following sections walk you through the process of setting up your data. +The following sections walk you through the process of setting up your data. ## Prerequisite steps The following sections describe the high-level conceptual work you need to do before starting to write your data and config files. {: entities} - ### Step 0.1: Determine whether you need new entities or entity types Data Commons is optimized to support aggregations of data at geographical levels, such as city, state, country, and so on. If your data is aggregated by place, these are supported as entities out of the box. If, however, you want to aggregate data for entities that are _not_ places, then you may need to define new entities, and possibly even entity types. -In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. +In addition, even if you aggregate by geographical area, you may want to measure things (known as a "population type" in the graph) that are not already in the graph. In that case, you might want to to define a new entity type, so that you can join with other data sets that measure the same thing. For example, let's say you have a metric that counts the number of beds in hospitals. The existence of the `Bed` entity type allows you to join your data with other sources with a similar metric. #### Entities and entity types Schema.org and the base Data Commons knowledge graph define entity types for just about everything in the world. An _entity type_ is a high-level concept, and is derived directly from a [`Class`](https://datacommons.org/browser/Class){: target="_blank"} type. Non-place entities are of two types: -* The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. -* The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. +- The thing you are measuring, known as the `populationType` in Data Commons. Often this is a `Person`, which is a commonly used population in Data Commons. But it could be something else entirely, like the beds in a hospital, the price of a commodity, Olympic medals won by a country, or the surface area of an ocean. +- The level at which you want to aggregate the data. Most commonly in Data Commons this is a place type such as `City`, `Country`, `AdministrativeArea1`, etc. Examples of other entity types are `Hospital`, `PublicSchool`, `Company`, `BusStation`, `Campground`, `Library` etc. It is rare that you would need to create a new entity type, unless you are working in a highly specialized domain. An _entity_ is an instance of an entity type. For example, for `PublicSchool`, base Data Commons has many U.S. schools in its knowledge graph, such as [`nces/010162001665`](https://datacommons.org/browser/nces/010162001665){: target="_blank"} (Adams Elementary School) or [`nces/010039000201`](https://datacommons.org/browser/nces/010039000201){: target="_blank"} (Wylam Elementary School). Base Data Commons contains thousands of places and other entities, but it's possible that it does not have specific entities that you need. For example, it has about 100 instances of `Company`, but you may want data for other companies besides those. As another example, let's say your organization wants to collect (possibly private) data about different divisions or departments of your org; in this case you would need to define entities for them. @@ -74,40 +70,35 @@ An _entity_ is an instance of an entity type. For example, for `PublicSchool`, b > **Note:** You should always reuse existing entity types and entities from base Data Commons rather than re-defining them. This way, you get all the properties already defined for those entities and all their linked nodes, and can more easily join with base data if needed. {: #search} - #### Search for an existing entity / entity type -Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. +Unfortunately, it is currently not possible to get a full list of entity types or entities in the Data Commons UI. To do a complete search for an entity type or entity, you need to use the REST or Python APIs. To search using the REST APIs: -1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Use the Node API through your browser to get a complete list of entity types: see [Get a list of all existing entity types](/api/rest/v2/node.html#list-entity-types) in the REST API V2 reference. Be sure to set the `nextToken` parameter until you find the relevant entity type or no `nextToken` is returned in the response. If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. -1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: +1. Use the Node API through your browser to look up all incoming arcs by the `typeof` property: 
https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=ENTITY_TYPE&property=<-typeOf
_ENTITY_TYPE_ is the DCID you've obtained in the previous step, such as `Hospital` or `PublicSchool`. For example: - ``` https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=PublicSchool&property=<-typeOf ``` - 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. To search using the Python APIs: 1. Start your Python interactive environment and [create a client for the base Data Commons](/api/python/v2/index.html). -1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). +1. Call the `Node` method `fetch_all_classes`: see [Get node properties](https://docs.datacommons.org/api/python/v2/node.html#fetch_all_classes) for details. (Tip: Use the `to_dict()` method on the response to get readable output.) If you don't find an entity type that matches your needs (very rare), you will need to [create one](custom_entities.md). 1. If you find a relevant entity type, note the DCID of the entity type of interest. The DCID of entity types is usually a meaningful name, capitalized, such as `Hospital` or `PowerPlant` or `PublicSchool`. 1. Use the `fetch_property_values` method to find all the instances of the type: 
client.node.fetch_property_values(node_dcids="ENTITY_TYPE", properties="typeOf", out=False)
_ENTITY_TYPE_ is the DCID you've obtained in the previous step. For example: - ``` client.node.fetch_property_values(node_dcids="PublicSchool", properties="typeOf", out=False) ``` - 1. If your entity is listed, note its DCID. If you are unable to find a relevant entity, you will need to create one. See [Work with custom entities](custom_entities.md) for complete information. ### Step 0.2: Identify your statistical variables @@ -134,12 +125,12 @@ If you do need to define new variables, they must follow a certain model. The va | San Jose | 2023 | private | secondary | 100 | The measure here is a simple count; the set of things is "schools"; and the constraints are the type and levels of the schools, namely "public", "private", "elementary", "middle" and "secondary". All of these things must be encoded as separate variables. Therefore, although the _properties_ of school type and school level may already be defined in the Data Commons knowledge graph (or you may need to define them), they _cannot_ be present as columns in the CSV files that you store in Data Commons. Instead, you must create separate "count" variables to represent each case. In our example, you would actually need 6 different variables: -* `Count_School_Public_Elementary` -* `Count_School_Public_Middle` -* `Count_School_Public_Secondary` -* `Count_School_Private_Elementary` -* `Count_School_Private_Middle` -* `Count_School_Private_Secondary` +- `Count_School_Public_Elementary` +- `Count_School_Public_Middle` +- `Count_School_Public_Secondary` +- `Count_School_Private_Elementary` +- `Count_School_Private_Middle` +- `Count_School_Private_Secondary` If you wanted totals or subtotals of combinations, you would need to create additional variables for these as well. @@ -168,25 +159,24 @@ Data Commons uses a schema that is called "variable-per-row". This means that ev The names and order of the columns aren't important, as you can map them to the expected columns in the JSON file. However, the city and variable names must be existing DCIDs. If such DCIDs don't already exist in the base Data Commons, you must provide definitions of them in MCF files. -> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. +> **Tip:** If your raw data does not conform to this structure (which is typically the case if you have relational data), you can usually easily convert the data by creating a pivot table (and renaming some columns) in a tool like Google Sheets or Microsoft Excel. ## Prepare your data In this section, we will walk you through a concrete example of how to go about setting up your MCF, CSV, and JSON files. {: #mcf} - ### Step 1: Define statistical variables in MCF If you are only reusing existing variables, you can skip this step entirely. -Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. +Nodes in the Data Commons knowledge graph are defined in Metadata Content Format (MCF) files. If you need to define new statistical variables, you must define them as new _nodes_ using MCF. When you define any variable in MCF, you explicitly assign it a DCID. > **Note:** You cannot "override" a variable definition by changing the value of existing fields. If you need to override the values of existing fields, you should create a new variable, with a new DCID. You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container. -Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. +Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. ``` Node: dcid:who/Percent_AdultSmokers @@ -215,7 +205,6 @@ measuredProperty: dcid:cigaretteSmoker statType: dcs:Percent measurementDenominator: dcs:Count_Person_Male ``` - The order of nodes and fields within nodes does not matter. The following fields are always required: @@ -226,11 +215,11 @@ The following fields are always required: * `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `schema:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. * `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: - * Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: + - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: ![domain incudes](/assets/images/custom_dc/customdc_screenshot9.png){: width="800"} - * Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. + - Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:` or `schema:`. Use the following guidelines to determine which to use: * If the node exists in [schema.org](https://schema.org/docs/schemas.html){: target="_blank"} (you can look them up in the **Term Finder**), use `schema`. @@ -247,24 +236,23 @@ The following fields are optional: Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. {: #naming} - #### Variable DCID naming conventions -* Variable DCIDs should be in PascalCase with underscores between properties. -* For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: +- Variable DCIDs should be in PascalCase with underscores between properties. +- For a basic variable without `measurementQualifier` or `measurementDenominator` properties, it should look like this: _`statType_measuredProperty_populationType_constraintValue1_constraintValue2`_ Example: `GrowthRate_Amount_EconomicActivity_GrossDomesticProduction` -* If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` -* For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: - * `Annual_Average_RetailPrice_Electricity` - * `Annual_Average_Wage` -* For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: - * `Count_Death_Female_AsAFractionOf_Count_Person_Female` - * `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` -* Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. +- If the `statType` is the default, `measuredValue`, omit it. For example: `Count_Person_Male_AsianAlone` +- For a variable with a `measurementQualifier` property, add the value to the prefix. Examples: + - `Annual_Average_RetailPrice_Electricity` + - `Annual_Average_Wage` +- For a variable with a `measurementDenominator` property, add the suffix `AsAFractionOf_`_`measurementDenominator`_. Examples: + - `Count_Death_Female_AsAFractionOf_Count_Person_Female` + - `Difference_Between_Median_Male_And_Female_Wages_AsAFractionOf_Median_Male_Wages` +- Multiple constraint values should be ordered according to the alphabetical precedence of the property name. For example, the property `gender` precedes `race` alphabetically, so constraint value `Male` would come before constraint value `AsianAlone`. For example: `Count_Person_Male_AsianAlone`. ### Step 2 (optional): Define a statistical variable group {#statvar-group} @@ -291,13 +279,12 @@ name: "WHO" specializationOf: dcid:dc/g/Root ``` - You can define as many statistical variable group nodes as you like. Each must include the following fields: -* `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. -* `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. -* `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. -* `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: +- `Node`: This is the DCID of the group you are defining. It must be prefixed by `g/` and may include an additional prefix before the `g`. +- `typeOf`: In the case of statistical variable group, this is always `dcid:StatVarGroup`. +- `name`: This is the name of the heading that will appear in the Statistical Variable Explorer. +- `specializationOf`: For a top-level group, this must be `dcid:dc/g/Root`, which is the root group in the statistical variable hierarchy in the Knowledge Graph.To create a sub-group, specify the DCID of another node you have already defined. For example, if you wanted to create a sub-group of `WHO` called `Smoking`, you would create a "Smoking" node with `specializationOf: dcid:who/g/WHO`. Here's an example: ``` Node: dcid:who/g/WHO @@ -341,7 +328,6 @@ memberOf: dcid:MyVariables ``` {: #exp_csv} - ### Step 3: Prepare the CSV observation files CSV files contain the following columns using the following headings: @@ -351,19 +337,19 @@ CSV files contain the following columns using the following headings: The columns can be in any order, and you can specify custom names for the headings and use the `columnMappings` field in the JSON file to map them accordingly (see below for details). These columns are required: -* `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. -* `variable`: The DCID of an existing variable or the node you have defined in the MCF -* `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. -* `value`: See [Observation values](#obs) for valid values of this column. +- `entity`: The DCID of an existing entity in the Data Commons knowledge graph, typically a place. +- `variable`: The DCID of an existing variable or the node you have defined in the MCF +- `date`: The date of the observation. This should be in the format _YYYY_, _YYYY_-_MM_, or _YYYY_-_MM_-_DD_. +- `value`: See [Observation values](#obs) for valid values of this column. > **Note:** The type of the entities in a single file should be unique; do not mix multiple entity types in the same CSV file. For example, if you have observations for cities and counties, put all the city data in one CSV file and all the county data in another one. These columns are optional, and allow you to specify additional per-observation properties: -* [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. -* [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. -* [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. -* [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. +- [`unit`](/glossary.html#unit): The unit of measurement used in the observations. This is a string representing a currency, area, weight, volume, etc. For example, `SquareFoot`, `USD`, `Barrel`, etc. +- [`observationPeriod`](/glossary.html#observation-period): The period of time in which the observations were recorded. This must be in ISO duration format, namely `P[0-9][Y|M|D|h|m|s]`. For example, `P1Y` is 1 year, `P3M` is 3 months, `P3h` is 3 hours. +- [`measurementMethod`](/glossary.html#measurement-method): The method used to gather the observations. This can be a random string or an existing DCID of [`MeasurementMethodEnum`](https://datacommons.org/browser/MeasurementMethodEnum){: target="_blank"} type; for example, `EDA_Estimate` or `WorldBankEstimate`. +- [`scalingFactor`](/glossary.html#scaling-factor): An integer representing the denominator used in measurements involving ratios or percentages. For example, for percentages, the denominator would be `100`. Here is an example of some real-world data from the WHO on the prevalance of smoking in adult populations, broken down by sex, in the correct CSV format: @@ -387,19 +373,18 @@ In this case, the columns need to be mapped to the expected columns listed above #### Observation values {#obs} Here are the rules for observation values: -* Variable values must be numeric. Do not include any special characters such as `*` or `#`. -* Zeros are accepted and recorded. -* For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. -* Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. +- Variable values must be numeric. Do not include any special characters such as `*` or `#`. +- Zeros are accepted and recorded. +- For null or not-a-number values, we recommend that you use blanks. (The strings `NaN`, `NA`, and `N/A` are also accepted.) These values will be ignored and not displayed in any charts or tables. +- Do not use negative numbers or inordinately large numbers to represent NaNs or nulls. {: #json} - ### Step 4: Write the JSON config file You must define a `config.json` in the top-level directory where your CSV files are located. You need to provide these specifications: -* The input files location and entity type -* The sources and provenances of the data -* Column mappings, if you are using custom names for the column headings +- The input files location and entity type +- The sources and provenances of the data +- Column mappings, if you are using custom names for the column headings Here is an example of how the config file would look for the CSV file we defined above. More details are below. @@ -430,19 +415,18 @@ Here is an example of how the config file would look for the CSV file we defined ``` The following fields are required: -* `input_files`: - * `format` must be `variablePerRow` - * `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. +- `input_files`: + - `format` must be `variablePerRow` + - `columnMappings` are required if you have used custom column heading names. The format is DEFAULT_NAME : CUSTOM_NAME. The following is optional: -* `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. +- `groupStatVarsByProperty` allows you to group your variables together according to population type. They will be displayed together in the Statistical Variable Explorer. Note that you don't specify your MCF files as input files; the Data Commons importer will identify them automatically. The other fields are explained in the [Data config file specification reference](config.md). {: #loadlocal} - ## Load local custom data The following procedures show you how to load and serve your custom data locally. @@ -450,12 +434,11 @@ The following procedures show you how to load and serve your custom data locally To load data in Google Cloud, see instead [Load data in Google Cloud](/custom_dc/deploy_cloud.html) for procedures. {: #env} - ### Configure environment variables Edit the `env.list` file you created [previously](/custom_dc/quickstart.html#env-vars) as follows: -* Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. -* Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. +- Set the `INPUT_DIR` variable to the full path to the directory where your input files are stored. +- Set the `OUTPUT_DIR` variable to the full path to the directory where you would like the output files to be stored. This can be the same or different from the input directory. When you rerun the Docker data management container, it will create a `datacommons` subdirectory under this directory. ### Start the Docker containers with local custom data {#docker-data} @@ -486,7 +469,7 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + @@ -494,7 +477,6 @@ Once you have configured everything, just run the `run_cdc_dev_docker.sh` script > **Note:** Any time you make changes to the CSV or JSON files and want to reload the data, you need to restart both containers. {:.no_toc} - #### (Optional) Start the data management container in schema update mode {#schema-update-mode} If you have tried to start a container, and have received a `SQL check failed` error, this indicates that a database schema update is needed. You need to restart the data management container, and you can specify an additional, optional, flag. This mode updates the database schema without re-importing data or re-building natural language embeddings. This is the quickest way to resolve a SQL check failed error during services container startup. @@ -525,13 +507,12 @@ If you have tried to start a container, and have received a `SQL check failed` e -v INPUT_DIRECTORY:INPUT_DIRECTORY \ -v OUTPUT_DIRECTORY:OUTPUT_DIRECTORY \ gcr.io/datcom-ci/datacommons-services:stable - + {: #verify} - ### Verify your data If the servers have started up without errors, check to ensure that your data is showing up as expected. @@ -539,7 +520,6 @@ If the servers have started up without errors, check to ensure that your data is 1. Verify statistical variables: go to the [Statistical Variable Explorer](https://localhost:8080/tools/statvar){: target="_blank"} to verify that your statistical variables are showing up correctly. You should see something like this: ![](/assets/images/custom_dc/customdc_screenshot11.png){: width="400"} - 1. Click on a variable name to get more information on the right panel. 1. Verify that your observations are loaded: Click on an **Example Place** link to open the detailed page for that place. Scroll to the bottom, where you should see a timeline graph of observations for the selected place. 1. Verify natural-language querying: go to the [Search page](https://localhost:8080/tools/explore){: target="_blank"} and enter a query related to your data. You should get relevant graphs using your data. @@ -560,7 +540,6 @@ At the prompt, enter SQL queries. For example, for the sample OECD data, this qu ```shell sqlite> select * from observations limit 10; ``` - returns output like this: ```shell @@ -575,4 +554,4 @@ country/BEL|average_annual_wage|2005|55662.21541|c/p/1 To exit the sqlite shell, press `Ctrl-D`. - + \ No newline at end of file From 3b374bb14b5ba390bf1b99384ed92930dcee6771 Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 9 Jul 2026 09:56:44 -0700 Subject: [PATCH 15/16] corrected type of measurementQualifier --- custom_dc/custom_data.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index 0230cb6b1..cb5f251d4 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -176,7 +176,7 @@ Nodes in the Data Commons knowledge graph are defined in Metadata Content Format You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container. -Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assume that`cigaretteSmoker` already exists as a property. +Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assum that there is already a property called `cigaretteSmoker`. ``` Node: dcid:who/Percent_AdultSmokers @@ -213,8 +213,7 @@ The following fields are always required: * `typeOf`: In the case of statistical variable, this is always `schema:StatisticalVariable`. * `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. * `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `schema:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. -* `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. - You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: +* `measuredProperty`: This is a property of the thing being measured. It must be property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. To view the list of properties for a given `populationType`, using either of the following methods: - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: ![domain incudes](/assets/images/custom_dc/customdc_screenshot9.png){: width="800"} @@ -230,7 +229,7 @@ All fields that do not reference another node must be in quotation marks. The following fields are optional: * `description`: A more detailed textual description of the variable. * `statType`: By default, if not specified, this is `dcs:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. -* `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. +* `measurementQualifier`: This is used to qualify the measurement represented in all observations using the variable. It must be a member of an enumeration, e.g. `Weekly`, `Monthly`, `Annual`, which are members of the [StatAccumulationPeriodEnum](https://datacommons.org/browser/StatAccumulationPeriodEnum){: target="_blank"} type. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. * `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. From 699f2ee4bda6edd7b25f9749361fbf6ba8744e9a Mon Sep 17 00:00:00 2001 From: Kara Moscoe Date: Thu, 9 Jul 2026 10:13:51 -0700 Subject: [PATCH 16/16] replace more dcs prefixes --- custom_dc/custom_data.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index cb5f251d4..5ef7d4e4c 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -202,7 +202,7 @@ name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Person gender: dcs:Male measuredProperty: dcid:cigaretteSmoker -statType: dcs:Percent +statType: schema:Percent measurementDenominator: dcs:Count_Person_Male ``` The order of nodes and fields within nodes does not matter. @@ -228,7 +228,7 @@ All fields that do not reference another node must be in quotation marks. The following fields are optional: * `description`: A more detailed textual description of the variable. -* `statType`: By default, if not specified, this is `dcs:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +* `statType`: By default, if not specified, this is `schema:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. * `measurementQualifier`: This is used to qualify the measurement represented in all observations using the variable. It must be a member of an enumeration, e.g. `Weekly`, `Monthly`, `Annual`, which are members of the [StatAccumulationPeriodEnum](https://datacommons.org/browser/StatAccumulationPeriodEnum){: target="_blank"} type. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. * `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`.