diff --git a/custom_dc/custom_data.md b/custom_dc/custom_data.md index ec96439c4..5ef7d4e4c 100644 --- a/custom_dc/custom_data.md +++ b/custom_dc/custom_data.md @@ -176,54 +176,63 @@ Nodes in the Data Commons knowledge graph are defined in Metadata Content Format You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container. -Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. +Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assum that there is already a property called `cigaretteSmoker`. ``` -Node: dcid:who/Adult_curr_cig_smokers -typeOf: dcid:StatisticalVariable +Node: dcid:who/Percent_AdultSmokers +typeOf: schema:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%)" -populationType: dcid:Person -measuredProperty: dcid:percent +populationType: schema:Person +measuredProperty: dcid:cigaretteSmoker +statType: dcs:Percent +measurementDenominator: dcs:Count_Person -Node: dcid:who/Adult_curr_cig_smokers_female -typeOf: dcid:StatisticalVariable +Node: dcid:who/Percent_AdultSmokers_Female +typeOf: schema:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Female]" -populationType: dcid:Person -measuredProperty: dcid:percent -gender: dcid:Female - -Node: dcid:who/Adult_curr_cig_smokers_male -typeOf: dcid:StatisticalVariable +populationType: schema:Person +gender: dcs:Female +measuredProperty: dcid:cigaretteSmoker +statType: dcs:Percent +measurementDenominator: dcs:Count_Person_Female + +Node: dcid:who/Percent_AdultSmokers_Male +typeOf: schema:StatisticalVariable name: "Prevalence of current cigarette smoking among adults (%) [Male]" populationType: dcid:Person -measuredProperty: dcid:percent -gender: dcid:Male +gender: dcs:Male +measuredProperty: dcid:cigaretteSmoker +statType: schema:Percent +measurementDenominator: dcs:Count_Person_Male ``` The order of nodes and fields within nodes does not matter. The following fields are always required: -- `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. +* `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. It must be preceded by the prefix `dcid:`. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you. > Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want. -- `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`. -- `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. -- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. -- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the `percent` of persons being measured. - You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods: +* `typeOf`: In the case of statistical variable, this is always `schema:StatisticalVariable`. +* `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI. +* `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `schema:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it. +* `measuredProperty`: This is a property of the thing being measured. It must be property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. To view the list of properties for a given `populationType`, using either of the following methods: - Go to https://datacommons.org/browser/POPULATION_TYPE, e.g. {: target="_blank"} and scroll to the **domainIncludes** section of the page. For example: ![domain incudes](/assets/images/custom_dc/customdc_screenshot9.png){: width="800"} - Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=POPULATION_TYPE&property=%3C-domainIncludes, e.g. {: target="_blank"}. -Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:`, which are interchangeable. All fields that do not reference another node must be in quotation marks. +Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:` or `schema:`. Use the following guidelines to determine which to use: +* If the node exists in [schema.org](https://schema.org/docs/schemas.html){: target="_blank"} (you can look them up in the **Term Finder**), use `schema`. +* If the node exists in the core Data Commons schema, [dcschema.mcf](https://github.com/datacommonsorg/schema/blob/main/core/dcschema.mcf){: target="blank"}, use `dcs`. +* Otherwise, use `dcid` for all others. +All fields that do not reference another node must be in quotation marks. The following fields are optional: -- `description`: A more detailed textual description of the variable. -- `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, `varianceValue`, `marginOfError`, `stdErr` and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. -- `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. -- `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. +* `description`: A more detailed textual description of the variable. +* `statType`: By default, if not specified, this is `schema:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to {: target="_blank"}, and scrolling to the **domainIncludes** section of the page. +* `measurementQualifier`: This is used to qualify the measurement represented in all observations using the variable. It must be a member of an enumeration, e.g. `Weekly`, `Monthly`, `Annual`, which are members of the [StatAccumulationPeriodEnum](https://datacommons.org/browser/StatAccumulationPeriodEnum){: target="_blank"} type. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate. +* `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`. -Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our example, there is one constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. +Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`. {: #naming} #### Variable DCID naming conventions @@ -251,15 +260,15 @@ By default, existing variables are shown in the Statistical Variable Explorer in Here is an example that defines a single group node with the heading "WHO" and assigns all 3 statistical variables to the same group. ``` -Node: dcid:who/Adult_curr_cig_smokers +Node: dcid:who/Percent_AdultSmokers ... memberOf: dcid:who/g/WHO -Node: dcid:who/Adult_curr_cig_smokers_female +Node: dcid:who/Percent_AdultSmokers_Female ... memberOf:dcid:who/g/WHO -Node: dcid:who/Adult_curr_cig_smokers_male +Node: dcid:who/Percent_AdultSmokers_Male ... memberOf: dcid:who/g/WHO @@ -291,15 +300,15 @@ You can define as many statistical variable group nodes as you like. Each must i You can also assign a variable to as many group nodes as you like: simply specify a comma-separated list of group DCIDs in the `memberOf`. For example, to assign the 3 variables to both groups: ``` -Node: dcid:who/Adult_curr_cig_smokers +Node: dcid:who/Percent_AdultSmokers ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking -Node: dcid:who/Adult_curr_cig_smokers_female +Node: dcid:who/Percent_AdultSmokers_Female ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking -Node: dcid:who/Adult_curr_cig_smokers_male +Node: dcid:who/Percent_AdultSmokers_Male ... memberOf: dcid:who/g/WHO, dcid:who/g/Smoking ``` @@ -313,7 +322,7 @@ name: "My variables" specializationOf: dcid:dc/g/Root Node: dcid:GenderIncomeInequality_Person_15OrMoreYears_WithIncome -typeOf: dcs:StatisticalVariable +typeOf: schema:StatisticalVariable memberOf: dcid:MyVariables ``` @@ -345,16 +354,17 @@ Here is an example of some real-world data from the WHO on the prevalance of smo ```csv SERIES,GEOGRAPHY,TIME_PERIOD,OBS_VALUE -dcs:who/Adult_curr_cig_smokers_female,dcid:country/AFG,2019,1.2 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/AFG,2019,13.4 -dcs:who/Adult_curr_cig_smokers,dcid:country/AFG,2019,7.5 -dcs:who/Adult_curr_cig_smokers_female,dcid:country/AGO,2016,1.8 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/AGO,2016,14.3 -dcs:who/Adult_curr_cig_smokers_female,dcid:country/ALB,2018,4.5 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/ALB,2018,35.7 -dcs:who/Adult_curr_cig_smokers_male,dcid:country/ARE,2018,11.1 -dcs:who/Adult_curr_cig_smoking_female,dcid:country/ARE,2018,1.6 -dcs:who/Adult_curr_cig_smokers,dcid:country/ARE,2018,6.3 +dcid:who/Percent_AdultSmokers_Female,dcid:country/AFG,2019,1.2 +dcid:who/Percent_AdultSmokers_Male,dcid:country/AFG,2019,13.4 +dcid:who/Percent_AdultSmokers,dcid:country/AFG,2019,7.5 +dcid:who/Percent_AdultSmokers_Female,dcid:country/AGO,2016,1.8 +dcid:who/Percent_AdultSmokers_Male,dcid:country/AGO,2016,14.3 +dcid:who/Percent_AdultSmokers_Female,dcid:country/ALB,2018,4.5 +dcid:who/Percent_AdultSmokers_Male,dcid:country/ALB,2018,35.7 +dcid:who/Percent_AdultSmokers_Male,dcid:country/ARE,2018,11.1 +dcid:who/Percent_AdultSmokers_Female,dcid:country/ARE,2018,1.6 +dcid:who/Percent_AdultSmokers,dcid:country/ARE,2018,6.3 +... ``` In this case, the columns need to be mapped to the expected columns listed above; see below for details. diff --git a/custom_dc/custom_entities.md b/custom_dc/custom_entities.md index 6f332238f..88d9ffc94 100644 --- a/custom_dc/custom_entities.md +++ b/custom_dc/custom_entities.md @@ -131,12 +131,12 @@ The other fields are explained in the [Data config file specification reference] If you are providing observations for the non-place entities, the observations must be in a separate file. You'll need a different CSV file for each entity type for which you are providing observations. For example, let's say you've already defined in MCF the following variables that measure weekly hospital capacity: -* `total_count_staffed_beds` -* `count_staffed_adult_beds` -* `count_staffed_inpatient_icu_beds` -* `count_staffed_adult_inpatient_icu_beds` -* `count_staffed_inpatient_icu_beds_occupied` -* `count_staffed_adult_icu_beds_occupied` +* `Count_StaffedBeds` +* `Count_StaffedBeds_Adult` +* `Count_StaffedBeds_Inpatient_ICU` +* `Count_StaffedBeds_Adult_Inpatient_ICU` +* `Count_StaffedBedsOccupied_Inpatient_ICU` +* `Count_StaffedBedsOccupied_Adult_ICU_beds` Aside: Note that the thing being measured here is "beds". There is an existing [Bed](https://datacommons.org/browser/Bed) class in Data Commons. So when defining such variables, you would specify `schema:bed` as the `populationType`. @@ -144,23 +144,23 @@ Just like for place entities, you provide observations for these variables in a ```csv entity,date,variable,value -20001,2023-01-27,count_staffed_adult_beds,1048 -20001,2023-01-27,count_staffed_adult_icu_beds_occupied,146 -20001,2023-01-27,count_staffed_adult_inpatient_icu_beds,146 -20001,2023-01-27,count_staffed_inpatient_icu_beds,264 -20001,2023-01-27,count_staffed_inpatient_icu_beds_occupied,264 -20001,2023-01-27,total_count_staffed_beds,1262 -20017,2023-01-27,count_staffed_adult_beds,0 -20017,2023-01-27,count_staffed_adult_icu_beds_occupied,0 -20017,2023-01-27,count_staffed_adult_inpatient_icu_beds, -20017,2023-01-27,count_staffed_inpatient_icu_beds, -20017,2023-01-27,count_staffed_inpatient_icu_beds_occupied,0 -21301,2023-01-27,count_staffed_adult_beds,780 -21301,2023-01-27,count_staffed_adult_icu_beds_occupied,62 -21301,2023-01-27,count_staffed_adult_inpatient_icu_beds,62 -21301,2023-01-27,count_staffed_inpatient_icu_beds,101 -21301,2023-01-27,count_staffed_inpatient_icu_beds_occupied,66 -21301,2023-01-27,total_count_staffed_beds,836 +20001,2023-01-27,Count_StaffedBeds,1048 +20001,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,146 +20001,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,146 +20001,2023-01-27,Count_StaffedBeds_Inpatient_ICU,264 +20001,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,264 +20001,2023-01-27,Count_StaffedBeds,1262 +20017,2023-01-27,Count_StaffedBeds_Adult,0 +20017,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,0 +20017,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU, +20017,2023-01-27,Count_StaffedBeds_Inpatient_ICU, +20017,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,0 +21301,2023-01-27,Count_StaffedBeds_Adult,780 +21301,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,62 +21301,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,62 +21301,2023-01-27,Count_StaffedBeds_Inpatient_ICU,101 +21301,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,66 +21301,2023-01-27,Count_StaffedBeds,836 ... ``` We could also have added an `observationPeriod` column, which would be set to `P7D` for all rows. @@ -241,17 +241,17 @@ Then, if desired, you could provide aggregated observations for each hospital ty ```csv entity,date,variable,value -ShortTermHospital,2023-01-27,count_staffed_adult_beds,... -ShortTermHospital,2023-01-27,count_staffed_adult_icu_beds_occupied,... -ShortTermHospital,2023-01-27,count_staffed_adult_inpatient_icu_beds,... -ShortTermHospital,2023-01-27,count_staffed_inpatient_icu_beds,... -ShortTermHospital,2023-01-27,count_staffed_inpatient_icu_beds_occupied,... -ShortTermHospital,2023-01-27,total_count_staffed_beds,... -LongTermHospital,2023-01-27,count_staffed_adult_beds,... -LongTermHospital,2023-01-27,count_staffed_adult_icu_beds_occupied,... -LongTermHospital,2023-01-27,count_staffed_adult_inpatient_icu_beds... -LongTermHospital,2023-01-27,count_staffed_inpatient_icu_beds... -LongTermHospital,2023-01-27,count_staffed_inpatient_icu_beds_occupied,... +ShortTermHospital,2023-01-27,Count_StaffedBeds_Adult... +ShortTermHospital,2023-01-27,Count_StaffedBedsOccuped_Adult_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBeds_Inpatient_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,... +ShortTermHospital,2023-01-27,Count_StaffedBeds,... +LongTermHospital,2023-01-27,Count_StaffedBeds_Adult,... +LongTermHospital,2023-01-27,Count_StaffedBedsOccuped_Adult_ICU,... +LongTermHospital,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU... +LongTermHospital,2023-01-27,Count_StaffedBeds_Inpatient_ICU... +LongTermHospital,2023-01-27,ount_StaffedBedsOccupied_Inpatient_ICU,... ... ``` diff --git a/llms-full.txt b/llms-full.txt index a00bd4042..17869bf93 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -35,7 +35,7 @@ {'title': 'Data coverage', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/datasets/index.md', 'desc': 'Overview of datasets in Data Commons and per-country coverage.'} {'title': 'Place types', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/place_types.md', 'desc': 'Reference for place types in Data Commons.'} {'title': 'Get support', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/support.md', 'desc': 'Support channels and feedback paths.'} -{'title': 'Contribute to Data Commons', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/contributing.md', 'desc': {}} +{'title': 'Contribute to Data Commons', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/contributing.md', 'desc': 'Contribute data, code, or documentation to Data Commons.'} {'title': 'Web Components', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/api/web_components/index.md', 'desc': 'Embed Data Commons charts in your own website.'} {'title': 'Map chart', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/api/web_components/map.md', 'desc': {}} {'title': 'Bar chart', 'url': 'https://raw.githubusercontent.com/datacommonsorg/docsite/master/api/web_components/bar.md', 'desc': {}} @@ -5275,6 +5275,8 @@ nav_order: 1 parent: API - Query data programmatically has_children: true published: true +redirect_from: + /api/python/index --- {:.no_toc} @@ -14988,7 +14990,7 @@ If you are using a Custom Data Commons instance, make sure to indicate that the ## Email support forum -For technical questions that you can't find answers to in this documentation, you can email support@datacommons.org.404: Not Found--- +For technical questions that you can't find answers to in this documentation, you can email support@datacommons.org.404: Not Found--- layout: default title: Embed data and visualizations in your own website nav_order: 60