Skip to content

Redo statvar examples #717

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kmoscoe wants to merge 17 commits into
base: master
Choose a base branch
from
Open

Redo statvar examples #717

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
6ce4283
Redo statvar examples
kmoscoe Jul 2, 2026
50048e1
Fix name
kmoscoe Jul 2, 2026
44c933b
Fix population type
kmoscoe Jul 2, 2026
2c53d1c
Fix typo
kmoscoe Jul 2, 2026
906d794
Remove nav entry for Python V1 API and redirect (#718)
kmoscoe Jul 6, 2026
1987c1b
Rename variables to conform to conventions
kmoscoe Jul 8, 2026
af45983
Merge branch 'datacommonsorg:master' into data
kmoscoe Jul 8, 2026
f172428
Revert "Fix typo"
kmoscoe Jul 8, 2026
e69ced2
fix capitalization
kmoscoe Jul 8, 2026
d8834b8
Fix more names
kmoscoe Jul 8, 2026
c7a1fd8
Fix prefixes
kmoscoe Jul 8, 2026
900afe9
Changes from Caro
kmoscoe Jul 9, 2026
fe15ba7
changes from Caro
kmoscoe Jul 9, 2026
2038c76
Regenerate llms-full
kmoscoe Jul 9, 2026
2ce21fc
Revert "Fix typo"
kmoscoe Jul 9, 2026
3b374bb
corrected type of measurementQualifier
kmoscoe Jul 9, 2026
699f2ee
replace more dcs prefixes
kmoscoe Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 54 additions & 44 deletions custom_dc/custom_data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,54 +176,63 @@ Nodes in the Data Commons knowledge graph are defined in Metadata Content Format

You can define your statistical variables in a single MCF file, or split them into as many separate MCF files as you like. MCF files must have a `.mcf` suffix. The importer will automatically find them when you start the Docker data container.

Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes.
Here's an example of defining some statistical variables representing data in a UN WHO dataset. It defines 3 new statistical variable nodes. Assum that there is already a property called `cigaretteSmoker`.

```
Node: dcid:who/Adult_curr_cig_smokers
typeOf: dcid:StatisticalVariable
Node: dcid:who/Percent_AdultSmokers
typeOf: schema:StatisticalVariable
name: "Prevalence of current cigarette smoking among adults (%)"
populationType: dcid:Person
measuredProperty: dcid:percent
populationType: schema:Person
measuredProperty: dcid:cigaretteSmoker
Comment thread
kmoscoe marked this conversation as resolved.
statType: dcs:Percent
measurementDenominator: dcs:Count_Person

Node: dcid:who/Adult_curr_cig_smokers_female
typeOf: dcid:StatisticalVariable
Node: dcid:who/Percent_AdultSmokers_Female
typeOf: schema:StatisticalVariable
name: "Prevalence of current cigarette smoking among adults (%) [Female]"
populationType: dcid:Person
measuredProperty: dcid:percent
gender: dcid:Female

Node: dcid:who/Adult_curr_cig_smokers_male
typeOf: dcid:StatisticalVariable
populationType: schema:Person
gender: dcs:Female
measuredProperty: dcid:cigaretteSmoker
statType: dcs:Percent
measurementDenominator: dcs:Count_Person_Female

Node: dcid:who/Percent_AdultSmokers_Male
typeOf: schema:StatisticalVariable
name: "Prevalence of current cigarette smoking among adults (%) [Male]"
populationType: dcid:Person
measuredProperty: dcid:percent
gender: dcid:Male
gender: dcs:Male
measuredProperty: dcid:cigaretteSmoker
statType: schema:Percent
measurementDenominator: dcs:Count_Person_Male
```
The order of nodes and fields within nodes does not matter.

The following fields are always required:
- `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you.
* `Node`: This is the DCID of the entity you are defining. DCIDs can be a maximum of 256 characters long. It must be preceded by the prefix `dcid:`. We recommend that you add an optional prefix, separated by a slash (/), for example, `who/`, to differentiate your custom variables from base DC variables. The prefix acts as a namespace, and should represent your organization, dataset, project, or whatever makes sense for you.
> Note: If you plan to contribute your data to base Data Commons, DCIDs should follow the [DCID naming conventions](#naming). Otherwise, you can name them however you want.
- `typeOf`: In the case of statistical variable, this is always `dcid:StatisticalVariable`.
- `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI.
- `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `dcid:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it.
- `measuredProperty`: This is a property of the thing being measured. It must be a `domainIncludes` property of the `populationType` you have specified. In this example, it is the `percent` of persons being measured.
You can see the set of `domainIncludes` properties for a given `populationType`, using either of the following methods:
* `typeOf`: In the case of statistical variable, this is always `schema:StatisticalVariable`.
* `name`: This is the descriptive name of the variable, that is displayed in the Statistical Variable Explorer and various other places in the UI.
* `populationType`: This is the type of the thing being measured, and its value must be an existing `Class` type. In this example it is `schema:Person`. To get a full list of existing entity types, see the section on [searching](#search) above. If the thing you are measuring does not exist in the knowledge graph, you will need to create a new [entity type](custom_entities.md#entity-type) for it.
* `measuredProperty`: This is a property of the thing being measured. It must be property of the `populationType` you have specified. In this example, it is the prevelance of smoking, represented as a property called `cigaretteSmoker` of persons, females, and males, being measured. To view the list of properties for a given `populationType`, using either of the following methods:
- Go to <code>https://datacommons.org/browser/<var>POPULATION_TYPE</var></code>, e.g. <https://datacommons.org/browser/Person>{: target="_blank"} and scroll to the **domainIncludes** section of the page. For example:

![domain incudes](/assets/images/custom_dc/customdc_screenshot9.png){: width="800"}

- Use the [Node API](/api/rest/v2/node.html#wildcard), filtering on `domainIncludes` incoming arcs: <code>https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=<var>POPULATION_TYPE</var>&property=%3C-domainIncludes</code>, e.g. <https://api.datacommons.org/v2/node?key=AIzaSyCTI4Xz-UW_G2Q2RfknhcfdAnTHq5X5XuI&nodes=Person&property=%3C-domainIncludes>{: target="_blank"}.

Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:`, which are interchangeable. All fields that do not reference another node must be in quotation marks.
Note that all fields that reference another node in the graph must be prefixed by `dcid:` or `dcs:` or `schema:`. Use the following guidelines to determine which to use:
* If the node exists in [schema.org](https://schema.org/docs/schemas.html){: target="_blank"} (you can look them up in the **Term Finder**), use `schema`.
* If the node exists in the core Data Commons schema, [dcschema.mcf](https://github.com/datacommonsorg/schema/blob/main/core/dcschema.mcf){: target="blank"}, use `dcs`.
* Otherwise, use `dcid` for all others.
All fields that do not reference another node must be in quotation marks.

The following fields are optional:
Comment thread
kmoscoe marked this conversation as resolved.
- `description`: A more detailed textual description of the variable.
- `statType`: By default, if not specified, this is `dcid:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, `varianceValue`, `marginOfError`, `stdErr` and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to <https://datacommons.org/browser/StatisticalVariable>{: target="_blank"}, and scrolling to the **domainIncludes** section of the page.
- `measurementQualifier`: This is similar to the [`observationPeriod`](#exp_csv) field for CSV files and applies to all observations of the variable. It can be any string representing additional properties of the variable, e.g. `Weekly`, `Monthly`, `Annual`. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate.
- `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`.
* `description`: A more detailed textual description of the variable.
* `statType`: By default, if not specified, this is `schema:measuredValue`, which is simply a raw value of an observation. If your variable is a calculated value, such as an average, a minimum or maximum, you can use `minValue`, `maxValue`, `meanValue`, `medianValue`, `sumvalue`, and so on. If you use a calculated value, your data set should only include the observations that correspond to those calculated values. You can see the full set of allowable values by going to <https://datacommons.org/browser/StatisticalVariable>{: target="_blank"}, and scrolling to the **domainIncludes** section of the page.
* `measurementQualifier`: This is used to qualify the measurement represented in all observations using the variable. It must be a member of an enumeration, e.g. `Weekly`, `Monthly`, `Annual`, which are members of the [StatAccumulationPeriodEnum](https://datacommons.org/browser/StatAccumulationPeriodEnum){: target="_blank"} type. For instance, if the `measuredProperty` is income, you can use `Annual` or `Monthly` to distinguish income over different periods. If the time interval affects the meaning of variable and and values change significantly by the time period, you should use this field keep them separate.
* `measurementDenominator`: For percentages or ratios, this refers to another statistical variable DCID. For example, for per-capita, the `measurementDenominator` is `Count_Person`.

Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our example, there is one constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`.
Additionally, you can specify any number of property-value pairs representing the constraints (known as `constraintProperties` in the schema) on the type identified by `populationType`. In our examples above, we use a constraint property, `gender`, which is a property of `Person`. The constraint property values are typically enumerations; such as `genderType`, which is a `rangeIncludes` property of `gender`.

{: #naming}
#### Variable DCID naming conventions
Expand Down Expand Up @@ -251,15 +260,15 @@ By default, existing variables are shown in the Statistical Variable Explorer in
Here is an example that defines a single group node with the heading "WHO" and assigns all 3 statistical variables to the same group.

```
Node: dcid:who/Adult_curr_cig_smokers
Node: dcid:who/Percent_AdultSmokers
...
memberOf: dcid:who/g/WHO

Node: dcid:who/Adult_curr_cig_smokers_female
Node: dcid:who/Percent_AdultSmokers_Female
...
memberOf:dcid:who/g/WHO

Node: dcid:who/Adult_curr_cig_smokers_male
Node: dcid:who/Percent_AdultSmokers_Male
...
memberOf: dcid:who/g/WHO

Expand Down Expand Up @@ -291,15 +300,15 @@ You can define as many statistical variable group nodes as you like. Each must i
You can also assign a variable to as many group nodes as you like: simply specify a comma-separated list of group DCIDs in the `memberOf`. For example, to assign the 3 variables to both groups:

```
Node: dcid:who/Adult_curr_cig_smokers
Node: dcid:who/Percent_AdultSmokers
...
memberOf: dcid:who/g/WHO, dcid:who/g/Smoking

Node: dcid:who/Adult_curr_cig_smokers_female
Node: dcid:who/Percent_AdultSmokers_Female
...
memberOf: dcid:who/g/WHO, dcid:who/g/Smoking

Node: dcid:who/Adult_curr_cig_smokers_male
Node: dcid:who/Percent_AdultSmokers_Male
...
memberOf: dcid:who/g/WHO, dcid:who/g/Smoking
```
Expand All @@ -313,7 +322,7 @@ name: "My variables"
specializationOf: dcid:dc/g/Root

Node: dcid:GenderIncomeInequality_Person_15OrMoreYears_WithIncome
typeOf: dcs:StatisticalVariable
typeOf: schema:StatisticalVariable
memberOf: dcid:MyVariables
```

Expand Down Expand Up @@ -345,16 +354,17 @@ Here is an example of some real-world data from the WHO on the prevalance of smo

```csv
SERIES,GEOGRAPHY,TIME_PERIOD,OBS_VALUE
dcs:who/Adult_curr_cig_smokers_female,dcid:country/AFG,2019,1.2
dcs:who/Adult_curr_cig_smokers_male,dcid:country/AFG,2019,13.4
dcs:who/Adult_curr_cig_smokers,dcid:country/AFG,2019,7.5
dcs:who/Adult_curr_cig_smokers_female,dcid:country/AGO,2016,1.8
dcs:who/Adult_curr_cig_smokers_male,dcid:country/AGO,2016,14.3
dcs:who/Adult_curr_cig_smokers_female,dcid:country/ALB,2018,4.5
dcs:who/Adult_curr_cig_smokers_male,dcid:country/ALB,2018,35.7
dcs:who/Adult_curr_cig_smokers_male,dcid:country/ARE,2018,11.1
dcs:who/Adult_curr_cig_smoking_female,dcid:country/ARE,2018,1.6
dcs:who/Adult_curr_cig_smokers,dcid:country/ARE,2018,6.3
dcid:who/Percent_AdultSmokers_Female,dcid:country/AFG,2019,1.2
dcid:who/Percent_AdultSmokers_Male,dcid:country/AFG,2019,13.4
dcid:who/Percent_AdultSmokers,dcid:country/AFG,2019,7.5
dcid:who/Percent_AdultSmokers_Female,dcid:country/AGO,2016,1.8
dcid:who/Percent_AdultSmokers_Male,dcid:country/AGO,2016,14.3
dcid:who/Percent_AdultSmokers_Female,dcid:country/ALB,2018,4.5
dcid:who/Percent_AdultSmokers_Male,dcid:country/ALB,2018,35.7
dcid:who/Percent_AdultSmokers_Male,dcid:country/ARE,2018,11.1
dcid:who/Percent_AdultSmokers_Female,dcid:country/ARE,2018,1.6
dcid:who/Percent_AdultSmokers,dcid:country/ARE,2018,6.3
...
```

In this case, the columns need to be mapped to the expected columns listed above; see below for details.
Expand Down
68 changes: 34 additions & 34 deletions custom_dc/custom_entities.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,36 +131,36 @@ The other fields are explained in the [Data config file specification reference]
If you are providing observations for the non-place entities, the observations must be in a separate file. You'll need a different CSV file for each entity type for which you are providing observations.

For example, let's say you've already defined in MCF the following variables that measure weekly hospital capacity:
* `total_count_staffed_beds`
* `count_staffed_adult_beds`
* `count_staffed_inpatient_icu_beds`
* `count_staffed_adult_inpatient_icu_beds`
* `count_staffed_inpatient_icu_beds_occupied`
* `count_staffed_adult_icu_beds_occupied`
* `Count_StaffedBeds`
* `Count_StaffedBeds_Adult`
* `Count_StaffedBeds_Inpatient_ICU`
* `Count_StaffedBeds_Adult_Inpatient_ICU`
* `Count_StaffedBedsOccupied_Inpatient_ICU`
* `Count_StaffedBedsOccupied_Adult_ICU_beds`

Aside: Note that the thing being measured here is "beds". There is an existing [Bed](https://datacommons.org/browser/Bed) class in Data Commons. So when defining such variables, you would specify `schema:bed` as the `populationType`.

Just like for place entities, you provide observations for these variables in a CSV file. The CSV observations file uses the same variable-per-row format and [column headings](custom_data.md#exp-csv) as places. The only difference from a place-based CSV is that the entity column contains the DCIDs of the entities you have defined in a separate CSV (or MCF) file, instead of places. In our example, the DCIDs are the CCNs of the hospitals.

```csv
entity,date,variable,value
20001,2023-01-27,count_staffed_adult_beds,1048
20001,2023-01-27,count_staffed_adult_icu_beds_occupied,146
20001,2023-01-27,count_staffed_adult_inpatient_icu_beds,146
20001,2023-01-27,count_staffed_inpatient_icu_beds,264
20001,2023-01-27,count_staffed_inpatient_icu_beds_occupied,264
20001,2023-01-27,total_count_staffed_beds,1262
20017,2023-01-27,count_staffed_adult_beds,0
20017,2023-01-27,count_staffed_adult_icu_beds_occupied,0
20017,2023-01-27,count_staffed_adult_inpatient_icu_beds,
20017,2023-01-27,count_staffed_inpatient_icu_beds,
20017,2023-01-27,count_staffed_inpatient_icu_beds_occupied,0
21301,2023-01-27,count_staffed_adult_beds,780
21301,2023-01-27,count_staffed_adult_icu_beds_occupied,62
21301,2023-01-27,count_staffed_adult_inpatient_icu_beds,62
21301,2023-01-27,count_staffed_inpatient_icu_beds,101
21301,2023-01-27,count_staffed_inpatient_icu_beds_occupied,66
21301,2023-01-27,total_count_staffed_beds,836
20001,2023-01-27,Count_StaffedBeds,1048
20001,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,146
20001,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,146
20001,2023-01-27,Count_StaffedBeds_Inpatient_ICU,264
20001,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,264
20001,2023-01-27,Count_StaffedBeds,1262
20017,2023-01-27,Count_StaffedBeds_Adult,0
20017,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,0
20017,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,
20017,2023-01-27,Count_StaffedBeds_Inpatient_ICU,
20017,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,0
21301,2023-01-27,Count_StaffedBeds_Adult,780
21301,2023-01-27,Count_StaffedBedsOccupied_Adult_ICU,62
21301,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,62
21301,2023-01-27,Count_StaffedBeds_Inpatient_ICU,101
21301,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,66
21301,2023-01-27,Count_StaffedBeds,836
...
```
We could also have added an `observationPeriod` column, which would be set to `P7D` for all rows.
Expand Down Expand Up @@ -241,17 +241,17 @@ Then, if desired, you could provide aggregated observations for each hospital ty

```csv
entity,date,variable,value
ShortTermHospital,2023-01-27,count_staffed_adult_beds,...
ShortTermHospital,2023-01-27,count_staffed_adult_icu_beds_occupied,...
ShortTermHospital,2023-01-27,count_staffed_adult_inpatient_icu_beds,...
ShortTermHospital,2023-01-27,count_staffed_inpatient_icu_beds,...
ShortTermHospital,2023-01-27,count_staffed_inpatient_icu_beds_occupied,...
ShortTermHospital,2023-01-27,total_count_staffed_beds,...
LongTermHospital,2023-01-27,count_staffed_adult_beds,...
LongTermHospital,2023-01-27,count_staffed_adult_icu_beds_occupied,...
LongTermHospital,2023-01-27,count_staffed_adult_inpatient_icu_beds...
LongTermHospital,2023-01-27,count_staffed_inpatient_icu_beds...
LongTermHospital,2023-01-27,count_staffed_inpatient_icu_beds_occupied,...
ShortTermHospital,2023-01-27,Count_StaffedBeds_Adult...
ShortTermHospital,2023-01-27,Count_StaffedBedsOccuped_Adult_ICU,...
ShortTermHospital,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU,...
ShortTermHospital,2023-01-27,Count_StaffedBeds_Inpatient_ICU,...
ShortTermHospital,2023-01-27,Count_StaffedBedsOccupied_Inpatient_ICU,...
ShortTermHospital,2023-01-27,Count_StaffedBeds,...
LongTermHospital,2023-01-27,Count_StaffedBeds_Adult,...
LongTermHospital,2023-01-27,Count_StaffedBedsOccuped_Adult_ICU,...
LongTermHospital,2023-01-27,Count_StaffedBeds_Adult_Inpatient_ICU...
LongTermHospital,2023-01-27,Count_StaffedBeds_Inpatient_ICU...
LongTermHospital,2023-01-27,ount_StaffedBedsOccupied_Inpatient_ICU,...
...
```

Expand Down
Loading
Loading