Skip to content

Commit 0bc4baf

Browse files
authored
HDXDSYS-2471 New non-HXL resource generation functions (#93)
* Remove unused generate_qc_resource_from_rows method, test and documentation Use save_iterable in generate_resource_from_rows * Add download_generate_resource * Document new methods and deprecations * Fix tests
1 parent a96c343 commit 0bc4baf

16 files changed

Lines changed: 788 additions & 191 deletions

‎documentation/index.md‎

Lines changed: 19 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,11 @@ The library has detailed API documentation which can be found in the menu at the
5454

5555

5656
## Breaking Changes
57+
From 6.5.2, remove unused `generate_qc_resource_from_rows` method.
58+
`generate_resource_from_rows`, `generate_resource_from_iterable` and
59+
`download_and_generate_resource` are deprecated. They are replaced by
60+
`generate_resource` and `download_generate_resource`.
61+
5762
From 6.5.0, files will not be uploaded to the HDX filestore if the hash and size have
5863
not changed, but if there are any resource metadata changes, except for last_modified,
5964
they will still take place.
@@ -821,26 +826,14 @@ dictionary. HEADERS is either a row number (rows start counting at 1), or the
821826
actual headers defined as a list of strings. If not set, all rows will be
822827
treated as containing values:
823828

824-
dataset.generate_resource_from_rows("FOLDER", "FILENAME", ROWS,
825-
RESOURCE DATA, HEADERS, "ENCODING")
826-
827-
A resource for the purpose of driving QuickCharts can be generated by taking
828-
ROWS, a list of dictionaries, and producing a cut down subset from it. HXLTAGS
829-
are added as the second row after the header. The reduction in rows is
830-
performed by only outputting the rows where COLUMN_NAME has a value in
831-
QC_IDENTIFIERS. Optionally the columns that are output can be limited by
832-
specifying them in HEADERS.
833-
834-
dataset.generate_qc_resource_from_rows("FOLDER", "FILENAME", ROWS,
835-
RESOURCE DATA, HXLTAGS, "COLUMN_NAME", QC_IDENTIFIERS, HEADERS, "ENCODING")
829+
dataset.generate_resource("FOLDER", "FILENAME", ROWS, RESOURCE DATA, HEADERS,
830+
COLUMNS, "FORMAT", "ENCODING", DATECOL or YEARCOL or
831+
DATE_FUNCTION)
836832

837-
Building on these basic resource generation methods, there are more powerful
838-
ones `generate_resource_from_iterator` and `download_and_generate_resource`.
839-
840-
A resource can be generated from a given list or tuple: HEADERS and an ITERATOR
841-
which can return rows in list, tuple or dictionary form. A mapping from headers
842-
to HXL hashtags, HXLTAGS, must be provided along with the FOLDER and FILENAME
843-
where the file will be generated for upload to the filestore. The dataset
833+
The first 4 parameters are mandatory, the rest are optional. A resource can be generated
834+
from a given list or tuple or other iterable. The method returns a tuple with a bool
835+
True is the resource was addeed and a dictionary of information. FOLDER and FILENAME
836+
specify where the file will be generated for upload to the filestore. The dataset
844837
time period can optionally be set by supplying DATECOL for looking up
845838
dates or YEARCOL for looking up years. DATECOl and YEARCOL can be a column name
846839
or the index of a column. Note that any timezone information is ignored and UTC
@@ -854,40 +847,9 @@ datetime. The lowest start date and highest end date are used to set the
854847
time period and are returned in the results dictionary in keys startdate
855848
and enddate.
856849

857-
dataset.generate_resource_from_iterator(HEADERS, ITERATOR, HXLTAGS,
858-
"FOLDER", "FILENAME", RESOURCE_DATA, DATECOL or YEARCOL or DATE_FUNCTION,
859-
QUICKCHARTS, "ENCODING")
860-
861-
If desired, `generate_resource_from_iterator` can generate a separate
862-
QuickCharts resource designed to be used in a time series QuickCharts bite
863-
provided that the input has #indicator+code, #date and #indicator+value+num.
864-
This is achieved by supplying the parameter QUICKCHARTS which activates various
865-
QuickCharts related actions depending upon the keys given in the dictionary.
866-
The returned dictionary will contain the QuickCharts resource in the key
867-
qc_resource. If the keys: hashtag - the HXL hashtag to examine - and values -
868-
the 3 values to look for in that column - are supplied, then a list of booleans
869-
indicating which QuickCharts bites should be enabled will be returned in the
870-
key bites_disabled in the returned dictionary. For the 3 values, if the key:
871-
numeric_hashtag is supplied then if that column for a given value contains no
872-
numbers, then the corresponding bite will be disabled. If the key: cutdown is
873-
given, if it is 1, then a separate cut down list is created containing only
874-
columns with HXL hashtags and rows with desired values (if hashtag and values
875-
are supplied) for the purpose of driving QuickCharts. It is returned in the key
876-
qcrows in the returned dictionary with the matching headers in qcheaders. If
877-
cutdown is 2, then a resource is created using the cut down list. If the key
878-
cutdownhashtags is supplied, then only the provided hashtags are used for
879-
cutting down otherwise the full list of HXL tags is used.
880-
881-
The QuickCharts resource will be of form similar to below:
882-
883-
GHO (CODE),ENDYEAR,Numeric
884-
#indicator+code,#date+year+end,#indicator+value+num
885-
VIOLENCE_HOMICIDERATE,1994,123.4
886-
MDG_0000000001,2015,123.4
887-
888-
`download_and_generate_resource` builds on `generate_resource_from_iterator`.
889-
It uses an DOWNLOADER, an object of class `Download`, `Retrieve` or other class
890-
that implements `BaseDownload` to download from URL. Additional arguments in
850+
`download_generate_resource` builds on `generate_resource`.
851+
It uses a DOWNLOADER, an object of class `Download`, `Retrieve` or other class
852+
that implements `BaseDownload` to download from a URL. Additional arguments in
891853
**KWARGS are passed to the `get_tabular_rows` method of the DOWNLOADER.
892854

893855
Optionally, headers can be inserted at specific positions. This is achieved
@@ -897,12 +859,11 @@ row. If supplied, it takes as arguments: headers (prior to any insertions) and
897859
row (which will be in dict or list form depending upon the dict_rows argument)
898860
and outputs a modified row.
899861

900-
The rest of the arguments are the same as for
901-
`generate_resource_from_iterator`.
862+
The rest of the arguments are the same as for `generate_resource`.
902863

903-
dataset.download_and_generate_resource(DOWNLOADER, "URL", HXLTAGS,
904-
"FOLDER", "FILENAME", RESOURCE_DATA, HEADER_INSERTIONS, ROW_FUNCTION,
905-
DATECOL or YEARCOL or DATE_FUNCTION, QUICKCHARTS, **KWARGS)
864+
dataset.download_generate_resource(DOWNLOADER, "URL", "FOLDER", "FILENAME",
865+
RESOURCE_DATA, HEADER_INSERTIONS, ROW_FUNCTION,
866+
DATECOL or YEARCOL or DATE_FUNCTION, **KWARGS)
906867

907868
### QuickCharts Generation
908869

‎pyproject.toml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,8 @@ dependencies = [
3737
"ckanapi>=4.8",
3838
"defopt>=7.0.0",
3939
"email_validator",
40-
"hdx-python-country>=3.9.6",
41-
"hdx-python-utilities>=3.9.1",
40+
"hdx-python-country>=3.9.8",
41+
"hdx-python-utilities>=3.9.5",
4242
"libhxl>=5.2.2",
4343
"makefun",
4444
"quantulum3",

‎requirements.txt‎

Lines changed: 33 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ annotated-types==0.7.0
44
# via pydantic
55
astdoc==1.3.2
66
# via mkapi
7-
attrs==25.3.0
7+
attrs==25.4.0
88
# via
99
# frictionless
1010
# jsonlines
@@ -14,9 +14,9 @@ babel==2.17.0
1414
# via mkdocs-material
1515
backrefs==5.9
1616
# via mkdocs-material
17-
cachetools==5.5.2
17+
cachetools==6.2.1
1818
# via google-auth
19-
certifi==2025.8.3
19+
certifi==2025.10.5
2020
# via requests
2121
cfgv==3.4.0
2222
# via pre-commit
@@ -26,60 +26,59 @@ charset-normalizer==3.4.3
2626
# via requests
2727
ckanapi==4.8
2828
# via hdx-python-api (pyproject.toml)
29-
click==8.2.1
29+
click==8.3.0
3030
# via
3131
# mkdocs
32-
# mkdocs-material
3332
# typer
3433
colorama==0.4.6
3534
# via mkdocs-material
36-
coverage==7.10.6
35+
coverage==7.10.7
3736
# via pytest-cov
3837
defopt==7.0.0
3938
# via hdx-python-api (pyproject.toml)
4039
distlib==0.4.0
4140
# via virtualenv
42-
dnspython==2.7.0
41+
dnspython==2.8.0
4342
# via email-validator
4443
docopt==0.6.2
4544
# via
4645
# ckanapi
4746
# num2words
48-
docutils==0.22
47+
docutils==0.22.2
4948
# via defopt
5049
email-validator==2.3.0
5150
# via hdx-python-api (pyproject.toml)
5251
et-xmlfile==2.0.0
5352
# via openpyxl
54-
filelock==3.19.1
53+
filelock==3.20.0
5554
# via virtualenv
5655
frictionless==5.18.1
5756
# via hdx-python-utilities
5857
ghp-import==2.1.0
5958
# via mkdocs
60-
google-auth==2.40.3
59+
google-auth==2.41.1
6160
# via
6261
# google-auth-oauthlib
6362
# gspread
6463
google-auth-oauthlib==1.2.2
6564
# via gspread
6665
gspread==6.2.1
6766
# via hdx-python-api (pyproject.toml)
68-
hdx-python-country==3.9.6
67+
hdx-python-country==3.9.8
6968
# via hdx-python-api (pyproject.toml)
70-
hdx-python-utilities==3.9.1
69+
hdx-python-utilities==3.9.5
7170
# via
7271
# hdx-python-api (pyproject.toml)
7372
# hdx-python-country
7473
humanize==4.13.0
7574
# via frictionless
76-
identify==2.6.13
75+
identify==2.6.15
7776
# via pre-commit
78-
idna==3.10
77+
idna==3.11
7978
# via
8079
# email-validator
8180
# requests
82-
ijson==3.4.0
81+
ijson==3.4.0.post0
8382
# via hdx-python-utilities
8483
inflect==7.5.0
8584
# via quantulum3
@@ -101,7 +100,7 @@ jsonschema==4.25.1
101100
# via
102101
# frictionless
103102
# tableschema-to-template
104-
jsonschema-specifications==2025.4.1
103+
jsonschema-specifications==2025.9.1
105104
# via jsonschema
106105
libhxl==5.2.2
107106
# via
@@ -111,7 +110,7 @@ loguru==0.7.3
111110
# via hdx-python-utilities
112111
makefun==1.16.0
113112
# via hdx-python-api (pyproject.toml)
114-
markdown==3.8.2
113+
markdown==3.9
115114
# via
116115
# mkdocs
117116
# mkdocs-material
@@ -120,7 +119,7 @@ markdown-it-py==4.0.0
120119
# via rich
121120
marko==2.2.0
122121
# via frictionless
123-
markupsafe==3.0.2
122+
markupsafe==3.0.3
124123
# via
125124
# jinja2
126125
# mkdocs
@@ -130,15 +129,15 @@ mergedeep==1.3.4
130129
# via
131130
# mkdocs
132131
# mkdocs-get-deps
133-
mkapi==4.4.5
132+
mkapi==4.5.0
134133
# via hdx-python-api (pyproject.toml)
135134
mkdocs==1.6.1
136135
# via
137136
# mkapi
138137
# mkdocs-material
139138
mkdocs-get-deps==0.2.0
140139
# via mkdocs
141-
mkdocs-material==9.6.18
140+
mkdocs-material==9.6.21
142141
# via mkapi
143142
mkdocs-material-extensions==1.3.1
144143
# via mkdocs-material
@@ -162,7 +161,7 @@ pathspec==0.12.1
162161
# via mkdocs
163162
petl==1.7.17
164163
# via frictionless
165-
platformdirs==4.4.0
164+
platformdirs==4.5.0
166165
# via
167166
# mkdocs-get-deps
168167
# virtualenv
@@ -184,9 +183,9 @@ pyasn1==0.6.1
184183
# rsa
185184
pyasn1-modules==0.4.2
186185
# via google-auth
187-
pydantic==2.11.7
186+
pydantic==2.12.0
188187
# via frictionless
189-
pydantic-core==2.33.2
188+
pydantic-core==2.41.1
190189
# via pydantic
191190
pygments==2.19.2
192191
# via
@@ -197,14 +196,14 @@ pymdown-extensions==10.16.1
197196
# via mkdocs-material
198197
pyphonetics==0.5.3
199198
# via hdx-python-utilities
200-
pytest==8.4.1
199+
pytest==8.4.2
201200
# via
202201
# hdx-python-api (pyproject.toml)
203202
# pytest-check
204203
# pytest-cov
205-
pytest-check==2.5.3
204+
pytest-check==2.6.0
206205
# via hdx-python-api (pyproject.toml)
207-
pytest-cov==6.2.1
206+
pytest-cov==7.0.0
208207
# via hdx-python-api (pyproject.toml)
209208
python-dateutil==2.9.0.post0
210209
# via
@@ -218,7 +217,7 @@ python-slugify==8.0.4
218217
# via
219218
# ckanapi
220219
# frictionless
221-
pyyaml==6.0.2
220+
pyyaml==6.0.3
222221
# via
223222
# frictionless
224223
# mkdocs
@@ -252,7 +251,7 @@ requests-oauthlib==2.0.0
252251
# via google-auth-oauthlib
253252
rfc3986==2.0.0
254253
# via frictionless
255-
rich==14.1.0
254+
rich==14.2.0
256255
# via typer
257256
rpds-py==0.27.1
258257
# via
@@ -262,15 +261,15 @@ rsa==4.9.1
262261
# via google-auth
263262
ruamel-yaml==0.18.15
264263
# via hdx-python-utilities
265-
ruamel-yaml-clib==0.2.12
264+
ruamel-yaml-clib==0.2.14
266265
# via ruamel-yaml
267266
setuptools==80.9.0
268267
# via ckanapi
269268
shellingham==1.5.4
270269
# via typer
271270
simpleeval==1.0.3
272271
# via frictionless
273-
simplejson==3.20.1
272+
simplejson==3.20.2
274273
# via ckanapi
275274
six==1.17.0
276275
# via
@@ -292,7 +291,7 @@ text-unidecode==1.3
292291
# via python-slugify
293292
typeguard==4.4.4
294293
# via inflect
295-
typer==0.17.3
294+
typer==0.19.2
296295
# via frictionless
297296
typing-extensions==4.15.0
298297
# via
@@ -302,7 +301,7 @@ typing-extensions==4.15.0
302301
# typeguard
303302
# typer
304303
# typing-inspection
305-
typing-inspection==0.4.1
304+
typing-inspection==0.4.2
306305
# via pydantic
307306
unidecode==1.4.0
308307
# via
@@ -314,7 +313,7 @@ urllib3==2.5.0
314313
# requests
315314
validators==0.35.0
316315
# via frictionless
317-
virtualenv==20.34.0
316+
virtualenv==20.35.3
318317
# via pre-commit
319318
watchdog==6.0.0
320319
# via mkdocs
@@ -326,7 +325,7 @@ xlrd3==1.1.0
326325
# via libhxl
327326
xlsx2csv==0.8.4
328327
# via hdx-python-utilities
329-
xlsxwriter==3.2.5
328+
xlsxwriter==3.2.9
330329
# via tableschema-to-template
331330
xlwt==1.3.0
332331
# via hdx-python-utilities

0 commit comments

Comments
 (0)