Skip to content

Commit 6832186

Browse files
authored
Add Activity class for CKAN activity streams (#116)
Wraps package/organization/group/user_activity_list, recently_changed_packages_activity_list, activity_show, activity_data_show and activity_diff. Dataset, Organization, Location and User gain get_activities().
1 parent 90f8935 commit 6832186

7 files changed

Lines changed: 574 additions & 0 deletions

File tree

‎documentation/index.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ upload your datasets to HDX.
3535
- [Organization Management](#organization-management)
3636
- [Location Management](#location-management)
3737
- [Vocabulary Management](#vocabulary-management)
38+
- [Activity Streams](#activity-streams)
3839
- [Pipeline State](#pipeline-state)
3940
- [Working Examples](#working-examples)
4041
- [Project Framework](#project-framework)
@@ -1188,6 +1189,40 @@ If you want to add a list of tags, you do it as follows:
11881189

11891190
vocabulary.add_tags(["TAG","TAG","TAG"...])
11901191

1192+
## Activity Streams
1193+
1194+
The **Activity** class wraps CKAN activities, which record who changed what and when.
1195+
Activities are read only. You can get the activity stream of a dataset, organization,
1196+
location or user:
1197+
1198+
activities = dataset.get_activities(**kwargs)
1199+
activities = organization.get_activities(**kwargs)
1200+
activities = location.get_activities(**kwargs)
1201+
activities = user.get_activities(**kwargs)
1202+
1203+
These return a list of **Activity** objects, each with fields like `timestamp`,
1204+
`user_id`, `object_id` and `activity_type` (eg. "changed package"). One page of results
1205+
is returned. Its size is set by `limit` (default 31, HDX maximum 100) and where it starts
1206+
by `offset`. Passing `all_pages=True` gets every page. Other arguments (`**kwargs`) like
1207+
`after`, `before` and `activity_types` depend on the object type and are detailed in the
1208+
API documentation. Getting a user's activity stream requires authorisation to view the
1209+
user's profile.
1210+
1211+
You can get the activity stream of all recently added or changed datasets:
1212+
1213+
activities = Activity.get_recently_changed_packages(limit=10)
1214+
1215+
With `all_pages=True`, this stream covers the whole history of HDX so it is capped by
1216+
`max_activities` (default 1000). Pass `max_activities=None` to remove the cap.
1217+
1218+
You can read a single activity, get its data (including a snapshot of the object at the
1219+
time of the activity) or get a diff against the object's previous version. HDX requires
1220+
permission to edit the object the activity relates to for these:
1221+
1222+
activity = Activity.read_from_hdx("ACTIVITY_ID")
1223+
data = activity.get_data()
1224+
diff = activity.get_diff(diff_type="unified") # or "context" or "html"
1225+
11911226
## Pipeline State
11921227

11931228
The HDXState class allows the reading and writing of state to a given dataset. Input

‎src/hdx/data/activity.py‎

Lines changed: 245 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,245 @@
1+
"""Activity class containing all logic for reading activity streams."""
2+
3+
import logging
4+
from collections.abc import Sequence
5+
from typing import Any, Optional
6+
7+
from hdx.api.configuration import Configuration
8+
from hdx.data.hdxobject import HDXError, HDXObject
9+
10+
logger = logging.getLogger(__name__)
11+
12+
13+
class Activity(HDXObject):
14+
"""Activity class containing all logic for reading activity streams. Activities
15+
are read only.
16+
17+
Args:
18+
initial_data: Initial activity metadata dictionary. Defaults to None.
19+
configuration: HDX configuration. Defaults to global configuration.
20+
"""
21+
22+
def __init__(
23+
self,
24+
initial_data: dict | None = None,
25+
configuration: Configuration | None = None,
26+
) -> None:
27+
if not initial_data:
28+
initial_data = {}
29+
super().__init__(initial_data, configuration=configuration)
30+
31+
@staticmethod
32+
def actions() -> dict[str, str]:
33+
"""Dictionary of actions that can be performed on object
34+
35+
Returns:
36+
Dictionary of actions that can be performed on object
37+
"""
38+
return {
39+
"show": "activity_show",
40+
"data_show": "activity_data_show",
41+
"diff": "activity_diff",
42+
"package_list": "package_activity_list",
43+
"organization_list": "organization_activity_list",
44+
"group_list": "group_activity_list",
45+
"user_list": "user_activity_list",
46+
"recently_changed_packages": "recently_changed_packages_activity_list",
47+
}
48+
49+
@classmethod
50+
def read_from_hdx(
51+
cls, identifier: str, configuration: Configuration | None = None
52+
) -> Optional["Activity"]:
53+
"""Reads the activity given by identifier from HDX and returns Activity object.
54+
HDX requires permission to edit the object the activity relates to.
55+
56+
Args:
57+
identifier: Identifier of activity
58+
configuration: HDX configuration. Defaults to global configuration.
59+
60+
Returns:
61+
Activity object if successful read, None if not
62+
"""
63+
return cls._read_from_hdx_class("activity", identifier, configuration)
64+
65+
def get_object_type(self) -> str:
66+
"""Get the CKAN object type the activity relates to eg. package,
67+
organization, group or user. This is the last word of the activity type
68+
(eg. "changed package") with "dataset" (from "follow dataset") mapped to
69+
"package".
70+
71+
Returns:
72+
Object type
73+
"""
74+
object_type = self.data["activity_type"].split(" ")[-1]
75+
if object_type == "dataset":
76+
return "package"
77+
return object_type
78+
79+
def get_data(self) -> dict:
80+
"""Get the data of the activity which includes a snapshot of the object
81+
at the time of the activity. HDX requires permission to edit the object
82+
the activity relates to.
83+
84+
Returns:
85+
Activity data
86+
"""
87+
return self._write_to_hdx("data_show", {"id": self.data["id"]}, "id")
88+
89+
def get_diff(self, diff_type: str = "unified") -> dict:
90+
"""Get a diff of the object compared to its previous version. HDX requires
91+
permission to edit the object the activity relates to.
92+
93+
Args:
94+
diff_type: One of unified, context or html. Defaults to unified.
95+
96+
Returns:
97+
Activity diff
98+
"""
99+
data = {
100+
"id": self.data["id"],
101+
"object_type": self.get_object_type(),
102+
"diff_type": diff_type,
103+
}
104+
return self._write_to_hdx("diff", data, "id")
105+
106+
@classmethod
107+
def _get_list(
108+
cls,
109+
action: str,
110+
data: dict,
111+
configuration: Configuration | None = None,
112+
all_pages: bool = False,
113+
max_activities: int | None = None,
114+
) -> list["Activity"]:
115+
"""Helper method to get a list of activities from HDX
116+
117+
Args:
118+
action: Action to perform
119+
data: Data to pass to CKAN including any offset and limit
120+
configuration: HDX configuration. Defaults to global configuration.
121+
all_pages: Whether to get all pages of results. Defaults to False.
122+
max_activities: Maximum number of activities if all_pages is True. Defaults to None (no maximum).
123+
124+
Returns:
125+
List of Activity objects
126+
"""
127+
activity = cls(configuration=configuration)
128+
if not all_pages:
129+
result = activity._write_to_hdx(action, data)
130+
return [cls(x, configuration=configuration) for x in result]
131+
data = dict(data)
132+
limit = data.setdefault("limit", 100)
133+
offset = data.get("offset", 0)
134+
activities = []
135+
while True:
136+
data["offset"] = offset
137+
result = activity._write_to_hdx(action, data)
138+
activities.extend(cls(x, configuration=configuration) for x in result)
139+
if max_activities is not None and len(activities) >= max_activities:
140+
return activities[:max_activities]
141+
if len(result) < limit:
142+
return activities
143+
offset += limit
144+
145+
@classmethod
146+
def get_activities(
147+
cls,
148+
object_type: str,
149+
identifier: str,
150+
configuration: Configuration | None = None,
151+
all_pages: bool = False,
152+
**kwargs: Any,
153+
) -> list["Activity"]:
154+
"""Get the activity stream of a CKAN object. One page of results (whose
155+
size is given by limit) is returned unless all_pages is True.
156+
157+
Args:
158+
object_type: One of package, organization, group or user
159+
identifier: Id or name of object
160+
configuration: HDX configuration. Defaults to global configuration.
161+
all_pages: Whether to get all pages of results. Defaults to False.
162+
**kwargs: See below
163+
offset (int): Where to start getting activity items from. Defaults to 0.
164+
limit (int): Maximum number of activities per page. Defaults to 31 (HDX maximum 100) or 100 if all_pages is True.
165+
after (int | str): Only activities after this timestamp (package and user only)
166+
before (int | str): Only activities before this timestamp (package and user only)
167+
include_hidden_activity (bool): Include hidden activity, sysadmins only (package, organization and group only)
168+
activity_types (list): Activity types to include (package only)
169+
exclude_activity_types (list): Activity types to exclude (package only)
170+
171+
Returns:
172+
List of Activity objects
173+
"""
174+
if not identifier:
175+
raise HDXError(f"No id or name for {object_type} to get activities!")
176+
data = {"id": identifier}
177+
data.update(kwargs)
178+
return cls._get_list(f"{object_type}_list", data, configuration, all_pages)
179+
180+
@classmethod
181+
def get_recently_changed_packages(
182+
cls,
183+
configuration: Configuration | None = None,
184+
all_pages: bool = False,
185+
max_activities: int | None = 1000,
186+
**kwargs: Any,
187+
) -> list["Activity"]:
188+
"""Get the activity stream of all recently added or changed datasets.
189+
One page of results (whose size is given by limit) is returned unless
190+
all_pages is True. all_pages is capped by max_activities because the
191+
stream covers the whole history of HDX.
192+
193+
Args:
194+
configuration: HDX configuration. Defaults to global configuration.
195+
all_pages: Whether to get all pages of results. Defaults to False.
196+
max_activities: Maximum number of activities if all_pages is True. Defaults to 1000. None means no maximum (very slow).
197+
**kwargs: See below
198+
offset (int): Where to start getting activity items from. Defaults to 0.
199+
limit (int): Maximum number of activities per page. Defaults to 31 (HDX maximum 100) or 100 if all_pages is True.
200+
201+
Returns:
202+
List of Activity objects
203+
"""
204+
return cls._get_list(
205+
"recently_changed_packages",
206+
kwargs,
207+
configuration,
208+
all_pages,
209+
max_activities,
210+
)
211+
212+
def check_required_fields(self, ignore_fields: Sequence[str] = ()) -> None:
213+
"""Activities are read only so this raises an exception
214+
215+
Args:
216+
ignore_fields: Fields to ignore. Default is ().
217+
218+
Returns:
219+
None
220+
"""
221+
raise HDXError("Activities are read only!")
222+
223+
def update_in_hdx(self, **kwargs: Any) -> None:
224+
"""Activities are read only so this raises an exception
225+
226+
Returns:
227+
None
228+
"""
229+
raise HDXError("Activities are read only!")
230+
231+
def create_in_hdx(self, **kwargs: Any) -> None:
232+
"""Activities are read only so this raises an exception
233+
234+
Returns:
235+
None
236+
"""
237+
raise HDXError("Activities are read only!")
238+
239+
def delete_from_hdx(self) -> None:
240+
"""Activities are read only so this raises an exception
241+
242+
Returns:
243+
None
244+
"""
245+
raise HDXError("Activities are read only!")

‎src/hdx/data/dataset.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040
from hdx.api.locations import Locations
4141
from hdx.api.utilities.date_helper import DateHelper
4242
from hdx.api.utilities.filestore_helper import FilestoreHelper
43+
from hdx.data.activity import Activity
4344
from hdx.data.hdxobject import HDXError, HDXObject
4445
from hdx.data.resource_matcher import ResourceMatcher
4546

@@ -1428,6 +1429,32 @@ def get_all_resources(
14281429
resources.append(resource)
14291430
return resources
14301431

1432+
def get_activities(self, all_pages: bool = False, **kwargs: Any) -> list[Activity]:
1433+
"""Get the activity stream of this dataset. One page of results (whose size
1434+
is given by limit) is returned unless all_pages is True.
1435+
1436+
Args:
1437+
all_pages: Whether to get all pages of results. Defaults to False.
1438+
**kwargs: See below
1439+
offset (int): Where to start getting activity items from. Defaults to 0.
1440+
limit (int): Maximum number of activities per page. Defaults to 31 (HDX maximum 100) or 100 if all_pages is True.
1441+
after (int | str): Only activities after this timestamp
1442+
before (int | str): Only activities before this timestamp
1443+
include_hidden_activity (bool): Include hidden activity (sysadmins only)
1444+
activity_types (list): Activity types to include
1445+
exclude_activity_types (list): Activity types to exclude
1446+
1447+
Returns:
1448+
List of Activity objects
1449+
"""
1450+
return Activity.get_activities(
1451+
"package",
1452+
self.data.get("id") or self.data.get("name"),
1453+
self.configuration,
1454+
all_pages,
1455+
**kwargs,
1456+
)
1457+
14311458
@classmethod
14321459
def autocomplete(
14331460
cls,

‎src/hdx/data/location.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
from typing import Any, Optional
66

77
from hdx.api.configuration import Configuration
8+
from hdx.data.activity import Activity
89
from hdx.data.hdxobject import HDXObject
910

1011
logger = logging.getLogger(__name__)
@@ -217,6 +218,28 @@ def get_data_grid_countries(
217218
if len(group["name"]) == 3 and group.get("data_completeness") == "active"
218219
)
219220

221+
def get_activities(self, all_pages: bool = False, **kwargs: Any) -> list[Activity]:
222+
"""Get the activity stream of this location. One page of results (whose size
223+
is given by limit) is returned unless all_pages is True.
224+
225+
Args:
226+
all_pages: Whether to get all pages of results. Defaults to False.
227+
**kwargs: See below
228+
offset (int): Where to start getting activity items from. Defaults to 0.
229+
limit (int): Maximum number of activities per page. Defaults to 31 (HDX maximum 100) or 100 if all_pages is True.
230+
include_hidden_activity (bool): Include hidden activity (sysadmins only)
231+
232+
Returns:
233+
List of Activity objects
234+
"""
235+
return Activity.get_activities(
236+
"group",
237+
self.data.get("id") or self.data.get("name"),
238+
self.configuration,
239+
all_pages,
240+
**kwargs,
241+
)
242+
220243
@classmethod
221244
def autocomplete(
222245
cls,

0 commit comments

Comments
 (0)