Skip to content

Commit 8316396

Browse files
mpcenmpcen
authored
Merge pull request #9 from qtomlinson/qt/post_optimization_cleanup
Post optimization clean up and documentation
2 parents 7442733 + 7d85ce2 commit 8316396

7 files changed

Lines changed: 12 additions & 77 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ CURATION_STORE_PROVIDER="mongo"
113113
DEFINITION_STORE_PROVIDER="mongo"
114114
DEFINITION_MONGO_CONNECTION_STRING="mongodb://clearlydefined_mongo_db"
115115
DEFINITION_MONGO_DB_NAME="clearlydefined"
116-
DEFINITION_MONGO_COLLECTION_NAME="definitions-paged"
116+
DEFINITION_MONGO_TRIMMED_COLLECTION_NAME="definitions-trimmed"
117117
118118
# Harvest Store Info
119119
HARVEST_STORE_PROVIDER="file"
@@ -249,11 +249,10 @@ This container holds a Mongo database called **clearlydefined**
249249

250250
The database contains two collections:
251251
* curations (contains curations)
252-
* definitions-paged (contains definitions)
252+
* definitions-trimmed (contains definitions)
253253

254-
The reason the definitions database is called definitions-paged is because, previously, the definitions collection was not paged. The pagination was added in [this January 2019 pull request](https://github.com/clearlydefined/service/pull/364). Our production Azure setup includes both definitions and definitions-paged collections - the definitions-paged collection is the one that is actively used. This development environment includes the definitions-paged collection in order to more closely mirror production.
255-
256-
These collections are seeded using the Clearly Defined Mongo Seed container.
254+
The reason the definitions database is called definitions-trimmed is because, previously, the definitions collection was paged. The pagination was added in [this January 2019 pull request](https://github.com/clearlydefined/service/pull/364). To improve performance and reduce cost of the definition database, [this Feb 2023 pull request](https://github.com/clearlydefined/service/pull/976) subsequently stores definitions without files. Our production Azure setup includes definitions-trimmed, that is actively used. This development environment includes the definitions-trimmed collection in order to mirror production.
255+
This collection is seeded using the Clearly Defined Mongo Seed container.
257256

258257
If you have [mongodb](https://docs.mongodb.com/manual/installation/) installed on your local system, you can attach to the Mongo database with:
259258

@@ -266,7 +265,7 @@ You can also do this through the [Docker desktop client](https://www.docker.com/
266265

267266
### Clearly Defined Mongo Seed
268267

269-
This container exists only to seed initial data into the Clearly Defined Mongo DB. It populates both the collections and definitions-paged collections with sample data.
268+
This container exists only to seed initial data into the Clearly Defined Mongo DB. It populates both the collections and definitions-trimmed collections with sample data.
270269

271270
## Using
272271

‎docker-compose.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,5 +47,8 @@ services:
4747
- "27017:27017"
4848
clearlydefined_mongo_seed:
4949
image: "clearlydefined/docker_dev_env_experiment_clearlydefined_mongo_seed"
50+
build:
51+
context: ./mongo_seed
52+
dockerfile: Dockerfile
5053
links:
5154
- clearlydefined_mongo_db

‎mongo_seed/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Copyright (c) Microsoft Corporation and others. Licensed under the MIT license.
22
# SPDX-License-Identifier: MIT
33

4-
FROM mongo
4+
FROM mongo:5.0.6
55
RUN mkdir /definitions
66
COPY ./definitions/* definitions/
77
COPY ./curations/* curations/

‎mongo_seed/definitions/angular.json‎

Lines changed: 0 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -32,62 +32,6 @@
3232
"source": 70
3333
}
3434
},
35-
"files": [
36-
{
37-
"path": "package/LICENSE.md",
38-
"license": "MIT",
39-
"natures": [
40-
"license"
41-
],
42-
"attributions": [
43-
"Copyright (c) 2016 Angular"
44-
],
45-
"hashes": {
46-
"sha1": "682346b34c58e0a9d2deb60ccd8b8c55cb474cea"
47-
}
48-
},
49-
{
50-
"path": "package/bower.json",
51-
"hashes": {
52-
"sha1": "d57e302f201d7ccff4430b1a81611812b210fb15"
53-
}
54-
},
55-
{
56-
"path": "package/angular.min.js",
57-
"attributions": [
58-
"(c) 2010-2018 Google, Inc. http://angularjs.org"
59-
],
60-
"hashes": {
61-
"sha1": "382ea6ce5bb85a666aa9fb35a4c24eed58441258"
62-
}
63-
},
64-
{
65-
"path": "package/package.json",
66-
"license": "MIT",
67-
"hashes": {
68-
"sha1": "c479ce5c51b4b5dac7b7a79700f22a50343475f9"
69-
}
70-
},
71-
{
72-
"path": "package/angular.js",
73-
"attributions": [
74-
"(c) 2010-2018 Google, Inc. http://angularjs.org"
75-
],
76-
"hashes": {
77-
"sha1": "4eb1606146f3aca57d69d1b55c7b0384f538b77d"
78-
}
79-
},
80-
{
81-
"path": "package/README.md",
82-
"license": "MIT",
83-
"attributions": [
84-
"Copyright (c) 2010-2015 Google, Inc. http://angularjs.org"
85-
],
86-
"hashes": {
87-
"sha1": "fc6245aa21a1c521fc39955cc04d3959c68aee95"
88-
}
89-
}
90-
],
9135
"licensed": {
9236
"declared": "MIT",
9337
"toolScore": {
@@ -139,10 +83,5 @@
13983
"scores": {
14084
"effective": 84,
14185
"tool": 84
142-
},
143-
"_mongo": {
144-
"partitionKey": "npm/npmjs/-/angular/1.6.9",
145-
"page": 1,
146-
"totalPages": 1
14786
}
14887
}]

‎mongo_seed/definitions/flyway-maven-plugin.json‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,11 +63,6 @@
6363
"scores": {
6464
"effective": 57,
6565
"tool": 57
66-
},
67-
"_mongo": {
68-
"partitionKey": "maven/mavencentral/org.flywaydb/flyway-maven-plugin/5.0.7",
69-
"page": 1,
70-
"totalPages": 1
7166
}
7267
}
7368
]

‎mongo_seed/seed_data.sh‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
# Copyright (c) Microsoft Corporation and others. Licensed under the MIT license.
22
# SPDX-License-Identifier: MIT
33

4-
mongoimport --host clearlydefined_mongo_db --db clearlydefined --collection definitions-paged --type json --file definitions/angular.json --jsonArray
5-
mongoimport --host clearlydefined_mongo_db --db clearlydefined --collection definitions-paged --type json --file definitions/babylon.json --jsonArray
6-
mongoimport --host clearlydefined_mongo_db --db clearlydefined --collection definitions-paged --type json --file definitions/flyway-maven-plugin.json --jsonArray
4+
mongoimport --host clearlydefined_mongo_db --db clearlydefined --collection definitions-trimmed --type json --file definitions/angular.json --jsonArray
5+
mongoimport --host clearlydefined_mongo_db --db clearlydefined --collection definitions-trimmed --type json --file definitions/flyway-maven-plugin.json --jsonArray
76
mongoimport --host clearlydefined_mongo_db --db clearlydefined --collection curations --type json --file curations/387.json --jsonArray

‎sample_env‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ GITLAB_TOKEN="<your GitLab token (unless you are working on code that interacts
1818
DEFINITION_STORE_PROVIDER="mongo"
1919
DEFINITION_MONGO_CONNECTION_STRING="mongodb://clearlydefined_mongo_db"
2020
DEFINITION_MONGO_DB_NAME="clearlydefined"
21-
DEFINITION_MONGO_COLLECTION_NAME="definitions-paged"
21+
DEFINITION_MONGO_TRIMMED_COLLECTION_NAME="definitions-trimmed"
2222

2323
# Harvest Store Info
2424
HARVEST_STORE_PROVIDER="file"

0 commit comments

Comments
 (0)