Add missing_since file/folder field and add verify and purge missing tasks - #7126
Open
WithoutPants wants to merge 23 commits into
Open
Add missing_since file/folder field and add verify and purge missing tasks#7126WithoutPants wants to merge 23 commits into
WithoutPants wants to merge 23 commits into
Conversation
…calls verifyPaths with purgeMissing set to true.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a missing_since timestamp for files and folders, replaces the old “Clean” behavior with a safer “Verify files” workflow, and adds a dedicated “Purge missing” task to remove already-marked missing items. It extends the DB schema, repository APIs, GraphQL schema/mutations, backend task implementations, and the UI tasks panel to support verifying/purging missing entries.
Changes:
- Add
missing_sincetofilesandfolders, plus indexes and filtering/sorting support. - Replace the legacy Clean task with new Verify/Purge tasks (Clean becomes deprecated wrapper behavior).
- Add UI/GraphQL wiring for verifyPaths and purgeMissing tasks (including task options and confirmations).
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| ui/v2.5/src/locales/en-GB.json | Updates task labels/descriptions and confirmation strings for Verify/Purge. |
| ui/v2.5/src/docs/en/Manual/Tasks.md | Replaces “Cleaning” documentation with Verify/Purge task documentation. |
| ui/v2.5/src/core/StashService.ts | Swaps metadataClean client call for verifyPaths/purgeMissing mutations. |
| ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx | Replaces Clean UI with Verify/Purge task UI, dialogs, and options. |
| ui/v2.5/graphql/mutations/metadata.graphql | Adds VerifyPaths and PurgeMissing mutations. |
| pkg/sqlite/migrations/86_file_missing_since.up.sql | Adds missing_since columns and partial indexes for files/folders. |
| pkg/sqlite/folder.go | Persists/queries missing_since; adds SetMissing and missing-only query helpers. |
| pkg/sqlite/folder_filter.go | Enables folder filtering by missing_since. |
| pkg/sqlite/file.go | Persists/queries missing_since; adds SetMissing, missing-only queries, and FindByFolderID. |
| pkg/sqlite/file_filter.go | Enables file filtering by missing_since. |
| pkg/sqlite/database.go | Bumps schema version to 86. |
| pkg/models/repository_folder.go | Extends folder repository interfaces for missing queries + SetMissing. |
| pkg/models/repository_file.go | Extends file repository interfaces for missing queries + SetMissing + FindByFolderID. |
| pkg/models/model_file.go | Adds MissingSince to DirEntry (propagated onto file/folder models). |
| pkg/models/folder.go | Adds missing_since to FolderFilterType. |
| pkg/models/file.go | Adds missing_since to FileFilterType. |
| pkg/file/verify.go | New verify job that marks missing items and optionally purges discovered missing items. |
| pkg/file/scan.go | Clears missing_since when scan finds previously-missing files/folders. |
| pkg/file/purge_missing.go | New job to purge files/folders already marked missing, optionally filtered by time. |
| pkg/file/handler.go | Renames CleanHandler to PurgeHandler interface for purge hooks. |
| pkg/file/clean.go | Removes legacy cleaner implementation. |
| internal/manager/task_verify.go | Adds manager task wrapper for VerifyPaths. |
| internal/manager/task_purge_missing.go | Replaces old clean task wrapper with purge-missing task wrapper and purge hooks. |
| internal/manager/manager_tasks.go | Deprecates Clean by routing it through VerifyPaths (with purge behavior). |
| internal/api/resolver_mutation_metadata.go | Adds GraphQL resolvers for verifyPaths and purgeMissing. |
| graphql/schema/types/metadata.graphql | Adds VerifyPathsInput and PurgeMissingInput types. |
| graphql/schema/types/filters.graphql | Adds missing_since criteria to file/folder filter inputs. |
| graphql/schema/types/file.graphql | Adds missing_since fields to Folder and BaseFile types. |
| graphql/schema/schema.graphql | Deprecates metadataClean and adds verifyPaths/purgeMissing mutations. |
Comments suppressed due to low confidence (4)
graphql/schema/types/file.graphql:68
- The BasicFile.missing_since field docstring incorrectly refers to a "folder" and says it is set by the clean task. This field applies to files and is set by the verify files task.
"If not null, indicates when the folder was detected as missing. Set by the clean task"
missing_since: Time
graphql/schema/types/file.graphql:92
- The VideoFile.missing_since field docstring incorrectly refers to a "folder" and says it is set by the clean task. This field applies to files and is set by the verify files task.
"If not null, indicates when the folder was detected as missing. Set by the clean task"
missing_since: Time
graphql/schema/types/file.graphql:127
- The ImageFile.missing_since field docstring incorrectly refers to a "folder" and says it is set by the clean task. This field applies to files and is set by the verify files task.
"If not null, indicates when the folder was detected as missing. Set by the clean task"
missing_since: Time
graphql/schema/types/file.graphql:159
- The GalleryFile.missing_since field docstring incorrectly refers to a "folder" and says it is set by the clean task. This field applies to files and is set by the verify files task.
"If not null, indicates when the folder was detected as missing. Set by the clean task"
missing_since: Time
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Toast.success( | ||
| intl.formatMessage( | ||
| { id: "config.tasks.added_job_to_queue" }, | ||
| { operation_name: intl.formatMessage({ id: "actions.verify" }) } |
Comment on lines
+584
to
+585
| operation_name: intl.formatMessage({ id: "actions.purge_missing" }), | ||
| } |
Comment on lines
+755
to
+780
| <VerifyDialog | ||
| dryRun={false} | ||
| purgeMissing={verifyOptions.purgeMissing ?? false} | ||
| pathSelection={dialogOpen.verify} | ||
| onClose={(p) => { | ||
| // undefined means cancelled | ||
| if (p !== undefined) { | ||
| if (dialogOpen.cleanAlert) { | ||
| if (dialogOpen.verifyAlert) { | ||
| // don't provide paths | ||
| onClean(); | ||
| onVerify(); | ||
| } else { | ||
| onClean(p); | ||
| onVerify(p); | ||
| } | ||
| } | ||
|
|
||
| setDialogOpen({ | ||
| clean: false, | ||
| cleanAlert: false, | ||
| verify: false, | ||
| verifyAlert: false, | ||
| }); | ||
| }} | ||
| /> | ||
| ) : ( | ||
| dialogOpen.clean | ||
| )} | ||
| ) : null} | ||
| {dialogOpen.purgeMissingAlert || dialogOpen.purgeMissing ? ( | ||
| <PurgeMissingDialog | ||
| dryRun={false} | ||
| pathSelection={dialogOpen.purgeMissing} |
Collaborator
There was a problem hiding this comment.
I don't think it's hardcoded so I think this is incorrect. The ?? false is just a null-default, shouldn't be an issue
Comment on lines
+482
to
+484
| function configureDefaults(partial: Record<string, object>) { | ||
| saveUI({ taskDefaults: { ...partial } }); | ||
| } |
Comment on lines
+477
to
+479
| if (taskDefaults?.verify) { | ||
| setVerifyOptions(taskDefaults.verify); | ||
| } |
Collaborator
There was a problem hiding this comment.
Disagree with this. I think this could happen but if we fix the issue on line 486 I don't think this will be an issue.
Comment on lines
+274
to
+277
| // short-cut, don't assess if already missing or added to delete set | ||
| if f.Base().MissingSince != nil || j.toDelete.has(fileID) { | ||
| continue | ||
| } |
Comment on lines
+22
to
+23
| "If not null, indicates when the folder was detected as missing. Set by the clean task" | ||
| missing_since: Time |
Comment on lines
+42
to
+44
| "If not null, indicates when the folder was detected as missing. Set by the clean task" | ||
| missing_since: Time | ||
|
|
DogmaDragon
approved these changes
Jul 24, 2026
DogmaDragon
left a comment
DogmaDragon
left a comment
Collaborator
There was a problem hiding this comment.
Documentation check passed.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (9)
ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx:554
- The toast uses message id "actions.verify", but that id is not defined in en-GB (and likely other locales), which will result in missing-translation output at runtime. Use the existing "actions.verify_files" id (or add "actions.verify" consistently across all locales).
{ operation_name: intl.formatMessage({ id: "actions.verify" }) }
ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx:584
- The purge-missing toast uses message id "actions.purge_missing", but the locale entry added in this PR is under "config.tasks.purge_missing". This will render as a missing translation.
operation_name: intl.formatMessage({ id: "actions.purge_missing" }),
ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx:756
- VerifyDialog is always passed dryRun={false}. If the user enables Dry run and opens the selective dialog, the dialog copy/behaviour will be incorrect because it can’t reflect dry-run mode.
dryRun={false}
ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx:779
- PurgeMissingDialog is always passed dryRun={false}. If the user enables Dry run and opens the selective dialog, the dialog will still show the destructive confirmation copy instead of the dry-run message.
dryRun={false}
ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx:484
- configureDefaults overwrites ui.taskDefaults with only the provided partial, which will wipe defaults for other tasks (scan/generate/cleanGenerated/etc) because saveUI only shallow-merges. Merge into existing ui.taskDefaults instead.
function configureDefaults(partial: Record<string, object>) {
saveUI({ taskDefaults: { ...partial } });
}
ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx:479
- Only verify defaults are restored from ui.taskDefaults. Purge-missing defaults are saved (onSetPurgeMissingOptions) but never reloaded, so they won’t persist across refreshes.
if (taskDefaults?.verify) {
setVerifyOptions(taskDefaults.verify);
}
pkg/file/verify.go:277
- assessFiles skips already-missing files without incrementing progress. Because the job total includes all files, this can leave the progress bar permanently short of 100% when any files are already marked missing.
// short-cut, don't assess if already missing or added to delete set
if f.Base().MissingSince != nil || j.toDelete.has(fileID) {
continue
}
graphql/schema/types/file.graphql:23
- The schema description still says missing_since is set by the "clean" task, but this PR deprecates metadataClean in favour of verifyPaths/purgeMissing. Update the description to match the new behaviour.
"If not null, indicates when the folder was detected as missing. Set by the clean task"
missing_since: Time
graphql/schema/types/file.graphql:43
- The BaseFile missing_since description refers to a "folder" and the "clean" task. This field is for files, and is now set by verifyPaths (and cleared by scan when rediscovered).
"If not null, indicates when the folder was detected as missing. Set by the clean task"
missing_since: Time
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 30 out of 32 changed files in this pull request and generated 2 comments.
Files not reviewed (2)
- pkg/models/mocks/FileReaderWriter.go: Generated file
- pkg/models/mocks/FolderReaderWriter.go: Generated file
Suppressed comments (10)
ui/v2.5/src/components/Settings/Tasks/DataManagementTasks.tsx:487
configureDefaultsreplaces the entireui.taskDefaultsobject with just the provided partial, which can drop previously saved defaults for other tasks (e.g. scan/autoTag/generate/cleanGenerated) in the local UI state and in the pending update payload. Merge with existing defaults instead of overwriting the whole object.
function configureDefaults(partial: Record<string, object>) {
saveUI({ taskDefaults: { ...partial } });
}
graphql/schema/types/file.graphql:23
- The
missing_sincefield comment still says it is set by the "clean" task, but this PR deprecates clean and setsmissing_sinceviaverifyPaths. This docstring should be updated to avoid confusing API consumers.
"If not null, indicates when the folder was detected as missing. Set by the clean task"
missing_since: Time
graphql/schema/types/file.graphql:43
- This
missing_sincedocstring says "folder" was detected as missing, butBaseFilerepresents a file. This is a copy/paste error and should say "file".
"If not null, indicates when the folder was detected as missing. Set by the verifyPaths task"
missing_since: Time
graphql/schema/types/file.graphql:67
- This
missing_sincedocstring says "folder" was detected as missing, butBasicFilerepresents a file. Update the wording to "file" for correctness.
"If not null, indicates when the folder was detected as missing. Set by the verifyPaths task"
missing_since: Time
graphql/schema/types/file.graphql:91
- This
missing_sincedocstring says "folder" was detected as missing, butVideoFilerepresents a file. Update the wording to "file" for correctness.
"If not null, indicates when the folder was detected as missing. Set by the verifyPaths task"
missing_since: Time
graphql/schema/types/file.graphql:126
- This
missing_sincedocstring says "folder" was detected as missing, butImageFilerepresents a file. Update the wording to "file" for correctness.
"If not null, indicates when the folder was detected as missing. Set by the verifyPaths task"
missing_since: Time
graphql/schema/types/file.graphql:158
- This
missing_sincedocstring says "folder" was detected as missing, butGalleryFilerepresents a file. Update the wording to "file" for correctness.
"If not null, indicates when the folder was detected as missing. Set by the verifyPaths task"
missing_since: Time
internal/manager/task_stashignore_test.go:65
- The failure message still refers to a "clean filter" even though the test now uses
verifyFilter, which makes the test output misleading when it fails.
if verifyFilter.Accept(context.Background(), ignoredFile, info, "") {
t.Fatalf("expected clean filter to reject file due to parent .stashignore")
}
ui/v2.5/src/docs/en/Manual/Tasks.md:116
- The new documentation for "Verify files" removed the prior warning about network/inaccessible media paths. Since verification marks items missing when the filesystem can’t be read (including transient network/share issues), the warning is still important to prevent accidental missing-marking/purging.
pkg/models/folder.go:25 MissingSinceis the only field inFolderFilterTypemissingomitemptyin its JSON tag, which is inconsistent with the surrounding optional filter fields. This can causemissing_since: nullto be serialized even when not provided.
MissingSince *TimestampCriterionInput `json:"missing_since"`
Comment on lines
+129
to
+135
| if len(files) != batchSize { | ||
| more = false | ||
| } else if j.options.DryRun { | ||
| // when not in dry run, we should be continuing until there's none left | ||
| // in dry run, we can just increment the offset and continue to the next batch | ||
| offset += batchSize | ||
| } |
Collaborator
There was a problem hiding this comment.
Agreed. Option 1 is probably what I would lean to. I think verify.go does this.
Comment on lines
+177
to
+183
| if len(folders) != batchSize { | ||
| more = false | ||
| } else if j.options.DryRun { | ||
| // when not in dry run, we should be continuing until there's none left | ||
| // in dry run, we can just increment the offset and continue to the next batch | ||
| offset += batchSize | ||
| } |
Gykes
requested changes
Jul 31, 2026
Gykes
left a comment
Gykes
left a comment
Collaborator
There was a problem hiding this comment.
So, I started a full review but decided to just review the Copilot ones first. Once we get those dealt with I can do another one.
Sorry, I kept getting distracted by its statements then going doing a rabbithole of trying to prove it right or wrong. Sadly I can't auto merge in any of the obvious comment fix stuff.
Comment on lines
+482
to
+484
| function configureDefaults(partial: Record<string, object>) { | ||
| saveUI({ taskDefaults: { ...partial } }); | ||
| } |
Comment on lines
+52
to
+57
| // HACK - use purge job to clean empty galleries | ||
| pj := &purgeMissingJob{ | ||
| // only need to provide repository | ||
| repository: j.repository, | ||
| } | ||
| pj.cleanEmptyGalleries(ctx) |
Collaborator
There was a problem hiding this comment.
This runs with empty options, so cleanEmptyGalleries skips its path filter and clears empty galleries library-wide even on a selective purge. Pass Paths: j.options.Paths through?
Comment on lines
+129
to
+135
| if len(files) != batchSize { | ||
| more = false | ||
| } else if j.options.DryRun { | ||
| // when not in dry run, we should be continuing until there's none left | ||
| // in dry run, we can just increment the offset and continue to the next batch | ||
| offset += batchSize | ||
| } |
Collaborator
There was a problem hiding this comment.
Agreed. Option 1 is probably what I would lean to. I think verify.go does this.
Comment on lines
+22
to
+23
| "If not null, indicates when the folder was detected as missing. Set by the clean task" | ||
| missing_since: Time |
Comment on lines
+177
to
+183
| if len(folders) != batchSize { | ||
| more = false | ||
| } else if j.options.DryRun { | ||
| // when not in dry run, we should be continuing until there's none left | ||
| // in dry run, we can just increment the offset and continue to the next batch | ||
| offset += batchSize | ||
| } |
Comment on lines
+755
to
+780
| <VerifyDialog | ||
| dryRun={false} | ||
| purgeMissing={verifyOptions.purgeMissing ?? false} | ||
| pathSelection={dialogOpen.verify} | ||
| onClose={(p) => { | ||
| // undefined means cancelled | ||
| if (p !== undefined) { | ||
| if (dialogOpen.cleanAlert) { | ||
| if (dialogOpen.verifyAlert) { | ||
| // don't provide paths | ||
| onClean(); | ||
| onVerify(); | ||
| } else { | ||
| onClean(p); | ||
| onVerify(p); | ||
| } | ||
| } | ||
|
|
||
| setDialogOpen({ | ||
| clean: false, | ||
| cleanAlert: false, | ||
| verify: false, | ||
| verifyAlert: false, | ||
| }); | ||
| }} | ||
| /> | ||
| ) : ( | ||
| dialogOpen.clean | ||
| )} | ||
| ) : null} | ||
| {dialogOpen.purgeMissingAlert || dialogOpen.purgeMissing ? ( | ||
| <PurgeMissingDialog | ||
| dryRun={false} | ||
| pathSelection={dialogOpen.purgeMissing} |
Collaborator
There was a problem hiding this comment.
I don't think it's hardcoded so I think this is incorrect. The ?? false is just a null-default, shouldn't be an issue
| Toast.success( | ||
| intl.formatMessage( | ||
| { id: "config.tasks.added_job_to_queue" }, | ||
| { operation_name: intl.formatMessage({ id: "actions.verify" }) } |
Comment on lines
+584
to
+585
| operation_name: intl.formatMessage({ id: "actions.purge_missing" }), | ||
| } |
Comment on lines
+274
to
+277
| // short-cut, don't assess if already missing or added to delete set | ||
| if f.Base().MissingSince != nil || j.toDelete.has(fileID) { | ||
| continue | ||
| } |
Comment on lines
+42
to
+44
| "If not null, indicates when the folder was detected as missing. Set by the clean task" | ||
| missing_since: Time | ||
|
|
|
This pull request has been mentioned on Stash Forum. There might be relevant details there: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
missing_sincefield to files and folders and replaces the existingCleantask withVerify filesandPurge Missingtasks.The
Verify filestask walks the filesystem and marks files and folders as missing where the system cannot read the file/folder. This works similarly to the existingCleantask, but does not remove any objects. Can optionally run as a dry run, which does not make any changes, logging only missing files and folders. Defaults to not check zip file contents, with an option to include them. Can also optionally purge the discovered missing files, folders and their associated objects and generated files, like the existingCleantask. The UI displays a confirmation dialog when this option is enabled. Can be run selectively with a list of paths.The
Purge missingtask purges files and folders previously marked as missing. Can optionally run as a dry run, which does not make any changes, logging only the files and folders that would be purged. Can be run with themissingSinceBeforeparameter which only purges files/folders where themissing_sincetime is before the provided value. This option is not currently implemented in the UI. The UI displays a confirmation dialog when not in dry mode. Can be run selectively with a list of paths.Existing
metadataCleanmutation has been marked as deprecated and runs theVerify filestask with the purge missing behaviour (unless dry run is enabled).Related Issue
None.
Testing
From memory, I've done the following testing, but this should be verified independently:
missingSinceBeforeoption correctlyfindFilesandfindFoldersqueries with themissing_sincefilter criteriamissing_sincefield when it finds a previously marked missing file/folderScreenshots
New tasks, replacing Clean task:
Verify files confirmation dialog - only used when purge missing is enabled:
Purge files confirmation dialog - used when dry run is disabled:
Additional Context
Future work will be to provide a mechanism to view missing files and folders.
The task names aren't set in stone and I'm open to suggestions for better terms.