[DMP 2026]: Enhancement: Anonymous Profile Data Consistency, Completeness and Count Reconciliation Across Registration Modules

[Varaniya201]

Ticket Contents

Description

As a country representative, I want the Registration Processor reporting data (Anonymous Profile) to accurately represent all processed registration packets so that reporting generated from Elasticsearch/Kibana is complete, consistent, and trustworthy across registration, transaction, and anonymous profile data sources.

As-Is
Multiple inconsistencies have been observed between the Registration Processor database tables and the generated Anonymous Profile data.
The reported issues include:

• The total number of records in the registration, registration_list, and anonymous_profile tables do not match.
• The number of processed packets differs from the number of generated Anonymous Profiles.
• Anonymous Profiles generated at different processing stages (SYNC, CredentialRequestorStage, UinGeneratorStage) contain inconsistent counts.
• Demographic attributes (for example, gender values) are present in some processing stages but missing in others.
• Packet uploads performed through DSL scripts also produce Anonymous Profile counts that do not match the registration records.
• Packet reprocessing may create additional Anonymous Profile records or leave existing records inconsistent depending on the processing stage.
• These inconsistencies reduce confidence in operational dashboards and reports built on top of Anonymous Profile data.
  To-Be
• Perform a comprehensive investigation and enhancement of the Anonymous Profile generation flow to ensure reporting consistency across the Registration Processor.
• The enhancement should:
  • Identify the root cause(s) of all reported count mismatches.
  • Analyse differences across Registration Processor tables and Anonymous Profile records.
  • Validate Anonymous Profile generation across every processing stage.
  • Ensure each registration packet produces the expected reporting data.
  • Define and implement the appropriate fixes to eliminate inconsistencies.
  • Maintain backward compatibility with the existing reporting pipeline.

JIRA Link: https://mosip.atlassian.net/browse/MOSIP-42841?search_id=ba59ea4f-4911-474a-864c-b18d88a587e3
https://mosip.atlassian.net/browse/MOSIP-36602

Goals & Mid-Point Milestone

Goals

• Perform complete Root Cause Analysis (RCA) for all reported Anonymous Profile mismatches.
• Identify every scenario that results in missing, duplicate, or inconsistent Anonymous Profiles.
• Validate Anonymous Profile generation across all Registration Processor stages.
• Ensure reporting counts remain consistent between:
  - [ ] Registration
  - [ ] Registration List
  - [ ] Registration Transaction
  - [ ] Anonymous Profile
• Validate packet reprocessing behaviour.
• Validate demographic information consistency across processing stages.
• Document implementation approach before code changes.
• Implement the agreed solution to resolve identified issues.
• Preserve backward compatibility.

Mid-Point Milestone

• Root Cause Analysis document completed.
• All mismatch scenarios identified and categorized.
• Impact assessment completed.
• Proposed implementation approach documented.
• Design review completed.
• Implementation plan approved.

Setup/Installation

This enhancement will be implemented as part of the Registration Processor.
Please refer to the existing Registration Processor developer guide for setup instructions.

Attaching a video links to understand how to reproduce this mismatch

https://github.com/user-attachments/assets/82113024-d53c-440a-9cdc-0e9776ec662f
https://github.com/user-attachments/assets/24c90e08-6ae7-4a76-987f-8e723d558c22

Expected Outcome

• Anonymous Profile counts match Registration Processor data.
• No unexpected duplicate Anonymous Profiles.
• No missing Anonymous Profiles for processed packets.
• Reporting counts remain consistent across all processing stages.
• Demographic fields remain consistent throughout packet lifecycle.
• Packet reprocessing does not introduce reporting inconsistencies.
• Existing dashboards continue functioning without modification.

Acceptance Criteria

Acceptance Criteria

• The team shall perform a complete Root Cause Analysis covering all reported Anonymous Profile count mismatches.
• The Root Cause Analysis shall identify every scenario resulting in missing, duplicate, or inconsistent Anonymous Profiles.
• The implementation plan shall clearly describe the proposed resolution for each identified issue.
• Anonymous Profile counts shall be consistent with processed registration packets after implementation.
• Registration, Registration List, Registration Transaction, and Anonymous Profile tables shall remain logically consistent.
• Anonymous Profile generation shall be validated across all supported processing stages.
• Packet reprocessing shall not introduce duplicate or inconsistent Anonymous Profiles unless explicitly expected.
• Demographic information stored within Anonymous Profiles shall remain consistent across processing stages.
• Existing reporting APIs and dashboards shall continue functioning without modification.
• Historical Anonymous Profiles shall remain accessible without requiring data migration.

Alternate Scenarios

Scenario ID	Scenario	Condition	System Behaviour	User/System Message
AS-01	Normal packet processing	Packet successfully completes processing	Exactly one expected Anonymous Profile is generated for applicable reporting stages	Packet processed successfully
AS-02	Packet reprocessing	Previously processed packet is reprocessed	Anonymous Profile remains consistent without creating unexpected duplicates	Packet reprocessed successfully
AS-03	DSL bulk upload	Packets uploaded using DSL scripts	Anonymous Profile count matches successfully processed registrations	Bulk upload completed successfully
AS-04	Multiple processing stages	Anonymous Profile generated at different workflow stages	Reporting data remains internally consistent across stages	Anonymous Profile generated successfully
AS-05	Historical data validation	Existing Anonymous Profiles are analysed	Historical records continue to remain accessible without affecting reporting	Historical data validated successfully

Scenario ID	Scenario	Condition	System Behaviour	User/System Message
ES-01	Missing Anonymous Profile	Registration completed but profile not generated	Missing profile identified during investigation and root cause documented	Anonymous Profile missing
ES-02	Duplicate Anonymous Profile	Multiple profiles generated unexpectedly	Duplicate generation scenario identified and analysed	Duplicate Anonymous Profile detected
ES-03	Count mismatch	Registration and Anonymous Profile counts differ	Root cause identified and implementation planned	Reporting count mismatch detected
ES-04	Demographic inconsistency	Demographic fields differ across stages	Root cause identified and corrective approach documented	Inconsistent reporting data detected
ES-05	Reprocessing inconsistency	Packet reprocessing changes reporting unexpectedly	Behaviour analysed and corrected	Reprocessing reporting inconsistency detected
ES-06	Implementation validation failure	Proposed fix does not resolve all mismatch scenarios	Additional investigation performed before closure	Validation failed

Implementation Details

This enhancement focuses on improving the reliability and consistency of the Registration Processor reporting pipeline by investigating and resolving Anonymous Profile inconsistencies.
The work includes analysing the complete lifecycle of Anonymous Profile generation beginning from registration through every processing stage until reporting.
The enhancement combines investigation, design, and implementation into a single effort.

Mockups/Wireframes

NA

Product Name

MOSIP

Organisation Name

RCTS-IIITH

Domain

Open Source Library

Tech Skills Needed

Java

Mentor(s)

@ashok-ksharma @dhanendra06

Category

Backend