Add OpenCRE mapping script with safe fallback handling

[Nik-ui]

This PR covers issue #623.

• This PR handles the issue and requires no additional PRs.
• You have validated the need for this change.

What did this PR accomplish?

• Introduced an OpenCRE enrichment script for WSTG checklist entries
• Implemented safe fallback handling for missing mappings to prevent runtime failures
• Added logging for unmapped WSTG test IDs to improve visibility
• Ensured existing checklist data remains unchanged when mappings are unavailable

[Nik-ui]

Hi @kingthorin

This PR introduces OpenCRE mapping enrichment with fallback handling for missing mappings.
Happy to make any adjustments based on feedback , thanks for your review.

[kingthorin]

Could you remove the unrelated commits?

If you're not comfortable with that it's okay, I can tiddy it up. Just let me know.

[Nik-ui]

Could you remove the unrelated commits?

If you're not comfortable with that it's okay, I can tiddy it up. Just let me know.

@kingthorin Okay, i will work on this.

[Nik-ui]

Thanks @kingthorin, I have cleaned up the branch and removed the unrelated commits. Let me know if anything else should be adjusted.

[kingthorin]

Need to ensure the two workflows that work with the checklist are using the same indenting rule(s). So that it isn't always the entire file that's updated. Also this should probably be a step in the other workflow, so that the CRE IDs are updated whenever the checklist(s) are.

[Nik-ui]

Need to ensure the two workflows that work with the checklist are using the same indenting rule(s). So that it isn't always the entire file that's updated. Also this should probably be a step in the other workflow, so that the CRE IDs are updated whenever the checklist(s) are.

Thanks @kingthorin , that makes sense.

I’ll update the script to follow the same indentation rules to avoid rewriting the entire file, and look into integrating it as part of the existing workflow so CRE IDs stay in sync with checklist updates.

[kingthorin]

Re-duplicates see these issues and the associated PR on 640:

• Checklists and Merged IDs #640
• Known Issue: WSTG-INPV-13 is listed twice in checklist.json #1165

[kingthorin]

@Nik-ui It seems like you still have some changes that were already merged.

You might need to reset the branch to a clean up-to-date state and re-implement your changes. Then force push to the PR. 🤷‍♂️

[Nik-ui]

@Nik-ui It seems like you still have some changes that were already merged.

You might need to reset the branch to a clean up-to-date state and re-implement your changes. Then force push to the PR. 🤷‍♂️

Thanks @kingthorin , I have now rebased the branch onto the latest upstream master and removed previously merged or unrelated changes.

The PR now only contains the OpenCRE mapping script and workflow integration. I also verified that it does not introduce any direct checklist modifications; the existing WSTG-INPV-13 duplication is already present in upstream and is not added by this branch.

Please let me know if anything else should be adjusted.

[kingthorin]

Thanks, I'll review/test tomorrow.

[Nik-ui]

Thanks @kingthorin for the guidance, this was really helpful.

I have updated the implementation to use the /rest/v1/standard/... ?section= endpoint as suggested (OWASP/OpenCRE#866). The script now directly retrieves CRE mappings for each WSTG section from the structured standards response instead of relying on text search.

This aligns better with the intended API usage and avoids regex-style lookups.

I have tested the endpoint across multiple WSTG IDs and confirmed that it returns consistent CRE mappings.

Please let me know if this approach looks correct or if there is anything else you would like adjusted.

[kingthorin]

Thanks for collaborating with the OpenCRE team, that should help ensure this is more stable and reliable going forward.

Have you tested locally or on a VM? (It's okay if you haven't, I just want to set my own expectations 😉).

[Nik-ui]

Yes, I tested this locally.

I verified the mapping logic by calling the endpoint directly for multiple WSTG IDs (e.g. WSTG-INPV-04, WSTG-CONF-02) and confirmed that the expected CRE IDs are returned.

I also ran the full checklist generation script to ensure it integrates correctly and only updates the file when mappings change.

let me know if anything else should be adjusted.

[Nik-ui]

Hold off on further changes for now. I'm going to look at adjusting the current generation/handling. Then we can come back around to this.

Okay.

[kingthorin]

After a bunch of debate and testing I decided to move away from the shell handling to python since that was already in use for the Excel and Google Drive handling.

Do you want:

• To re-work this using python?
• Me to re-work this using python?
• You to close this and me to tackle it in a new branch/PR?

[Nik-ui]

After a bunch of debate and testing I decided to move away from the shell handling to python since that was already in use for the Excel and Google Drive handling.

Do you want:

• To re-work this using python?
• Me to re-work this using python?
• You to close this and me to tackle it in a new branch/PR?

Hi @kingthorin, I will definitely give this a try and get back to you once I am done.

[kingthorin]

Do you plan to clean this up and finish it?

[Nik-ui]

Do you plan to clean this up and finish it?

currently on it.

[kingthorin]

If you need help I can probably get it back on track and provide instructions to update your local branch. Just let me know.

[Nik-ui]

If you need help I can probably get it back on track and provide instructions to update your local branch. Just let me know.

Quick update: I have now fixed the markdown and link-check issues, and all checks are passing.

The PR is currently down to 3 commits and appears ready for review on my side. I’d still appreciate your guidance on the best next step for getting this fully aligned with the intended Python-based approach, if that’s still the preferred direction.

[kingthorin]

The files currently in the PR are completely irrelevant to the OpenCRE work. They seem to be something that you added during a merge or rebase.

The shell script was removed from the repo during the period in which you've been working with this PR. (So that was probably either from a merge or conflict.)

I'll go through the history tomorrow and try to find the python you had submitted at one point. Then I'll try to put some instructions together to get you back on track.

Edit: Oh maybe you never did do a python implementation, maybe the python I saw was just artifacts from you trying to update this branch.

[Nik-ui]

The files currently in the PR are completely irrelevant to the OpenCRE work. They seem to be something that you added during a merge or rebase.

The shell script was removed from the repo during the period in which you've been working with this PR. (So that was probably either from a merge or conflict.)

I'll go through the history tomorrow and try to find the python you had submitted at one point. Then I'll try to put some instructions together to get you back on track.

Edit: Oh maybe you never did do a python implementation, maybe the python I saw was just artifacts from you trying to update this branch.

Thanks for the clarification, that helped a lot.

I reviewed the commit history and traced this back to my earlier Python-based OpenCRE/checklist work. The earlier PR had picked up unrelated changes, so I restored the relevant Python script from that commit and removed the shell script and other unrelated changes to keep the PR focused.

Please let me know if this now aligns with what you expected.

[kingthorin]

[Nik-ui]

Thanks, I see what you were pointing out in that screenshot.

At that stage the PR was still showing the shell script, so it didn’t reflect the intended work. I will redeploy the Python version tomorrow and test, so it reflects it properly.

[kingthorin]

Superseded by #1404