Skip to content

Add OIDC authentication customizer for GitLab#1052

Closed
jhoward-lm wants to merge 186 commits into
DependencyTrack:mainfrom
jhoward-lm:gitlab-integration
Closed

Add OIDC authentication customizer for GitLab#1052
jhoward-lm wants to merge 186 commits into
DependencyTrack:mainfrom
jhoward-lm:gitlab-integration

Conversation

@jhoward-lm
Copy link
Copy Markdown
Contributor

@jhoward-lm jhoward-lm commented Feb 4, 2025

Description

This PR adds an implementation of the OidcAuthenticationCustomizer service provider interface specific to GitLab. It is meant to synchronize a user's projects and roles/max access levels per project within GitLab to a Hyades instance.

Depends on stevespringett/Alpine#755 and #1069.

Note

WIP: a rough to-do list with outstanding items is in a comment block in GitLabSyncTask.java

Addressed Issue

Closes DependencyTrack/hyades#1632

Additional Details

This PR is a request for early review while still in development in order to change direction if necessary.

Checklist

  • I have read and understand the contributing guidelines
  • This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
  • This PR introduces changes to the database model, and I have updated the migration changelog accordingly
  • This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

@jhoward-lm
Copy link
Copy Markdown
Contributor Author

Tagging for awareness:
@ashearin @lmphil @pkwiatkowski1 @Strakeln @jmayer-lm @EphraimEM

@jhoward-lm
Copy link
Copy Markdown
Contributor Author

@nscuro Would you mind doing a preliminary review and letting me know if this implementation/direction needs adjustment? Thanks!

Copy link
Copy Markdown
Member

@nscuro nscuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far. I think concurrency control of the task will become interesting.

The workflow orchestration functionality we're currently working on will provide the ability to serialize based on a concurrency group, similar to what GitHub Actions has. So this undertaking will become easier in the future, but until then we need another solution.


import static org.dependencytrack.model.ConfigPropertyConstants.GITLAB_ENABLED;

public class GitLabSyncTask implements LoggableSubscriber {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume we need a way to serialize execution of these tasks? If two or more users with overlapping GitLab project accesses login in close succession, we will encounter race conditions.

Before we go into the how (there are multiple options), can you think of any property/properties of the OIDC profile that can act as event key? i.e. what would you use as concurrency group?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing in the OIDC profile comes to mind. The only thing I can think of is maybe the groups claim, if they can be locked individually, but that doesn't give us full GitLab project paths.

The name of each GitLab project as it is read from the GitLab API response can be used as well, but that isn't available in the profile

Comment thread pom.xml Outdated
@jhoward-lm
Copy link
Copy Markdown
Contributor Author

@nscuro We noticed that changes in the alpine schema aren't yet reflected in hyades or the apiserver, so we created a changeset that seems to work. The changeset is now pushed to this PR. Do we need to open a PR in hyades to update schema.sql or anything?

/cc @pkwiatkowski1

@nscuro
Copy link
Copy Markdown
Member

nscuro commented Feb 5, 2025

The changeset is now pushed to this PR. Do we need to open a PR in hyades to update schema.sql or anything?

Leave it for now. We will add that migration in a separate PR anyway, since it involves a bit more non-SQL migration logic. You can keep the migration in your PR so you can test, but will likely need to rebase later down the road.

@jhoward-lm
Copy link
Copy Markdown
Contributor Author

@nscuro Just wanted to see if you had any ideas on a couple of bugs we're seeing. It looks like the event is somehow firing off twice, and getting a weird error about classes not being found in the CLASSPATH. For example:

2025-02-06 16:06:00,793 INFO [GitLabSyncTask] Starting GitLab sync task
2025-02-06 16:06:00,801 INFO [GitLabSyncTask] Starting GitLab sync task
2025-02-06 16:06:00,822 INFO [GitLabSyncer] Creating project example-project
2025-02-06 16:06:00,823 INFO [GitLabSyncer] Creating project example-project
2025-02-06 16:06:00,832 ERROR [Schema] An exception was thrown while adding/validating class(es) :
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.

I'll continue debugging, just seeing if anything stands out as immediately obvious to you. Thanks!

@nscuro
Copy link
Copy Markdown
Member

nscuro commented Feb 6, 2025

@jhoward-lm For the redundant invokation, see my comment here: stevespringett/Alpine#755 (comment)

2025-02-06 16:06:00,832 ERROR [Schema] An exception was thrown while adding/validating class(es) :
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.

Admittedly not sure, I'd have to checkout the code and see if I can reproduce it.

@jhoward-lm
Copy link
Copy Markdown
Contributor Author

@nscuro Here is the full stack trace if it helps

Full error output

2025-02-06 12:15:25,284 ERROR [Schema] An exception was thrown while adding/validating class(es) : Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
org.datanucleus.exceptions.ClassNotResolvedException: Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
        at org.datanucleus.ClassLoaderResolverImpl.classForName(ClassLoaderResolverImpl.java:219)
        at org.datanucleus.ClassLoaderResolverImpl.classForName(ClassLoaderResolverImpl.java:347)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getMappingType(MappingManagerImpl.java:291)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getDefaultJavaTypeMapping(MappingManagerImpl.java:1348)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getMappingClass(MappingManagerImpl.java:829)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getMapping(MappingManagerImpl.java:631)
        at org.datanucleus.store.rdbms.table.ClassTable.manageMembers(ClassTable.java:648)
        at org.datanucleus.store.rdbms.table.ClassTable.manageClass(ClassTable.java:554)
        at org.datanucleus.store.rdbms.table.ClassTable.initializeForClass(ClassTable.java:1380)
        at org.datanucleus.store.rdbms.table.ClassTable.initialize(ClassTable.java:273)
        at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.initializeClassTables(RDBMSStoreManager.java:3460)
        at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:3083)
        at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:118)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1666)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:670)
        at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:430)
        at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:911)
        at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:377)
        at org.datanucleus.store.query.Query.executeQuery(Query.java:1965)
        at org.datanucleus.store.query.Query.executeWithMap(Query.java:1911)
        at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:437)
        at org.datanucleus.api.jdo.JDOQuery.executeResultUnique(JDOQuery.java:393)
        at org.dependencytrack.persistence.ProjectQueryManager.doesProjectExist(ProjectQueryManager.java:1301)
        at org.dependencytrack.persistence.QueryManager.doesProjectExist(QueryManager.java:607)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.createProjectStructure(GitLabSyncer.java:174)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.synchronize(GitLabSyncer.java:112)
        at org.dependencytrack.tasks.GitLabSyncTask.inform(GitLabSyncTask.java:78)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)

2025-02-06 12:15:25,284 WARN [Query] Query for candidates of org.dependencytrack.model.Project and subclasses resulted in no possible candidates : Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
Exception in thread "ForkJoinPool-1-worker-1" javax.jdo.JDOUserException: Don't support result clause of count(this) with resultClass of java.lang.Long
        at org.datanucleus.api.jdo.JDOAdapter.getJDOExceptionForNucleusException(JDOAdapter.java:701)
        at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:456)
        at org.datanucleus.api.jdo.JDOQuery.executeResultUnique(JDOQuery.java:393)
        at org.dependencytrack.persistence.ProjectQueryManager.doesProjectExist(ProjectQueryManager.java:1301)
        at org.dependencytrack.persistence.QueryManager.doesProjectExist(QueryManager.java:607)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.createProjectStructure(GitLabSyncer.java:174)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.synchronize(GitLabSyncer.java:112)
        at org.dependencytrack.tasks.GitLabSyncTask.inform(GitLabSyncTask.java:78)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)
NestedThrowablesStackTrace:
Don't support result clause of count(this) with resultClass of java.lang.Long
org.datanucleus.exceptions.NucleusUserException: Don't support result clause of count(this) with resultClass of java.lang.Long
        at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:442)
        at org.datanucleus.store.query.Query.executeQuery(Query.java:1965)
        at org.datanucleus.store.query.Query.executeWithMap(Query.java:1911)
        at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:437)
        at org.datanucleus.api.jdo.JDOQuery.executeResultUnique(JDOQuery.java:393)
        at org.dependencytrack.persistence.ProjectQueryManager.doesProjectExist(ProjectQueryManager.java:1301)
        at org.dependencytrack.persistence.QueryManager.doesProjectExist(QueryManager.java:607)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.createProjectStructure(GitLabSyncer.java:174)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.synchronize(GitLabSyncer.java:112)
        at org.dependencytrack.tasks.GitLabSyncTask.inform(GitLabSyncTask.java:78)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)

@jhoward-lm
Copy link
Copy Markdown
Contributor Author

@nscuro In src/main/resources/META-INF/persistence.xml, it looks like org.dependencytrack.model.Classifier is missing. Does it need to be added manually, or is this file generated from somewhere?

@nscuro
Copy link
Copy Markdown
Member

nscuro commented Feb 6, 2025

Adding Classifier there should not be necessary, because Classifier is only ever stored as simple String.

persistence.xml only needs to list classes that are "persistence capable" (i.e. represent a table, for a lack of better terms).

So either there's a bug in the ORM, or perhaps you have some changes locally that are not included in this PR?

for (String group : toCreate) {
LOGGER.debug("Creating project " + group);

Project existingProject = qm.getProject(group, null);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't find any place where the qm being used here is set. The code should currently throw a NPE. Is there some code you haven't yet pushed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I have some local code that sets the qm and some other fixes. We think the use of UnblockedEvent might have something to do with the Classifier not being found. I'll push what I have shortly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the change here acceptable?

@jhoward-lm jhoward-lm force-pushed the gitlab-integration branch 3 times, most recently from fa505bb to 6cad00c Compare February 13, 2025 19:56
@jhoward-lm jhoward-lm force-pushed the gitlab-integration branch 2 times, most recently from 29f5a81 to cc785c8 Compare February 24, 2025 18:26
@ashearin ashearin mentioned this pull request Feb 24, 2025
5 tasks
dependabot Bot and others added 12 commits March 3, 2025 17:53
Bumps debian from `b5ace51` to `5724d31`.

---
updated-dependencies:
- dependency-name: debian
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Some legacy code ported over from v4 still uses `DbUtil#isPostgreSQL` checks to determine if certain SQL syntax can be used.

Since our move to Liquibase, `DbUtil` has not been initialized anymore, and hence always returned `false` for the aforementioned check.

Ultimately, usages of `DbUtil` should be removed entirely.

Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Bumps `lib.net.javacrumbs.shedlock.version` from 6.2.0 to 6.3.0.

Updates `net.javacrumbs.shedlock:shedlock-provider-jdbc` from 6.2.0 to 6.3.0

Updates `net.javacrumbs.shedlock:shedlock-provider-jdbc-internal` from 6.2.0 to 6.3.0

---
updated-dependencies:
- dependency-name: net.javacrumbs.shedlock:shedlock-provider-jdbc
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: net.javacrumbs.shedlock:shedlock-provider-jdbc-internal
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Analogue to DependencyTrack/hyades#1672

Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
A mix-up of `"V"."VULNID" != ANY(:vulnIdsToExclude)` and `"V"."VULNID" != ALL(:vulnIdsToExclude)` caused all but one Snyk vulnerability to be suppressed for a component.

https://stackoverflow.com/a/11730845
Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
…1064)

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Bumps `lib.testcontainers.version` from 1.20.4 to 1.20.5.

Updates `org.testcontainers:kafka` from 1.20.4 to 1.20.5
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/main/CHANGELOG.md)
- [Commits](testcontainers/testcontainers-java@1.20.4...1.20.5)

Updates `org.testcontainers:postgresql` from 1.20.4 to 1.20.5
- [Release notes](https://github.com/testcontainers/testcontainers-java/releases)
- [Changelog](https://github.com/testcontainers/testcontainers-java/blob/main/CHANGELOG.md)
- [Commits](testcontainers/testcontainers-java@1.20.4...1.20.5)

---
updated-dependencies:
- dependency-name: org.testcontainers:kafka
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: org.testcontainers:postgresql
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Bumps [org.apache.maven.plugins:maven-clean-plugin](https://github.com/apache/maven-clean-plugin) from 3.4.0 to 3.4.1.
- [Release notes](https://github.com/apache/maven-clean-plugin/releases)
- [Commits](apache/maven-clean-plugin@maven-clean-plugin-3.4.0...maven-clean-plugin-3.4.1)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-clean-plugin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
lmphil and others added 26 commits May 7, 2025 16:38
* roleQueryManagerTest

Signed-off-by: Philippe <philippe.a.aviles@lmco.com>

* update role query manager test

Signed-off-by: Philippe <philippe.a.aviles@lmco.com>

* fix RoleQueryManagerTest and the USER_PROJECT_EFFECTIVE_PERMISSIONS view

Signed-off-by: Philippe <philippe.a.aviles@lmco.com>

* add remaining RoleQueryManager tests

Signed-off-by: Philippe <philippe.a.aviles@lmco.com>

* add TODO to failing test

Signed-off-by: Philippe <philippe.a.aviles@lmco.com>

* fix formatting

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>

* fix RoleQueryManagerTest.testGetRole

Signed-off-by: Philippe <philippe.a.aviles@lmco.com>

---------

Signed-off-by: Philippe <philippe.a.aviles@lmco.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Co-authored-by: Allen Shearin <allen.p.shearin@gmail.com>
…erver into pull-upstream-main

Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
…te-user-tables

Roles consolidate user tables
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
…apiserver into gitlab-integration

Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Co-authored-by: jhoward-lm <140011346+jhoward-lm@users.noreply.github.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
…nto add-roles-model

Signed-off-by: Allen Shearin <allen.p.shearin@gmail.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
…apiserver into gitlab-integration

Signed-off-by: Jonathan Howard <jonathan.w.howard@lmco.com>
@jhoward-lm jhoward-lm mentioned this pull request Jul 16, 2025
5 tasks
@jhoward-lm
Copy link
Copy Markdown
Contributor Author

Closing as superseded by #1325

@jhoward-lm jhoward-lm closed this Jul 16, 2025
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Aug 16, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Project-level RBAC

10 participants