Skip to content

[KYUUBI #7407] STGroup free to avoid OOM Kill #7408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oh0873 wants to merge 5 commits into from
+14 −4
Closed

[KYUUBI #7407] STGroup free to avoid OOM Kill #7408

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
28cadbc
STGroup free to avoid OOM Kill
oh0873 Apr 16, 2026
321c430
Update kyuubi-common/src/main/scala/org/apache/kyuubi/service/authent…
oh0873 Apr 20, 2026
dd0f910
Explicit group definition
oh0873 Apr 20, 2026
18ee9bb
Added () to createFilter and render
oh0873 Apr 21, 2026
72740e0
Update kyuubi-common/src/main/scala/org/apache/kyuubi/service/authent…
oh0873 Apr 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions kyuubi-common/src/main/scala/org/apache/kyuubi/service/authentication/ldap/Query.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,14 @@ package org.apache.kyuubi.service.authentication.ldap
import java.util
import javax.naming.directory.SearchControls

import org.stringtemplate.v4.ST
import org.stringtemplate.v4.{ST, STGroup}

/**
* The object that encompasses all components of a Directory Service search query.
*
* The caller must and can only call each [[filter]] and [[build]] once,
* otherwise [[ST]] internal cache may leak and cause heap OOM.
*
* @see [[LdapSearch]]
*/
object Query {
Expand All @@ -40,6 +43,9 @@ object Query {
* A builder of the [[Query]].
*/
final class QueryBuilder {

/** The [[STGroup]] used only for this builder's filter template; unloaded in [[build]]. */
private var filterTemplateGroup: Option[STGroup] = None
private var filterTemplate: ST = _
private val controls: SearchControls = {
val _controls = new SearchControls
Expand All @@ -56,7 +62,9 @@ object Query {
* @return the current instance of the builder
*/
def filter(filterTemplate: String): Query.QueryBuilder = {
this.filterTemplate = new ST(filterTemplate)
val group = new STGroup()
this.filterTemplateGroup = Some(group)
this.filterTemplate = new ST(group, filterTemplate)
Copy link
Copy Markdown
Member

@pan3793 pan3793 Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does cache only happen in filterTemplate.render?

what will happen if we call def filter multi times?

Copy link
Copy Markdown
Contributor Author

@oh0873 oh0873 Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cache is created in def filter, but it won't be needed after createFilter that's why I added unload right after createFilter. I'm not sure if this is the best place to unload.

At least in our use case it seems to unload most if not all STToken used by LDAP auth process.

Copy link
Copy Markdown
Member

@pan3793 pan3793 Apr 21, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cache is created in def filter

so cache still leaks if we call def filter multi times?

Copy link
Copy Markdown
Contributor Author

@oh0873 oh0873 Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. IF def filter is called but build never called those STToken will stay in heap forever.

Copy link
Copy Markdown
Member

@pan3793 pan3793 Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean

builder
  .filter("exp1")
  .filter("exp2") // this replaces exp1
  .build()

will this case cause a leak?

Copy link
Copy Markdown
Contributor Author

@oh0873 oh0873 Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think this would cause a leak.

Copy link
Copy Markdown
Member

@pan3793 pan3793 Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, as this is only used internally, we don't have such anti-patterns yet, let's somehow document those dangerous behaviors

BTW, this part of the code is ported from Apache Hive, it should have the same issue

https://github.com/apache/hive/blob/branch-4.2/service/src/java/org/apache/hive/service/auth/ldap/Query.java#L90

this
}

Expand Down Expand Up @@ -112,7 +120,7 @@ object Query {
require(filterTemplate != null, "filter is required for LDAP search query")
}

private def createFilter: String = filterTemplate.render
private def createFilter(): String = filterTemplate.render()

private def updateControls(): Unit = {
if (!returningAttributes.isEmpty) controls.setReturningAttributes(
Expand All @@ -126,7 +134,9 @@ object Query {
*/
def build: Query = {
validate()
val filter: String = createFilter
val filter: String = createFilter()
// Unload template cache after render to avoid CompiledST/STToken retention
filterTemplateGroup.foreach(_.unload())
updateControls()
new Query(filter, controls)
}
Expand Down
Loading