Skip to content

Stop cascading logging errors in the Windows desktop app - #15197

Open
rtibbles wants to merge 2 commits into
learningequality:developfrom
rtibbles:fix-desktop-app-logging-loop
Open

Stop cascading logging errors in the Windows desktop app#15197
rtibbles wants to merge 2 commits into
learningequality:developfrom
rtibbles:fix-desktop-app-logging-loop

Conversation

@rtibbles

Copy link
Copy Markdown
Member

Summary

  1. CPU and memory growth was being caused by errors when two processes tried to log to the same file on Windows
  2. These errors in logging then triggered another log attempt to report the error - so it cascaded
  3. A similar issue happened when trying to rotate log files that were contended
  4. This would grow without bound because the logging queue was unbounded

Fixed by:

  1. Using a different log file for the server specific process on Windows - this avoids the log file contention
  2. Because the app captures stdout/stderr, errors during logging were re-logged by the stream handler there - stops those reports being emitted at all, and stops reentrant writes being captured
  3. Catches OSErrors during log file rotation and does a back off on the rotation
  4. Adds a max size to the logging queue so that there's a limit to how many queued logs there can be

References

Fixes #15150

Reviewer guidance

  1. Install Windows installer
  2. Confirm kolibri-app-server.txt in C:\ProgramData\kolibri\logs
  3. Launching the UI adds kolibri-app.txt alongside it
  4. Confirm that CPU and memory do not increase when idle

Noting that the thing that seemed to set off the error in @pcenov's case was an error from a scheduled task - so running a background task might be sufficient to trigger this.

I have not tested on Windows.

AI usage

Used Claude Code to write the fix and its tests, then to review the resulting diff — that pass caught the Windows smoke test still looking for the renamed log file, and two guards that only closed the in-thread half of the loop. Verified with the kolibri/utils and desktop-app test suites, prek, and a local review pass over the diff.

@github-actions github-actions Bot added DEV: backend Python, databases, networking, filesystem... SIZE: medium labels Aug 14, 2026
@rtibbles
rtibbles requested a review from rtibblesbot August 14, 2026 20:24
@rtibblesbot

rtibblesbot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🟡 Waiting for changes

Last updated: 2026-09-09 16:09 UTC

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

@rtibblesbot rtibblesbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #15197 — the four-part diagnosis (contended file → handler error → re-logged through the captured stderr → unbounded queue) holds up against the code, and each fix lands where the cycle closes.

One blocking issue: bounding the queue makes QueueListener.stop() raise queue.Full in exactly the saturated state this PR expects, breaking server shutdown. Rest are suggestions/nitpicks inline.

Verified locally: pytest kolibri/utils/tests (526 passed), pytest platforms/desktop-app/tests/test_streams.py (5 passed), prek run --from-ref upstream/develop --to-ref HEAD clean. CI: no failures, one check still running. No UI files changed, so visual verification does not apply.


@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly

How was this generated?

Ran a phased review pipeline over the pull request diff:

  • Classified the diff to select review passes (core, frontend, backend) and whether manual QA was required
  • Core review pass checked correctness, design, architecture, testing, completeness, and DRY/SRP/Rule-of-Three principles
  • Specialized frontend/backend review passes applied framework-specific lenses where those files changed
  • For UI changes: manual QA and an accessibility audit against a live dev server, when available
  • Checked CI status and linked issue acceptance criteria
  • Synthesized one review from those passes and chose the verdict from the findings, CI status, and QA evidence

Comment thread kolibri/utils/logger.py
Comment thread kolibri/utils/logger.py Outdated
Comment thread kolibri/utils/logger.py
record._logger_name = self.logger_name
return record

def enqueue(self, record: logging.LogRecord) -> None:

@rtibblesbot rtibblesbot Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved — addressed in the current code.

suggestion: Dropped records leave no trace. Together with log.raiseExceptions = False in the desktop app, a log with a hole in it is indistinguishable from a quiet one — when the next report of this bug arrives, nothing in kolibri.txt will say records were lost.

The listener thread is outside the reentrant path, so a counter incremented here and reported from there (on drain, or at cleanup_queue_logging) would not reopen the cycle.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declined both counter suggestions because there's no obvious place to log this information that doesn't get us into the same reentrant logging loop.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a reasonable suggestion that doesn't seem impossible. Instead of immediately logging this, it could simply track the quantity of dropped logs, and at some point in the future when the queue is freed, enqueue a message that N logs were dropped.

Comment thread kolibri/utils/logger.py
Comment thread platforms/desktop-app/src/kolibri_app/logger.py
Comment thread kolibri/utils/logger.py
Comment thread kolibri/utils/tests/test_handler.py Outdated
Comment thread platforms/desktop-app/tests/test_streams.py
@rtibbles
rtibbles force-pushed the fix-desktop-app-logging-loop branch from dfb4c61 to 76d7c0f Compare August 19, 2026 03:46

@rtibblesbot rtibblesbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #15197 — 5 of 8 prior findings resolved; 3 open, none new. CI passing.

  • suggestion logger.py:66 — silent drops.
  • suggestion logger.py:187 — rollover failure undiscoverable.
  • nitpick logger.py:212 — no DST correction.
Prior-finding status

RESOLVED — kolibri/utils/logger.py:470 — stop() raises Full
RESOLVED — kolibri/utils/logger.py:17 — KOLIBRI_LOG_QUEUE_MAX_SIZE=0
RESOLVED — platforms/desktop-app/src/kolibri_app/logger.py:20 — raiseExceptions global
RESOLVED — kolibri/utils/tests/test_handler.py:120 — kwarg assertion
UNADDRESSED — kolibri/utils/logger.py:66 — silent drops
UNADDRESSED — kolibri/utils/logger.py:187 — rollover undiscoverable
UNADDRESSED — kolibri/utils/logger.py:212 — DST correction
ACKNOWLEDGED — platforms/desktop-app/tests/test_streams.py:50 — bounded writer


@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly

How was this generated?

Ran an automatic code-only delta review triggered by new commits on a previously reviewed PR:

  • Retrieved prior bot reviews via the GitHub API
  • Classified each prior finding as RESOLVED, UNADDRESSED, ACKNOWLEDGED, or CONTESTED
  • Only raised NEW findings for newly introduced code
  • Core review pass only — specialized frontend/backend lenses and manual QA run when a review is explicitly requested
  • Synthesized one review from the passes and chose the verdict from the findings, CI status, and QA evidence

@rtibbles
rtibbles marked this pull request as ready for review August 19, 2026 20:53

@bjester bjester left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The suggestion about having some visibility into dropped logs does seem reasonable, but it should certainly take a different approach than immediately logging that. With AI being used here, that does not seem like a tall ask.

Comment thread kolibri/utils/logger.py
record._logger_name = self.logger_name
return record

def enqueue(self, record: logging.LogRecord) -> None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a reasonable suggestion that doesn't seem impossible. Instead of immediately logging this, it could simply track the quantity of dropped logs, and at some point in the future when the queue is freed, enqueue a message that N logs were dropped.

rtibbles and others added 2 commits September 9, 2026 08:55
Windows will not rename a log file a second process still holds open. The
failed rollover left rolloverAt in the past, so every subsequent record
retried the rename, and each retry's own error report fed back in as more
records. Postpone the next attempt instead, and bound the log queue so a
listener that cannot drain stops growing it, counting what that costs and
reporting the total once the queue has room again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KXc8oMzGTawizmmAuxeJfg
The UI process and the server subprocess shared kolibri-app.txt, so at
midnight one of them could never rotate it. Give the server its own file,
and stop logging's handler-error reports from re-entering logging through
the stdout/stderr redirect the app installs, both when they recurse
in-thread and when the queue carries them across threads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KXc8oMzGTawizmmAuxeJfg
@rtibbles
rtibbles force-pushed the fix-desktop-app-logging-loop branch from 76d7c0f to 7e01fa4 Compare September 9, 2026 15:57

@rtibblesbot rtibblesbot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #15197 — 6 of 8 prior findings resolved; 2 acknowledged (declined by the author), none still open. No new findings. CI still pending at the time of review.

Prior-finding status

RESOLVED — kolibri/utils/logger.py:496 — listener.stop() raises queue.Full on a saturated queue
RESOLVED — kolibri/utils/logger.py:67 — dropped records leave no trace
RESOLVED — kolibri/utils/logger.py — KOLIBRI_LOG_QUEUE_MAX_SIZE=0 silently restores the unbounded queue
RESOLVED — platforms/desktop-app/src/kolibri_app/logger.py:20 — logging.raiseExceptions is module state
RESOLVED — kolibri/utils/tests/test_handler.py — bounded-queue test asserted only the maxsize= kwarg
RESOLVED — platforms/desktop-app/tests/test_streams.py:50 — praise, bounded recursive writer
ACKNOWLEDGED — kolibri/utils/logger.py:213 — the except OSError guard is wider and narrower than the failure it documents
ACKNOWLEDGED — kolibri/utils/logger.py:238 — _postpone_rollover omits the DST correction


@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly

How was this generated?

Ran an automatic code-only delta review triggered by new commits on a previously reviewed PR:

  • Retrieved prior bot reviews via the GitHub API
  • Classified each prior finding as RESOLVED, UNADDRESSED, ACKNOWLEDGED, or CONTESTED
  • Only raised NEW findings for newly introduced code
  • Core review pass only — specialized frontend/backend lenses and manual QA run when a review is explicitly requested
  • Synthesized one review from the passes and chose the verdict from the findings, CI status, and QA evidence

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DEV: backend Python, databases, networking, filesystem... SIZE: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows app - High RAM and CPU usage

3 participants