Lighteval's LCB grader fails correct solutions that read sys.stdin.buffer

[RasmusHoier]

Describe the bug

Lighteval performs the swap @patch("sys.stdin", StringIO(inputs)) here link, which causes code that relies on sys.stdin.buffer.read() to fail.

Livecodebench avoids this issue by swapping in a mock object with a buffer attributes here link.

To Reproduce

This code reproduces the error and suggests a fix to the call_method.

from unittest.mock import mock_open, patch
from io import StringIO
import io
from typing import Callable
from lighteval.tasks.tasks.lcb.codegen_metrics import call_method

def call_method_fixed(method: Callable, inputs: list | str):
    if isinstance(inputs, list):
        inputs = "\n".join(inputs)

    inputs_line_iterator = iter(inputs.split("\n"))
    mock_stdin = StringIO(inputs)   # <-- new stuff here
    mock_stdin.buffer = io.BytesIO(inputs.encode())  # <-- new stuff here

    @patch("builtins.open", mock_open(read_data=inputs))
    @patch("sys.stdin", mock_stdin)   # <-- the fix
    @patch("sys.stdin.readline", lambda *args: next(inputs_line_iterator))
    @patch("sys.stdin.readlines", lambda *args: inputs.split("\n"))
    @patch("sys.stdin.read", lambda *args: inputs)
    def _inner_call_method(_method):
        try:
            return _method()
        except SystemExit:
            pass
        finally:
            pass

    return _inner_call_method(method)
    

SOLUTION_1   = "nums = input() and input().split()\nprint(sum(map(int, nums)))"
SOLUTION_2  = "import sys\nprint(sum(map(int, sys.stdin.buffer.read().split()[1:])))"

print("using call_method_fixed on solution 1: (should print 6)")
call_method_fixed(lambda: exec(SOLUTION_1), "3\n1 2 3")
print("using call_method_fixed on solution 2: (should print 6)")
call_method_fixed(lambda: exec(SOLUTION_2), "3\n1 2 3")
print("using original call_method on solution 1: (should print 6)")
call_method(lambda: exec(SOLUTION_1), "3\n1 2 3")
print("using original call_method on solution 2: (should fail with AttributeError: '_io.StringIO' object has no attribute 'buffer'")
call_method(lambda: exec(SOLUTION_2), "3\n1 2 3")

Expected behavior

The use of sys.stdin.buffer.read() should not cause an error.

Version info

I am using lighteval v 0.13 and I installed it via uv.