Skip to content

[FLINK-xxxxx] Add GEOGRAPHY logical type support#19

Open
gkalashyan-akv wants to merge 74 commits into
devfrom
task/add-geography-type
Open

[FLINK-xxxxx] Add GEOGRAPHY logical type support#19
gkalashyan-akv wants to merge 74 commits into
devfrom
task/add-geography-type

Conversation

@gkalashyan-akv

Copy link
Copy Markdown
Collaborator

What is the purpose of the change

(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)

Brief change log

(for example:)

  • The TaskInfo is stored in the blob store on job creation time as a persistent artifact
  • Deployments RPC transmits only the blob storage reference
  • TaskManagers retrieve the TaskInfo from the blob cache

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (100MB)
  • Extended integration test for recovery after master (JobManager) failure
  • Added test that validates that TaskInfo is transferred only once across recoveries
  • Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

spuru9 and others added 30 commits July 7, 2026 15:09
…onfig)

Replace .eslintrc.js and .eslintignore with eslint.config.js flat
config. @angular-eslint 21 already supports eslint 9, so this adds the
unified angular-eslint and typescript-eslint packages (which provide
flat-config-ready presets and parser wiring) alongside the existing
split @angular-eslint/* packages needed by the Angular CLI builder and
schematics. eslint-plugin-prettier stays on v4.x since v5 requires
prettier 3, which is a separate deferred upgrade.

Generated-by: Claude Code (claude-opus-4-8)
… sort

---------

Co-authored-by: Arvind Kandpal <arvind.kandpal@ksolves.com>
…e#28489)

Close the JNI Statistics wrapper in `RocksDBNativeMetricMonitor.close()`
and on partial init in `RocksDBHandle.loadDb()`.
…ion README (apache#28677)

* [FLINK-40090][docs] Modernize cryptography export notice in distribution README

* [FLINK-40090][tests] Update expected WordCount row count in CLI e2e test

Generated-by: Claude Opus 4.8
…erminatedRescaleMergingWithNewRecoverableFailureTriggerCause

After terminating a TaskManager to trigger a recoverable failover, the test
waited via waitForVertexParallelismReachedAndJobRunning for PARALLELISM. That
condition is already satisfied by the stale pre-failover state, so the wait
returns immediately without synchronizing with the failover. The snapshot could
therefore be taken before AdaptiveScheduler merged the failover into the
in-progress UPDATE_REQUIREMENT rescale and re-stamped its trigger cause, letting
the test observe UPDATE_REQUIREMENT instead of the expected RECOVERABLE_FAILOVER.

Wait for the job to be RUNNING again at the parallelism the single remaining
TaskManager can host (NUMBER_SLOTS_PER_TASK_MANAGER) instead. The merge
re-stamps the trigger cause in goToRestarting, before the job runs again at that
reduced parallelism, so the job coming back up synchronizes the snapshot with
the merge. The assertion is unchanged.

Generated-by: Claude Opus 4.8 (via Claude Code)
…nputs finished

StreamMultipleInputProcessor.processInput returned NOTHING_AVAILABLE when no
input was selectable, even once all inputs were finished. Since
getAvailableFuture() returns AVAILABLE as soon as all inputs are finished, the
mailbox loop then never blocks and would spin on processInput at 100% CPU if it
were ever invoked again after the operator finished.

This is a latent inconsistency with single-input processors (which keep
returning END_OF_INPUT once finished) and is independent of any particular
feature; it is just not reached today because a task suspends on the first
END_OF_INPUT and processInput is not called again. The checkpointing-during-
recovery work (later in this series) does re-invoke processInput after an
operator finishes during recovery, which would otherwise turn this into a hang.

Report END_OF_INPUT when there is nothing to read and all inputs are finished,
consistent with single-input processors and
MultipleInputSelectionHandler#calculateOverallStatus.
…m toBeConsumedBuffers

Split the single toBeConsumedBuffers queue into two queues with disjoint
responsibilities:

  - recoveredBuffers (new): holds buffers migrated from RecoveredInputChannel
    during construction; consumed by getNextRecoveredBuffer() which retains
    the priority-event interleaving and last-buffer dynamic next-data-type
    detection introduced by FLINK-39018.
  - toBeConsumedBuffers (existing): reverted to its pre-FLINK-39018 role of
    holding FullyFilledBuffer partial-buffer splits only. The recovery-aware
    early branch in getNextBuffer() and the checkpointStarted inflight scan
    no longer touch this queue.

Restores the checkState(toBeConsumedBuffers.isEmpty()) guard in
requestSubpartitions() (removed by cebc174). hasPendingPriorityEvent,
notifyPriorityEvent, and the constructor signature are unchanged.

Pure refactor: no public API change, no new tests; verified by the 9 existing
LocalInputChannelTest regression cases.
…andler with concrete no-filtering/filtering handlers

Single-file class split of RecoveredChannelStateHandler.java; all classes stay
package-private in this file. No behavior change.

- New abstract AbstractInputChannelRecoveredStateHandler: fields inputGates,
  channelMapping, rescaledChannels, oldToNewMappings; methods getMappedChannels,
  calculateMapping, getChannel - verbatim extraction from
  InputChannelRecoveredStateHandler. Adds a closeInternal() template hook used
  by later phases.
- New NoSpillingHandler: recover() is the verbatim old non-filtering branch
  (onRecoveredStateBuffer(descriptor) followed by
  onRecoveredStateBuffer(buffer.retainBuffer()), same try/finally recycle);
  inherits the verbatim network-pool getBuffer.
- New FilteringHandler (the v1 filtering behavior, verbatim): recover() = old
  recoverWithFiltering (filteringHandler.filterAndRewrite(...,
  channel::requestBufferBlocking) delivering List<Buffer> via
  onRecoveredStateBuffer); getBuffer = old getPreFilterBuffer (reusable heap
  MemorySegment, preFilterBufferInUse invariant); closeInternal = old
  segment-freeing close.
- New static create(...) factory selecting NoSpillingHandler vs
  FilteringHandler - equivalent to the old internal if-branch.
  SequentialChannelStateReaderImpl switches from direct construction to the
  factory (mechanical).
- ResultSubpartitionRecoveredStateHandler: untouched.
- Trace label in RecoveredInputChannel.onRecoveredStateBuffer:
  "InputChannelRecoveredStateHandler#recover" -> "NoSpillingHandler#recover"
  (final label).
- Tests: mechanical adaptation of InputChannelRecoveredStateHandlerTest,
  RecoveredChannelStateHandlerTest.
- InputGate: new abstract InputChannel getChannel(InputChannelInfo).
- SingleInputGate: channels[channelInfo.getInputChannelIdx()]. UnionInputGate:
  resolve via inputGatesByGateIndex.get(channelInfo.getGateIdx())
  .getChannel(channelInfo) (correct global-vs-member index semantics).
  InputGateWithMetrics: delegate. CheckpointedInputGate: delegate helper.
- AbstractStreamTaskNetworkInput: getChannel(channelInfo.getInputChannelIdx())
  -> getChannel(channelInfo).
- Test impls: MockInputGate, MockIndexedInputGate,
  AlignedCheckpointsMassiveRandomTest's inner gate.
…ources visibility

- NetworkActionsLogger: additive tracePersist(String, Object, Object, long)
  overload; the existing Buffer overload delegates to it.
- RecoveredInputChannel#releaseAllResources: package-private -> public.
…beros keytab logs before the application is killed
)

* [FLINK-39191][runtime-web] Upgrade monaco-editor to 0.55.1

* Apache 2.0 license is chosen for dompurify
…se randomization

Drop the randomize(conf, CHECKPOINTING_DURING_RECOVERY_ENABLED, true, false)
line from TestStreamEnvironment.randomizeConfiguration (the flag stays at
its default false unless a test sets it explicitly) and @disabled the
explicitly flag-on RecoveredStateFilteringLargeRecordITCase.

Rationale: a following commit of this PR retires the v1
conversion-migration path, so ~50% of randomized ITCases would otherwise
run flag-on and fail until the StreamTask recovery rework and the
checkpoint-coordination-during-recovery PRs land. The randomization and
the ITCase are restored in the coordination PR (PR4 of the series).
… notifyInitiallyEnabled

Add a notifyInitiallyEnabled constructor parameter to BufferManager and a
new enableNotify() method: while notification is disabled, requested and
recycled floating buffers are queued but never announced to the producer
as credit; enableNotify() opens the gate and announces the queued buffers
atomically with respect to concurrent recycle/floating-buffer callbacks.

All existing callers (RecoveredInputChannel, RemoteInputChannel) pass
true, so behavior is unchanged. The disabled mode is used by a later
commit of this PR, where a remote channel created in recovery state
withholds credit until its recovered state has been consumed.
…ery sentinels

Introduce the push-based recovery contract and its sentinels:

- RecoverableInputChannel (@internal): getChannelInfo(),
  onRecoveredStateBuffer(Buffer), finishRecoveredBufferDelivery(),
  insertRecoveryCheckpointBarrierIfInRecovery(long),
  requestRecoveryBufferBlocking(), onRecoveredStateConsumed().
- EndOfFetchedChannelStateEvent: singleton RuntimeEvent tail sentinel of
  the recovered-buffer stream; reflective write()/read() throw;
  deliberately distinct from EndOfInputChannelStateEvent.
- RecoveryCheckpointBarrier (checkpoint.channel): per-checkpoint sentinel
  carried inside the recovery queue; write()/read() throw.
- EventSerializer: type tags 13 (RECOVERY_CHECKPOINT_BARRIER_EVENT) and
  14 (END_OF_FETCHED_CHANNEL_STATE_EVENT).

Recovery completion is unified on the InputChannel base class: the new
abstract getStateConsumedFuture() completes once a channel has consumed
all of its recovered state (or never had any / was released). It is
abstract rather than defaulted so a future subclass must state its
recovery semantics explicitly. RecoveredInputChannel's package-private
getter becomes a public override (field tightened to
CompletableFuture<Void>); UnknownInputChannel returns a completed future
(a channel with persisted state is always a RecoveredInputChannel at
restore time). Local/RemoteInputChannel temporarily return a completed
future until their push-based recovery state lands in the next commits.

TestInputChannel implements the new contract as a spying test double.
Teach LocalInputChannel to be created directly in a recovery state and to
receive recovered buffers through the RecoverableInputChannel interface,
instead of migrating them via the constructor:

- Constructor: drop ArrayDeque<Buffer> initialRecoveredBuffers, add
  networkBuffersPerChannel and needsRecovery. With needsRecovery=false
  the stateConsumedFuture is pre-completed, inRecovery stays false and no
  recovery BufferManager exists -- behavior is byte-for-byte unchanged.
- recoveredBuffers becomes a Deque<Buffer> whose monitor also guards
  inRecovery and recoverySequenceNumber (starts at Integer.MIN_VALUE).
- RecoverableInputChannel implementation: push append with released-state
  recycling; finishRecoveredBufferDelivery() waits for upstreamReady and
  appends the EndOfFetchedChannelStateEvent sentinel;
  insertRecoveryCheckpointBarrierIfInRecovery();
  requestRecoveryBufferBlocking() lends exclusive buffers via the
  recovery-only BufferManager (setup() requests them) -- no production
  caller until the drainer PR; onRecoveredStateConsumed() flips recovery
  off and completes the stateConsumedFuture.
- getNextBuffer() is rewritten under a single inRecovery predicate:
  recovered buffer first, then a pending priority event pulled from the
  subpartition view, otherwise empty (live data stays hidden while in
  recovery). wrapRecoveredBufferAsAvailability() materializes
  FileRegionBuffer/CompositeBuffer, assigns recovery sequence numbers and
  computes the next data type via peekNextDataType.
- upstreamReady completes on subpartition-view creation;
  completeUpstreamReadyForTest() for tests.
- checkpointStarted() splits into mutually exclusive in-recovery/normal
  branches. In recovery it only starts persisting (the persist window
  stays open) with the buffers collected by collectPreRecoveryBarrier,
  which walks the queue to the matching RecoveryCheckpointBarrier,
  retains pre-barrier data buffers and removes the sentinel; a missing
  barrier is an IOException wrapped as
  CheckpointException(CHECKPOINT_DECLINED).

Mechanical constructor-caller adaptations ride along (UnknownInputChannel,
LocalRecoveredInputChannel, InputChannelBuilder, benchmark factory), all
passing needsRecovery=false at this stage.

Includes a port of the FLINK-40016 double-persist regression test to the push-based recovery API.
Teach RemoteInputChannel to be created directly in a recovery state and to
receive recovered buffers through the RecoverableInputChannel interface.
The not-in-recovery checkpointStarted branch stays byte-identical to the
previous behavior, and with needsRecovery=false the channel behaves
exactly as before.

- Constructor: drop ArrayDeque<Buffer> initialRecoveredBuffers, add
  needsRecovery (pre-completed stateConsumedFuture when false, same as
  LocalInputChannel). The BufferManager starts with credit notification
  disabled while in recovery (notifyInitiallyEnabled=!needsRecovery).
- appendRecoveredBuffer() appends straight into receivedBuffers (NONE
  subpartition id, recovery sequence numbers from Integer.MIN_VALUE), so
  the consume path stays identical to the normal case.
- recoveryEventStash (@GuardedBy receivedBuffers): ordinary upstream
  events arriving while credit is suppressed are stashed and replayed
  after the sentinel is consumed; live data buffers during recovery are a
  protocol violation asserted in onBuffer.
- upstreamReady as CountDownLatch(1), counted down by the first onBuffer
  or by release; finishRecoveredBufferDelivery() awaits it and appends
  the EndOfFetchedChannelStateEvent sentinel.
- onRecoveredStateConsumed(): unstash + bufferManager.enableNotify()
  (credit reopens) + complete stateConsumedFuture.
- checkReadability() allows in-recovery reads before the partition
  request client is initialized.
- checkpointStarted() in-recovery branch persists the buffers collected
  by collectPreRecoveryBarrier (walks receivedBuffers past the priority
  head to the matching RecoveryCheckpointBarrier, retains pre-barrier
  data buffers, removes the sentinel); only startPersisting is called, so
  the persist window stays open. A missing barrier declines the
  checkpoint (CHECKPOINT_DECLINED).
- New public needsRecovery() getter, consumed by the netty stack in a
  later commit of this PR.

Mechanical constructor-caller adaptations ride along
(UnknownInputChannel, RemoteRecoveredInputChannel, InputChannelBuilder,
benchmark factory, netty test channels), all passing needsRecovery=false.
RemoteRecoveredInputChannel keeps calling remoteInputChannel.setup()
until the conversion rework in the next commit moves setup() into
RecoveredInputChannel#toInputChannel.
…ace; thread needsRecovery through gates

RecoveredInputChannel#toInputChannel(boolean needsRecovery) replaces the
no-arg variant; constructor migration of recovered buffers is retired:

- needsRecovery=false: checkState(receivedBuffers.isEmpty()) +
  inputChannel.setup() (setup moves from RemoteRecoveredInputChannel into
  the conversion) + checkpointStopped(...). Flag-off conversion always
  happens after full consumption, so default behavior is unchanged.
- needsRecovery=true (first used by the StreamTask recovery rework PR):
  create the physical channel in recovery state, then synchronously push
  every queued recovered data buffer via onRecoveredStateBuffer and
  append the EndOfFetchedChannelStateEvent sentinel (the legacy
  EndOfInputChannelStateEvent in the queue is dropped in translation).
  This is the simple in-memory drain -- FLINK-38544 transitional code,
  replaced by the disk drain when the spilling backend lands. The
  sentinel is appended directly rather than via
  finishRecoveredBufferDelivery(), which waits for upstream readiness
  that cannot arrive while the mailbox thread is still converting
  channels.
- abstract toInputChannelInternal(ArrayDeque<Buffer>) becomes
  toInputChannelInternal(boolean).

Gates thread the flag through: InputGate gains a concrete
requestPartitions(boolean) (default overload ignores the flag);
SingleInputGate implements requestPartitions(boolean) and
convertRecoveredInputChannels(boolean) (no-arg kept as a
@VisibleForTesting shim); getStateConsumedFuture() aggregation is
unified on InputChannel#getStateConsumedFuture across all channels;
UnionInputGate fans the flag out to member gates; InputGateWithMetrics
delegates.

Deliberate consequence: the base v1 flag-on StreamTask still converts at
filtering-complete with unconsumed buffers, so the empty-queue checkState
fires and the flag-on path is degraded until the StreamTask recovery
rework lands -- covered by the randomization pin in the first commit of
this PR. Default (flag-off) configuration is unaffected.

Tests: RecoveredInputChannelTest is rewritten around the new conversion
semantics (reject-while-unconsumed, empty-queue assert, and a transitional
push-conversion case); Local/RemoteRecoveredInputChannelTest cover the
empty-queue assert of the subclasses; UnionInputGateTest's helper adapts
to the new toInputChannelInternal signature.
…tart view reader with zero credit

Propagate the consumer channel's needsRecovery flag to the producer so it
withholds credit while the consumer's exclusive buffers are on loan to
recovery:

- NettyMessage.PartitionRequest gains a needsRecovery boolean (ctor arg,
  writeBoolean/readBoolean, +Byte.BYTES in the length calculation). This
  is a wire-format change (+1 byte per PartitionRequest); safe under the
  usual single-version-cluster assumption for the TM network protocol,
  stated in the PR description.
- CreditBasedSequenceNumberingViewReader ctor gains needsRecovery;
  numCreditsAvailable = needsRecovery ? 0 : initialCredit.
- PartitionRequestServerHandler passes request.needsRecovery;
  NettyPartitionRequestClient passes inputChannel.needsRecovery().

Mechanical test adaptations: NettyMessageServerSideSerializationTest
(round-trips the new field), PartitionRequestServerHandlerTest,
PartitionRequestQueueTest, PartitionRequestRegistrationTest,
CreditBasedSequenceNumberingViewReaderTest, CancelPartitionRequestTest,
ServerTransportErrorHandlingTest.
…nnelStateEvent

When the consume path polls the EndOfFetchedChannelStateEvent sentinel,
assert the originating channel is a RecoverableInputChannel and call its
onRecoveredStateConsumed(): the channel flips out of recovery, releases
any upstream events held back during recovery and reopens the upstream so
live data may flow again. The event itself is never delivered to the
operator.

The sentinel-consumption path is exercised end to end by the
Local/RemoteInputChannel recovery tests of this PR
(onRecoveredStateConsumed after polling the sentinel) and by the
StreamTask recovery tests of the follow-up PRs.
…is killed

FLINK-40099 moved YARNSessionFIFOSecuredITCase's keytab check into a pre-kill
callback polling waitUntilCondition(cond, 500). The 500 int literal binds to
the (cond, int retryAttempts) overload (500*100ms, then "Exhausted retry
attempts."), and reading the JobManager log while the application is still
alive is racy: the polled jobmanager.log does not reliably contain the keytab
login line before teardown.

Read the container logs after the application is killed (output flushed) but
poll each log with a bounded wait instead of a single read, preserving the
FLINK-17662 intent without the pre-kill race. On timeout, fail with the log
file and expected strings so a recurrence is diagnosable from the uploaded
container logs.

Generated-by: Claude Opus 4.8 (1M context)
alpinegizmo and others added 2 commits July 15, 2026 21:05
…n test snapshots (apache#28753)

Generated-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… host field (apache#28712)

FLINK-36355 removed the deprecated `host` field from the REST API's
subtask-level responses, keeping only `endpoint`, but the web-dashboard
frontend was never updated. The SubTask TimeLine and the SubTask
Accumulators table each read `task.host`, which is now always undefined,
so their subtask rows render as "N - undefined" instead of showing the
TaskManager endpoint.

Update the two frontend interfaces (JobSubTaskTime, SubTaskAccumulators)
and their consuming components to read `endpoint` instead of `host`.

Generated-by: Claude Code (claude-opus-4-8)
@gkalashyan-akv
gkalashyan-akv force-pushed the task/add-geography-type branch 2 times, most recently from 8bbb5c5 to 0505c8b Compare July 16, 2026 13:25
spuru9 and others added 23 commits July 16, 2026 17:00
…tier 3 and stylelint 17 (apache#28710)

* [FLINK-40034][runtime-web] Bump prettier to 3, eslint-plugin-prettier to 5, stylelint to 17

Generated-by: Claude Code (claude-opus-4-8)
…ously paused

Usually pauseOrResumeSplits pauses idleness timer for the split so it isn't marked idle while paused. However with low idleness timeout (observed with 1s) + low allowed WM drift, a race condition could cause paused splits to never resume though they have records:

1. A split becomes paused due to too advanced records.
2. pauseOrResumeSplits pauses the split.
3. pauseOrResumeSplits reaches to pause the split idleness clock but is {idlenessTimeout} too late, and the split becomes idle.
4. The watermark advances but the split is excluded from the watermark alignment check due to its idleness.
5. More records arrive but the split is paused at the connector level so they aren't processed, nor seen by watermarkGenerator so it still considers the split idle
The PR aims to fix it by preserving the part where idle splits are excluded from alignment pause (to not override their idle status) while allowing alignment check to resume splits even if they are currently idle.
They are considered idle until they emit the next record.

[FLINK-40093][Runtime] Refactor the test to reproduce the race condition
…sync future chain

Rewrite StreamTask channel-state recovery to run asynchronously on the
channelIOExecutor and unify recovery completion on the gates' state-consumed
futures. Read the during-recovery flag once via
CheckpointingOptions.isCheckpointingDuringRecoveryEnabled(jobConfig) and split
restoreStateAndGates into recoverChannelsWithCheckpointing /
recoverChannelsWithoutCheckpointing;
recoveryCompletionFuture.whenComplete(-> mailboxProcessor.suspend()).

Without checkpointing (flag off): submit readInputChannelState to the
channelIOExecutor; convert and request partitions for each gate as soon as ITS
OWN getStateConsumedFuture() completes (per-gate, avoiding the
selective-reading multi-input deadlock); return completeAll(all
stateConsumedFutures). Observable behavior equals master.

With checkpointing (flag on, in-memory backend, transitional until the
spilling backend lands): future chain runAsync(readInputChannelState,
channelIOExecutor) -> requestPartitions(true) for all gates on the mailbox
(conversion pushes the queued recovered buffers plus the sentinel) ->
completeAll(the gates' stateConsumedFutures). An empty-input-gates
short-circuit completes synchronously so a finite source cannot suspend the
restore loop before recovery completes.

restoreInternal now calls get() on the recovery future and rethrows the
underlying cause, so recovery failures surface instead of being swallowed; the
channelIOExecutor is shut down after the restore loop.

Also move the gate-finishing loop (finishReadRecoveredState() for all gates)
out of AbstractInputChannelRecoveredStateHandler.close() into
StreamTask#readInputChannelState; close() only runs closeInternal() now.
Adapt SequentialChannelStateReaderImplTest to call finishReadRecoveredState()
itself and put TaskCheckpointingBehaviourTest under @timeout(120).

Co-authored-by: Rui Fan <1996fanrui@gmail.com>
…te future from the gate API

Pure deletion of the now-unused v1 gating: the previous commit made StreamTask
gate the flag-on RUNNING transition on the recovery future chain instead of the
filtering-progress future, so the API has no callers left.

Drop InputGate#getBufferFilteringCompleteFuture and its overrides in
SingleInputGate, UnionInputGate and InputGateWithMetrics;
IndexedInputGate#setCheckpointingDuringRecoveryEnabled /
#isCheckpointingDuringRecoveryEnabled together with SingleInputGate's volatile
backing field; and RecoveredInputChannel's bufferFilteringCompleteFuture field,
getter and the finishReadRecoveredState lock-ordering commentary tied to it.

The one remaining consumer of the flag -- the unbounded heap-buffer fallback in
RecoveredInputChannel#requestBufferBlocking, which stays until disk spilling
supersedes it -- now has the flag threaded explicitly from the job
configuration through the recovery-handler call sites
(requestBufferBlocking(boolean)) instead of reading it off the gate.

Remove the corresponding testBufferFilteringCompleteFutureAggregation cases in
SingleInputGateTest/UnionInputGateTest and adapt MockInputGate,
MockIndexedInputGate, SingleInputGateBuilder and
AlignedCheckpointsMassiveRandomTest.

Co-authored-by: Roman Khachatryan <khachatryan.roman@gmail.com>
A task that reaches END_OF_INPUT during recovery -- e.g. a bounded/batch
operator whose input is already fully available behind a blocking exchange, as
seen on TPC-H/TPC-DS HashAggregate -- previously suspended the restore mailbox
loop immediately. With checkpointing during recovery the recovery future
completes asynchronously, so this could end the restore loop before recovery
finished, tripping checkState(allGatesRecoveredFuture.isDone()) in
restoreInternal with "Mailbox loop interrupted before recovery was finished".

Input is intentionally processed during recovery (so checkpoint barriers can
arrive from inputs); the bug is only the premature lifecycle finish. When
END_OF_INPUT is reached while recovery is still in progress, suspend only the
default action (to avoid busy-spinning) and resume it once recovery completes,
via a new recoveryCompletionFuture field set in restoreStateAndGates.
processInput then re-fires END_OF_INPUT from the main mailbox loop and finishes
normally. Sources remain covered by the no-input-gates short-circuit.

Validated with a deterministic repro (bounded BATCH job, recovery window
widened): the race reproduces with this deferral disabled and is gone with it
enabled. Full regression green: heavy-deployment e2e 6/6; ChainOrderTest,
KeyedComplexChainTest, KeyedPartitionWindowedStreamITCase; UnalignedCheckpoint
{,Rescale}ITCase (the with-recovered-state path) 11/11 and 50/50.
…h-down source

Wire the existing source.sleep-after-elements / source.sleep-time options into TestValuesScanTableSourceWithWatermarkPushDown

Generated-By: Claude Opus 4.8 (1M context)
…st framework

Lets restore tests take the stop-with-savepoint at a point defined by an
input-side signal rather than sink output, so operators that emit nothing at
the point of interest can be captured.

* RestoreTestBase: extract the trigger into an overridable awaitSavepointReady
  (default unchanged: waits for sinks to reach their before-restore rows) and
  retry stop-with-savepoint while the job is not yet fully running.
* TestValues watermark-push-down NewSource: add a per-table emission barrier
  (TestValuesTableFactory#awaitSourceEmitted) completed as rows are emitted.

Generated-By: Claude Opus 4.8 (1M context)
Adds end-to-end coverage for the LATERAL SNAPSHOT processing-time temporal join:

* LateralSnapshotJoinSemanticTests: Semantic tests LATERAL SNAPSHOT join
* LateralSnapshotJoinITCase: non-deterministic result tests and tests over HEAP and ROCKSDB backends
* LateralSnapshotJoinRestoreTest / LateralSnapshotJoinTestPrograms: savepoint restore tests
* Configure UTC as local timezone for CommonSemanticTestBase and RestoreTestBase

Generated-By: Claude Opus 4.8 (1M context)
…ode (apache#28763)

In batch, all input is bounded and append-only, so the processing-time
LATERAL SNAPSHOT join degenerates to a regular join of the probe side
against the (final) build side; the SNAPSHOT-specific arguments are
dropped. BatchPhysicalLateralSnapshotJoinRule converts the logical
snapshot join into a shuffle hash join that builds the (smaller) SNAPSHOT
side, mirroring StreamPhysicalLateralSnapshotJoinRule.

Generated-By: Claude Opus 4.8 (1M context)
…led (apache#28675)

* [FLINK-40079][table] Reject PTF calls with sys-args if they are disabled

* Add a check in SqlValidator to reject PTF calls with system-args (on_time, uid) in SQL querys if the function disabled them.
* Add a check in ResolveCallByArgumentsRule to reject system-args in functions that disabled them from Table API.

Generated-By: Claude Opus 4.8 (1M context)
…oMultisetCastRule` should check for null values in runtime

This closes apache#28777.
@davidchaava
davidchaava force-pushed the task/add-geography-type branch from 0505c8b to 82198d3 Compare July 21, 2026 12:10
@davidchaava
davidchaava force-pushed the task/add-geography-type branch from 82198d3 to 4ba21a0 Compare July 21, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.