lots of AssertionError: 'Attempting to create an event with no prev_events'

[richvdh]

These suggest something is wrong with the room in question. We should filter out attempts to send to such rooms earlier on.

[richvdh]

https://sentry.matrix.org/sentry/synapse-matrixorg/issues/119955

[richvdh]

This assertionerror is what happens where previously we would have returned 'no create event in auth events'; as such this is closely related to #7642.

These rooms have no entries in event_forward_extremities or current_state_events.

They seem to divide into two sets:

• rooms with an appservice user (typically an RSS bot) where the last regular user left some time ago. I suspect they have been cleared out by the the delete_old_current_state_events database schema update, though am a bit confused as to why this has happened given the appservice user is still a member.
• rooms where we have received one or more invite events over federation, but apparently never (successfully?) joined the room.

[richvdh]

• rooms with an appservice user (typically an RSS bot) where the last regular user left some time ago. I suspect they have been cleared out by the the delete_old_current_state_events database schema update, though am a bit confused as to why this has happened given the appservice user is still a member.

It might be worth noting that, at the time we ran this update on matrix.org, it looked at current_state_events.membership to see which rooms were still active (that was later changed in #7854). But I don't know why that wouldn't be populated either, at least on matrix.org (other servers might have had a horrible mess, due to #6923).

[clokep]

New sentry issue: https://sentry.matrix.org/sentry/synapse-matrixorg/issues/189680/

[richvdh]

Still very much a thing: https://sentry.matrix.org/sentry/synapse-matrixorg/issues/200257

[anoadragon453]

This was noticed again when @neb_alertmanager:matrix.org was trying to join a room on 2021-12-01 14:08:56 with ID containing TZrnFfySWI after being invited by @jaywink.

Relevant stack trace:

Stack trace

2021-12-01 14:08:56,850 - synapse.http.server - 101 - ERROR - PUT-2311876 - Failed handle request via 'RoomSendEventRestServlet': <XForwardedForRequest at 0x7fe7a53f3bd8 method='PUT' uri='/_matrix/client/r0/rooms/%21redacted:federator.dev/send/m.room.message/go1638367736821254338?access_token=<redacted>' clientproto='HTTP/1.1' site='8138'>
Capture point (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/synapse/src/synapse/app/generic_worker.py", line 508, in <module>
    start(sys.argv[1:])
  File "/home/synapse/src/synapse/app/generic_worker.py", line 503, in start
    _base.start_worker_reactor("synapse-generic-worker", config)
  File "/home/synapse/src/synapse/app/_base.py", line 125, in start_worker_reactor
    run_command=run_command,
  File "/home/synapse/src/synapse/app/_base.py", line 177, in start_reactor
    run()
  File "/home/synapse/src/synapse/app/_base.py", line 161, in run
    run_command()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1318, in run
    self.mainLoop()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 1328, in mainLoop
    reactorBaseSelf.runUntilCurrent()
  File "/home/synapse/src/synapse/metrics/__init__.py", line 645, in f
    ret = func(*args, **kwargs)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/base.py", line 967, in runUntilCurrent
    f(*a, **kw)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 662, in callback
    self._startRunCallbacks(result)
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 859, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1751, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
Traceback (most recent call last):
  File "/home/synapse/env-py37/lib/python3.7/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/home/synapse/src/synapse/rest/client/room.py", line 272, in on_POST
    requester, event_dict, txn_id=txn_id
  File "/home/synapse/src/synapse/handlers/message.py", line 858, in create_and_send_nonmember_event
    depth=depth,
  File "/home/synapse/src/synapse/handlers/message.py", line 615, in create_event
    depth=depth,
  File "/home/synapse/src/synapse/util/metrics.py", line 106, in measured_func
    r = await func(self, *args, **kwargs)
  File "/home/synapse/src/synapse/handlers/message.py", line 959, in create_new_client_event
    ), "Attempting to create an event with no prev_events"
AssertionError: Attempting to create an event with no prev_events
2021-12-01 14:08:56,872 - synapse.access.http.8138 - 421 - INFO - PUT-2311876 - x.x.x.x - 8138 - {@neb_alertmanager:matrix.org} Processed request: 0.029sec/0.007sec (0.015sec, 0.000sec) (0.001sec/0.006sec/3) 55B 500 "PUT /_matrix/client/r0/rooms/%21redacted:federator.dev/send/m.room.message/go1638367736821254338?access_token=<redacted> HTTP/1.1" "Go-http-client/2.0" [0 dbevts]

[jaywink]

Re above, this disappeared now, not sure if it was me 1) joining my matrix.org user to the room, 2) writing some events to the room (it was previously empty, just state events) or 3) re-inviting the bot to the room - but it's now happy.

[ginkel]

I am seeing this issue on my self-hosted Synapse instance in one e2e-encrypted direct-message room with another user ("u2"). u2 dropped out of the Element Android client after I made a configuration error on the Identity server. This config error has been fixed meanwhile.

Still, when I attempt to send a message to u2 the following exception happens:

2021-12-12 08:39:57,112 - synapse.http.server - 97 - ERROR - PUT-606- Failed handle request via 'RoomSendEventRestServlet': <XForwardedForRequest at 0x7fb4fcc730d0 method='PUT' uri='/_matrix/client/r0/rooms/!rgZTHfiXTwUdnFmRPp:tgbyte.de/send/m.room.encrypted/$local.2e913437-5cb7-46c1-8574-214eec80b52b' clientproto='HTTP/1.1' site='8008'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 271, in on_POST
    ) = await self.event_creation_handler.create_and_send_nonmember_event(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/message.py", line 850, in create_and_send_nonmember_event
    event, context = await self.create_event(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/message.py", line 610, in create_event
    event, context = await self.create_new_client_event(
  File "/usr/local/lib/python3.8/site-packages/synapse/util/metrics.py", line 106, in measured_func
    r = await func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/message.py", line 957, in create_new_client_event
    assert (
AssertionError: Attempting to create an event with no prev_events

When u2 attempts to join the room (after restoring keys) synapse logs:

2021-12-12 08:36:02,567 - synapse.http.server - 97 - ERROR - POST-481- Failed handle request via 'RoomCreateRestServlet': <XForwardedForRequest at 0x7fb4f9c49ee0 method='POST' uri='/_matrix/client/r0/createRoom' clientproto='HTTP/1.1' site='8008'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 262, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 451, in _async_render
    callback_return = await raw_callback_return
  File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 89, in on_POST
    info, _ = await self._room_creation_handler.create_room(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/room.py", line 817, in create_room
    last_stream_id = await self._send_events_for_new_room(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/room.py", line 982, in _send_events_for_new_room
    await self.room_member_handler.update_membership(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/room_member.py", line 486, in update_membership
    result = await self.update_membership_locked(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/room_member.py", line 886, in update_membership_locked
    return await self._local_membership_update(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/room_member.py", line 328, in _local_membership_update
    event, context = await self.event_creation_handler.create_event(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/message.py", line 610, in create_event
    event, context = await self.create_new_client_event(
  File "/usr/local/lib/python3.8/site-packages/synapse/util/metrics.py", line 106, in measured_func
    r = await func(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/message.py", line 957, in create_new_client_event
    assert (
AssertionError: Attempting to create an event with no prev_events

I have nightly backups of the Synapse state that I could restore. I also do have an Element Desktop client of u2 that has not been force-logged-out and of whose state I created a backup.

My homeserver is running Synapse v1.48.0.

Any ideas?

Edit: I see that #11243, which deals with this assertion, got merged yesterday. Any idea whether this will land in v1.49?

[DMRobertson]

I wonder if the errors from @ginkel here are related to #11568?

Edit: I see that #11243, which deals with this assertion, got merged yesterday. Any idea whether this will land in v1.49?

Afraid not. That's scheduled for 1.50, to be released in the new year.

[reivilibre]

For searchability: the new phrase for this log line is 'Attempting to create a non-m.room.create event with no prev_events'.

https://sentry.matrix.org/sentry/synapse-matrixorg/issues/242065/?query=is%3Aunresolved appears to be another recent example, again with neb/RSS bot.