Feature/graylog server

CHANGELOG.md

@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+* **plugin:graylog_input, plugin:graylog_index_set**: New in-house modules that drive the Graylog REST API directly. They support `--check` (dry-run preview) and `--diff` (per-field before/after), report `changed=true` only when a field actually differs, and replace the previous `ansible.builtin.uri` chains in the `graylog_server` role.
 * **plugin:platform_select**: New filter plugin for selecting a value from a platform-keyed dictionary by OS family / distribution / version.
 * **role:alternatives**: Support managing `subcommands` (slaves/followers) and the Red Hat-only `family` grouping. The role now also ensures the alternatives tooling is installed (`chkconfig` on RHEL 8, `alternatives` on RHEL 9/10; bundled with `dpkg` on Debian/Ubuntu), and can be included without variables as a no-op.
 * **role:redis**: Add template for version 8.8
@@ -31,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Breaking Changes
 
+* **role:graylog_server**: System inputs and index sets now follow the standard `__group_var` / `__host_var` pattern with `state: present/absent` per entry, and the on-demand tag `graylog_server:configure_defaults` has been split into `graylog_server:configure_system_inputs` and `graylog_server:configure_system_index_sets`. The role no longer ships a default index set. Operators must define at least one index set in the inventory and mark one as `default: true`. The four default system inputs (Beats, GELF TCP, GELF UDP, Syslog UDP) are preserved in `graylog_server__system_inputs__role_var`. Migration: rename `graylog_server__system_default_index_set` (dict) to a list entry under `graylog_server__system_index_sets__host_var` (or `__group_var`), add `default: true` and `state: 'present'`. Rename `graylog_server__system_inputs` (list) to `graylog_server__system_inputs__host_var` (or `__group_var`); add `state: 'present'` to each entry.
 * **role:minio_client, role:objectstore_backup**: Both roles and their playbooks (`playbooks/minio_client.yml`, `playbooks/objectstore_backup.yml`) have been removed, along with the corresponding role blocks in `playbooks/setup_nextcloud.yml` and the `setup_nextcloud__skip_minio_client` / `setup_nextcloud__skip_objectstore_backup` variables. MinIO Server has been archived as no-longer-maintained since February 2026, and we are moving away from using object storage for critical data. Users relying on these roles must replace the MinIO-based object-store backup with their own solution (e.g. `rclone`); the `mc` binary, its config under `/etc/mc/`, the `objectstore-backup` systemd timer/service, and `/usr/local/bin/mc-mirror.sh` are no longer managed by lfops and will remain on existing hosts until removed manually ([#241](https://github.com/Linuxfabrik/lfops/issues/241)).
 * **role:infomaniak_vm**: Always create a managed port for every entry in `infomaniak_vm__networks`, even when no `fixed_ip` is set. Previously only networks with a `fixed_ip` got a managed port; networks without one relied on OpenStack's auto-created port. To avoid creating unused (but billed) managed ports on VMs provisioned under the old behavior, make sure to manually rename the existing port in OpenStack to match the `port_name`. Note that this port will not survive VM deletion / detachment, since it was automatically created and therefore is owned by OpenStack, not the user.
 

plugins/module_utils/graylog.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8; py-indent-offset: 4 -*-
+#
+# Author:  Linuxfabrik GmbH, Zurich, Switzerland
+# Contact: info (at) linuxfabrik (dot) ch
+#          https://www.linuxfabrik.ch/
+# License: The Unlicense, see LICENSE file.
+
+"""Shared client and diff helpers for the linuxfabrik.lfops.graylog_* modules.
+
+Wraps `ansible.module_utils.urls.fetch_url` with the headers and auth shape
+Graylog expects (basic auth, `X-Requested-By`, JSON bodies). All HTTP errors
+surface as `GraylogAPIError` so the modules can fail uniformly.
+"""
+
+from __future__ import absolute_import, division, print_function
+
+__metaclass__ = type
+
+import base64
+import json
+
+from ansible.module_utils.urls import fetch_url
+
+
+DEFAULT_HEADERS = {
+    'Accept': 'application/json',
+    'X-Requested-By': 'ansible-lfops',
+}
+
+
+class GraylogAPIError(Exception):
+    """Raised on any non-2xx response from the Graylog API."""
+
+    def __init__(self, status, url, body):
+        self.status = status
+        self.url = url
+        self.body = body
+        super().__init__(f'HTTP {status} from {url}: {body}')
+
+
+class GraylogClient:
+    """Thin REST client around `fetch_url`. One instance per module call."""
+
+    def __init__(self, module, url, username, password, validate_certs=True, timeout=30):
+        self._module = module
+        self._base = url.rstrip('/')
+        self._validate_certs = validate_certs
+        self._timeout = timeout
+        token = base64.b64encode(f'{username}:{password}'.encode('utf-8')).decode('ascii')
+        self._auth_header = f'Basic {token}'
+
+    # --- low-level ------------------------------------------------------------
+
+    def _request(self, method, path, body=None, expected_status=None):
+        url = self._base + path
+        headers = dict(DEFAULT_HEADERS)
+        headers['Authorization'] = self._auth_header
+        data = None
+        if body is not None:
+            headers['Content-Type'] = 'application/json'
+            data = json.dumps(body).encode('utf-8')
+
+        resp, info = fetch_url(
+            self._module,
+            url,
+            data=data,
+            headers=headers,
+            method=method,
+            timeout=self._timeout,
+        )
+        status = info.get('status', -1)
+
+        if expected_status is not None:
+            allowed = expected_status if isinstance(expected_status, (list, tuple, set)) else (expected_status,)
+            if status not in allowed:
+                raise GraylogAPIError(status, url, info.get('body') or info.get('msg') or '')
+        elif status < 200 or status >= 300:
+            raise GraylogAPIError(status, url, info.get('body') or info.get('msg') or '')
+
+        if resp is None:
+            return None
+        raw = resp.read()
+        if not raw:
+            return None
+        try:
+            return json.loads(raw.decode('utf-8'))
+        except (ValueError, AttributeError) as exc:
+            raise GraylogAPIError(status, url, f'invalid JSON: {exc}')
+
+    # --- verbs ----------------------------------------------------------------
+
+    def get(self, path, expected_status=200):
+        return self._request('GET', path, expected_status=expected_status)
+
+    def post(self, path, body, expected_status=(200, 201)):
+        return self._request('POST', path, body=body, expected_status=expected_status)
+
+    def put(self, path, body=None, expected_status=(200, 201, 204)):
+        return self._request('PUT', path, body=body, expected_status=expected_status)
+
+    def delete(self, path, expected_status=(200, 204)):
+        return self._request('DELETE', path, expected_status=expected_status)
+
+
+# --- diff helpers ------------------------------------------------------------
+
+
+def strip_keys(entry, keys):
+    """Return a shallow copy of `entry` with the given role-only `keys` removed.
+
+    Replaces the role's `dict2items | rejectattr | items2dict` chain.
+    """
+    if not entry:
+        return {}
+    return {k: v for k, v in entry.items() if k not in keys}
+
+
+def _equal(a, b):
+    """Recursive equality. Treats dicts and lists structurally; falls back to
+    `==` for scalars. Used by `diff_changed_fields` so nested config dicts
+    (e.g. Graylog input `configuration`, index-set `data_tiering`) compare
+    correctly.
+    """
+    if isinstance(a, dict) and isinstance(b, dict):
+        if set(a.keys()) != set(b.keys()):
+            return False
+        return all(_equal(a[k], b[k]) for k in a)
+    if isinstance(a, list) and isinstance(b, list):
+        if len(a) != len(b):
+            return False
+        return all(_equal(x, y) for x, y in zip(a, b))
+    return a == b
+
+
+def diff_changed_fields(current, desired, ignore=None):
+    """Return `(before, after)` containing only the keys whose values differ.
+
+    Keys present in `desired` are checked against `current`; keys not in
+    `desired` are not compared (we never want to "unset" a field the user
+    did not mention). Keys listed in `ignore` are skipped entirely. Suitable
+    for an Ansible `result["diff"]` payload that highlights just the user-
+    visible delta on update.
+    """
+    if ignore is None:
+        ignore = ()
+    before = {}
+    after = {}
+    current = current or {}
+    for key, desired_value in (desired or {}).items():
+        if key in ignore:
+            continue
+        current_value = current.get(key)
+        if not _equal(current_value, desired_value):
+            before[key] = current_value
+            after[key] = desired_value
+    return before, after