Skip to content

Bootstrap semantic PR #21 rebuild - #22

Merged
Komzpa merged 45 commits into
masterfrom
ai/pr21-semantic-rebuild
Jul 26, 2026
Merged

Bootstrap semantic PR #21 rebuild#22
Komzpa merged 45 commits into
masterfrom
ai/pr21-semantic-rebuild

Conversation

@Komzpa

@Komzpa Komzpa commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

This draft rebuilds the useful parts of #21 as a clean semantic stack on top of Komzpa/twms master. It intentionally keeps TWMS tiny and general-purpose: no broad service-specific defaults, no large framework rewrite, no license/author deletion, and no Java-factory-shaped architecture.

Current commits:

  • 88cfbb9 bootstraps packaging and GitHub Actions CI.
  • 1320671 adds legacy package smoke tests and compatibility shims.
  • 47455de clears Python 3 syntax warnings without changing behavior.
  • 996ce0b adds a no-network legacy GetTile smoke test.
  • 60bd619 adds a tiny stdlib ThreadingHTTPServer entry point while keeping web.py/WSGI available as twms-webpy.
  • f121d79 exposes TileJSON-style tileset metadata.
  • 1ddb2bb exposes WMTS 1.0.0 capabilities plus REST and KVP tile access, with Radioxoma authorship preserved.
  • 3989521 modernizes the filesystem tile cache with TTL, .tne, stale fallback, atomic replacement, and Radioxoma-style dead_tile dictionaries.
  • c7c3b49 modernizes projection fallback with pole-edge clamping and optional pyproj.Transformer.
  • 317dc88 adds tiny WMS 1.3.0 and CRS:84 support for QGIS-style clients without replacing WMS 1.1.1.
  • 4aa6b65 rewrites README client/deployment docs for stdlib, web.py, QGIS, JOSM, WMTS, TileJSON, GetTile, shared caches, and optional dependencies.
  • daed6f6 adds a Windows executable artifact job for twms.exe and twms-webpy.exe.
  • 35c847e preserves filters, corrections, drawing, canvas, and reprojection surfaces with current Python/Pillow fixes.
  • c24f53d validates downloaded image bytes before caching and converts mismatched tile responses to the configured layer extension.
  • 237dac5 records HTTP 404 and configured dead_tile.http_status values as .tne cache misses.
  • 2aa87ce accepts readable named tile URL placeholders while preserving legacy %s templates.
  • 2980eaa generates JOSM maps-1.0 imagery XML.
  • 041580c fixes atomic WMS cache writes by passing the intended image format explicitly.
  • 61df8a7 supports optional per-layer HTTP headers for tile and WMS upstreams.
  • a1339fa enforces per-layer min_zoom while preserving existing exclusive max_zoom.
  • 7852d8a adds opt-in cache_layout: "zxy" while keeping TWMS' historical grouped cache layout by default.
  • 98c39bd accepts readable named WMS URL placeholders while preserving legacy base-URL query appending.
  • 794940b bounds upstream tile/WMS fetch waits with global/per-layer timeout config.
  • faadb4e hardens stdlib REST routing and WMTS REST URLs with query strings.
  • 2fe1b21 applies bounded upstream waits to the preserved legacy WmsCanvas.
  • ad5ead4 restores legacy empty_color overlay transparency and empty_color_delta.
  • 637ef8b keeps historical cache lookups and cached GetTile bytes binary-safe.
  • eb261e0 keeps legacy cache_tile_responses binary-safe.
  • a4e0d9e adds optional transient upstream retries without retrying HTTP/TNE statuses.
  • 618a447 makes the RAM tile cache least-recently-used and fixes its bound.
  • 0dc7b06 accepts MIME format keys such as image/png in cache_tile_responses.
  • d99e15a documents the cookie decision: per-layer headers yes, automatic browser cookie discovery no.
  • a85b8b4 keeps generated-artifact ignore hygiene without broad unrelated IDE/tooling ignores.
  • 952f1ae accepts readable layer bounds metadata aliases.
  • f168719 enriches JOSM imagery XML with bounds, overlay/default/georeference hints, attribution URLs, zoom limits, and no-tile checksums.
  • b29d80d aligns the overview page with layer bounds aliases and provider URLs.
  • edabf43 ships optional Windows batch and Linux desktop-entry launcher templates as packaged contrib files.
  • b8b3d18 cancels stale CI runs for the same workflow event and branch.
  • a86e856 accepts layer mimetype metadata as a readable alias/source for tile extensions while preserving legacy ext configs.
  • f2522cc supports image/webp as a normal WMS output format while reusing the existing Pillow format path.
  • d552b8a supports optional layer_defaults in Python configs through a tiny dict wrapper rather than importing the fork config rewrite.
  • d0b1ed1 adds tiny stdlib route aliases for /wms tile/query URLs and /josm/maps.xml while keeping routing delegated to twms_main.
  • 2cd346c prints useful stdlib-server startup URLs and includes the TWMS version in the HTTP Server header without importing the fork logging rewrite.
  • fbf33d1 accepts readable string fetcher aliases such as "tms" and "wms" while resolving them back to the existing legacy fetcher functions.
  • 7e4d2ed makes the import smoke check shell-portable so the same GitHub Actions workflow passes under Bash on Linux and PowerShell on Windows.

Branch behavior

The branch currently:

  • adds GitHub Actions CI for Ubuntu and Windows on Python 3.11 and 3.13;
  • adds a Windows executable artifact job for stdlib and preserved web.py entry points;
  • modernizes packaging with pyproject.toml;
  • preserves TWMS' public keyboard-style version as 0.07z, while exposing PEP 440 metadata as 0.7+z;
  • replaces the removed Python 3.13 imp loader with importlib;
  • keeps old package-style imports working without a large relative-import rewrite;
  • keeps WMS 1.1.1, adds WMS 1.3.0/CRS:84, WMTS, TileJSON, JOSM imagery XML, overview metadata, and WebP WMS output;
  • keeps GetTile because it is still useful for passing filters through tile URLs, including direct /wms/<layer>/<z>/<x>/<y> route aliases;
  • keeps TWMS' built-in projection shortlist tiny, with optional pyproj for configured non-core projections;
  • keeps historical cache layout by default, with optional z/x/y shared-cache layout;
  • supports cache TTL, stale fallback, .tne, dead_tile, binary-safe response caches, layer headers, min zoom, timeouts, retries, and named upstream URL placeholders;
  • accepts layer mimetype metadata as an alias/source for tile extensions while preserving existing ext configs;
  • supports optional layer_defaults for shared layer metadata without merging defaults into every layer;
  • accepts readable fetch aliases "tms" / "tile" and "wms" while preserving callable legacy fetchers;
  • keeps /josm/imagery.xml, /josm/maps.xml, and /maps.xml as generated imagery-list aliases;
  • prints stdlib-server startup URLs for overview, WMS, WMTS, and JOSM clients, and reports a versioned HTTP Server header;
  • ships optional launcher templates for old Windows/JOSM proxy and Linux desktop/manual downstream deployments without installing desktop integration by default;
  • preserves legacy filter, correction, WKT/GPX drawing, canvas, reprojection, tools, manpage, WSGI, and web.py deployment surfaces;
  • credits Eugene Dvoretsky (Radioxoma) in COPYING and README for modernization work;
  • keeps upstream Public Domain/ISC license text instead of taking the fork's license rewrite.

Validation

Clean-venv validation passed through current head 7e4d2ed2bf7ed793d040335f0a2af672a7150154:

  • editable install in a clean venv
  • python -m compileall -q setup.py index.py tools twms tests
  • WSGI application import
  • assertions that twms.__version__ == "0.07z" and installed package metadata is 0.7+z
  • python -m unittest discover -s tests -v (75 tests, 1 optional pyproj skip)
  • python -m build
  • source distribution and wheel contents include contrib/twms.bat and contrib/twms.desktop
  • git diff --check

The optional projection extra was also checked separately:

  • pip install -e '.[proj]'
  • python -m unittest tests.test_legacy_smoke.LegacySmokeTest.test_optional_pyproj_projection_uses_modern_transformer -v

The smoke suite covers WMS 1.1.1/1.3.0, CRS:84, GetTile, stdlib serving, stdlib startup banner, versioned HTTP Server header, TileJSON, WMTS capabilities/KVP/REST, JOSM imagery XML, overview HTML, cache TTL/TNE/stale/dead-tile behavior, binary cache reads/writes, layer headers, min/max zoom, zxy cache layout, named URL placeholders, upstream timeout/retry behavior, projection fallback, optional pyproj, legacy rendering helpers, layer bounds aliases, layer mimetype aliases, layer defaults, string fetcher aliases, WebP WMS output, stdlib WMS/JOSM route aliases, and Windows/package artifact contents.

GitHub Actions passed on current head 7e4d2ed2bf7ed793d040335f0a2af672a7150154:

Both runs passed Ubuntu and Windows package smoke jobs for Python 3.11 and 3.13, plus the Windows executable artifact job that builds and uploads twms.exe and twms-webpy.exe.

Selection notes

This branch deliberately rebuilds #21 as semantic pieces instead of merging the fork wholesale.

Taken, but reshaped to fit upstream TWMS:

  • modern packaging, tests, GitHub Actions, and Windows executable artifacts;
  • stdlib serving while preserving the existing web.py/WSGI deployment path;
  • WMS 1.3.0, WMTS, TileJSON, JOSM imagery XML, and QGIS/JOSM documentation;
  • cache TTL/TNE/stale-cache behavior, image validation, named URL templates, headers, min zoom, optional retries, and opt-in shared slippy-cache layout;
  • projection modernization with a tiny built-in shortlist plus optional pyproj for non-core projections;
  • readable metadata aliases such as bounds and mimetype;
  • WebP output support as a tiny MIME/format-map extension;
  • optional layer defaults as a small config-loader feature;
  • readable string fetcher aliases as a config-loader adapter to the existing fetchers;
  • tiny route aliases from the fork without importing the fork api.py split;
  • stdlib startup URL hints and a versioned Server header without importing the fork logging/color-output rewrite;
  • optional launcher templates as packaged contrib files.

Not taken from #21:

  • the license rewrite and author/copyright removals;
  • deletion of legacy filters, corrections, drawing, canvas, reprojection, tools, manpage, WSGI, or web.py deployment surfaces;
  • the service-specific default provider config, DZZ/Firefox cookie auto-discovery, and browser-like global headers;
  • the broad OOP fetcher/cache/config rewrite, including the thread-pool code that was described as async but still blocks on submit(...).result();
  • provider-specific dynamic Google tile URL discovery from JavaScript;
  • colored logging, debug trace output, and global logging rewrites;
  • JOSM remote-control links embedded in the overview HTML; generated maps-1.0 imagery XML and documented TMS/file-cache setup cover JOSM without making the overview page active;
  • the broad black/pre-commit/mypy/docstring formatting pass;
  • the public version jump to semver-style 2.x naming.

Scope

Ready for review. The reciprocal Radioxoma branch update has been posted as radioxoma#1; that fork PR is expected to be a replace/rebase PR rather than a clean merge because the fork branch history diverged from upstream.

Komzpa and others added 30 commits July 26, 2026 13:18
Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep TWMS' historical filesystem cache layout, but teach it the useful cache behavior from Radioxoma's fork: cache_ttl checks, reusable .tne markers, stale fallback after network errors, and atomic replacement for freshly downloaded tiles.

Also accept Radioxoma-style dead_tile dictionaries while preserving the legacy dead_tile file comparison.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep TWMS' built-in projection shortlist tiny, but make the core WebMercator path tolerate pole-edge bounds and route optional pyproj transforms through the modern Transformer API.

This deliberately preserves the existing optional twms[proj] behavior for non-core projections instead of taking Radioxoma's hard pyproj removal.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest Radioxoma's WMS 1.3.0 capability work without taking the fork's API/config rewrite or legacy module removals. Keep the existing WMS 1.1.1 generator intact, add a tiny ElementTree WMS 1.3.0 response, accept case-insensitive request keys, and support CRS:84 as the lon/lat WMS 1.3.0 request alias without advertising it as a WMS 1.1.1 SRS.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest the useful deployment and client guidance from Radioxoma's README without taking the fork framing, unsupported-feature removals, or repo layout rewrite. Document the current upstream entry points, QGIS WMS/WMTS URLs, JOSM TMS/GetTile/file-cache forms, cache behavior, and optional dependencies while keeping TWMS described as a general tiny WMS/tile service.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the Windows/JOSM single-executable deployment path without adding runtime framework code to TWMS. Build both the stdlib server entry point and the preserved web.py entry point with PyInstaller in GitHub Actions, upload the executables as CI artifacts, and run the workflow for branches, tags, releases, PRs, and manual dispatch.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Do not take Radioxoma's hard removal of filters, corrections, canvas, and reprojection helpers. Instead keep the legacy surfaces covered and fix current Python/Pillow breakage: use a Pillow resampling compatibility helper, keep canvas usable on Python 3, keep GetCorrections returning text/plain, pass WKT colors correctly when no color parameter is provided, and use Python 3 iterator semantics for GPX colors.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Validate downloaded WMS/TMS image bytes before saving them to the filesystem cache, keep stale cached tiles on invalid downloads, and convert mismatched tile responses to the configured layer extension when possible.

This keeps the useful cache-safety behavior from Radioxoma's fork without adopting the larger fetcher rewrite.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Record HTTP 404 tile responses as .tne cache misses and allow layer configs to opt in additional HTTP miss statuses with dead_tile.http_status.

This keeps the useful HTTP-status TNE behavior from Radioxoma's fork while avoiding service-specific heuristics and noisy response-body logging.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Accept readable tile URL templates with {z}, {x}, {y}, {-y}, and {q} while preserving the legacy percent-template path and transform_tile_number behavior.

This takes the useful URL placeholder idea from Radioxoma's fork without migrating existing configs or adopting the larger fetcher rewrite.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Generate a JOSM maps-1.0 imagery XML document from configured TWMS layers and expose it through both request=GetJOSMImagery and stdlib routes at /josm/imagery.xml and /maps.xml.

This keeps the useful JOSM imagery-list idea from Radioxoma's fork without taking the heavy route/config reshuffle or launching JOSM remote-control side effects.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Pass the target image format explicitly when saving through a temporary cache path, so Pillow does not infer the format from the .tmp.<pid> suffix.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Pass optional per-layer HTTP headers to tile and WMS upstream downloads while leaving layers without headers on the legacy string-url path.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Honor per-layer min_zoom in both tile and WMS fetchers while preserving the existing exclusive max_zoom behavior.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Add an opt-in cache_layout for slippy/MOBAC-style <z>/<x>/<y> cache paths while keeping TWMS' historical grouped cache layout as the default.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Allow upstream WMS remote_url templates to supply {bbox}, {width}, {height}, and {proj} while preserving the legacy base-URL append behavior when no WMS placeholders are present.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the stdlib threaded server tiny, but avoid waiting forever on a stalled upstream tile or WMS source. Layer configs may override the timeout when an old deployment deliberately needs different behavior.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep malformed REST-style protocol paths from falling through to the overview or disconnecting handler threads. Also fix the legacy GetTile fast-cache layer lookup exposed by WMTS REST tile requests with query strings.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the preserved WmsCanvas helper from waiting forever on remote WMS tiles while retaining an explicit timeout=None escape hatch for deployments that relied on the historical unbounded wait.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the legacy overlay/filter surface by making configured empty_color values transparent again, including per-channel empty_color_delta tolerance. This repairs the behavior instead of deleting the old rendering helpers.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Use integer cache shard directories under Python 3 and read cached GetTile responses as bytes, so the historical grouped cache layout remains usable alongside the new opt-in zxy layout.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the legacy cache_tile_responses fast path under Python 3 by reading and writing cached image responses as bytes instead of text.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest Radioxoma's HTTP retry idea without taking the larger backend rewrite: keep urllib, default to one attempt for legacy behavior, and let global or per-layer config opt in retries for transient transport errors.

HTTP errors such as 404 still flow to the existing TNE/cache handling instead of being retried as generic network failures.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the legacy max_ram_cached_tiles cache, but make eviction least-recently-used instead of FIFO/off-by-one history popping.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Komzpa and others added 9 commits July 26, 2026 15:53
Keep legacy cache_tile_responses configs working when they use MIME strings such as image/png, as shown in the packaged example, while still accepting the normalized Pillow format keys used internally.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Document the decision not to import Radioxoma's Firefox cookie-store discovery into upstream TWMS. Private deployments can still provide cookies through per-layer headers or their own Python config code without adding a browser-profile dependency to the tiny server.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the useful hygiene from Radioxoma's added ignore file, but reduce it to artifacts TWMS actually produces during tests, builds, and PyInstaller packaging.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Accept Radioxoma's readable bounds layer metadata key as an alias for historical data_bounding_box/bbox in tile filtering and generated TileJSON/WMTS metadata without replacing the legacy config contract.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest Radioxoma's useful JOSM imagery metadata idea without importing the larger API/config rewrite or launching JOSM remote-control side effects. The generated maps-1.0 XML now carries layer bounds, overlays, attribution URLs, valid-georeference/default hints, and fixed per-MD5 no-tile checksums while preserving TWMS' existing URL and zoom semantics.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the legacy overview page but harvest Radioxoma's useful layer metadata behavior: accept the readable bounds alias for preview/bbox links and link configured provider URLs without reshaping the overview UI.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the small launcher deployment idea from Radioxoma's branch without installing desktop integration by default. Package Windows batch and Linux desktop-entry templates under share/twms/contrib and document them as optional manual deployment helpers.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the CI queue useful while the PR is being rebuilt commit-by-commit. Cancel older runs for the same workflow event and branch, but keep push and pull_request groups separate so the latest checks remain visible.
Treat a layer mimetype as the source for its tile extension when ext is not configured, and normalize legacy ext-only configs back to MIME types at config load time.

Keep existing ext configs working while letting TileJSON, WMTS, JOSM imagery, overview HTML, GetTile cache paths, and downloaded cache writes use mimetype-only layers.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
@Komzpa
Komzpa force-pushed the ai/pr21-semantic-rebuild branch from 592f9bd to a86e856 Compare July 26, 2026 12:37
Komzpa and others added 6 commits July 26, 2026 16:40
Accept image/webp as a normal WMS GetMap output format and advertise it in the legacy WMS 1.1.1 capabilities response.

Keep the change tiny: it reuses Pillow's WEBP encoder through the existing format map instead of adding a new protocol or rendering path.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Accept an optional layer_defaults mapping from Python configs and expose it through a tiny dict wrapper rather than importing the fork's full config rewrite.

Defaults participate in get() and [] lookups while explicit layer keys still remain distinguishable, and format metadata is normalized for defaults as well as per-layer values.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the stdlib server compatible with the useful route shape from the fork by accepting /wms/<layer>/<z>/<x>/<y> tile URLs, /wms query requests with a /wms service URL, and /josm/maps.xml.

This keeps the route layer tiny and delegates behavior to the existing twms_main requests instead of importing the fork's larger api.py split.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
List the useful local overview, WMS, WMTS, and JOSM imagery URLs when the stdlib server starts, and expose the TWMS version in the Server header.

This harvests the helpful operator-facing part of the fork's server startup changes without importing its logging/color-output rewrite.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Allow readable config fetch values such as tms, tile, and wms to resolve to the existing legacy fetcher functions.

This harvests the useful config readability part of the fork without importing its TileFetcher class rewrite or provider-specific dynamic URL discovery.

Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Use python -c for the import smoke check so the same workflow step works under both Bash on Linux and PowerShell on Windows.
@Komzpa
Komzpa marked this pull request as ready for review July 26, 2026 14:04
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@Komzpa
Komzpa merged commit d190a43 into master Jul 26, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant