Rebase TWMS modernization onto semantic upstream stack - #1
Open
Komzpa wants to merge 51 commits into
Open
Conversation
Commit cb7d39a erroneously removed a circular import of "filter" which was necessary for filters to work. This needs to be dealt with properly in future.
Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep TWMS' historical filesystem cache layout, but teach it the useful cache behavior from Radioxoma's fork: cache_ttl checks, reusable .tne markers, stale fallback after network errors, and atomic replacement for freshly downloaded tiles. Also accept Radioxoma-style dead_tile dictionaries while preserving the legacy dead_tile file comparison. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep TWMS' built-in projection shortlist tiny, but make the core WebMercator path tolerate pole-edge bounds and route optional pyproj transforms through the modern Transformer API. This deliberately preserves the existing optional twms[proj] behavior for non-core projections instead of taking Radioxoma's hard pyproj removal. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest Radioxoma's WMS 1.3.0 capability work without taking the fork's API/config rewrite or legacy module removals. Keep the existing WMS 1.1.1 generator intact, add a tiny ElementTree WMS 1.3.0 response, accept case-insensitive request keys, and support CRS:84 as the lon/lat WMS 1.3.0 request alias without advertising it as a WMS 1.1.1 SRS. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest the useful deployment and client guidance from Radioxoma's README without taking the fork framing, unsupported-feature removals, or repo layout rewrite. Document the current upstream entry points, QGIS WMS/WMTS URLs, JOSM TMS/GetTile/file-cache forms, cache behavior, and optional dependencies while keeping TWMS described as a general tiny WMS/tile service. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the Windows/JOSM single-executable deployment path without adding runtime framework code to TWMS. Build both the stdlib server entry point and the preserved web.py entry point with PyInstaller in GitHub Actions, upload the executables as CI artifacts, and run the workflow for branches, tags, releases, PRs, and manual dispatch. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Do not take Radioxoma's hard removal of filters, corrections, canvas, and reprojection helpers. Instead keep the legacy surfaces covered and fix current Python/Pillow breakage: use a Pillow resampling compatibility helper, keep canvas usable on Python 3, keep GetCorrections returning text/plain, pass WKT colors correctly when no color parameter is provided, and use Python 3 iterator semantics for GPX colors. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Validate downloaded WMS/TMS image bytes before saving them to the filesystem cache, keep stale cached tiles on invalid downloads, and convert mismatched tile responses to the configured layer extension when possible. This keeps the useful cache-safety behavior from Radioxoma's fork without adopting the larger fetcher rewrite. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Record HTTP 404 tile responses as .tne cache misses and allow layer configs to opt in additional HTTP miss statuses with dead_tile.http_status. This keeps the useful HTTP-status TNE behavior from Radioxoma's fork while avoiding service-specific heuristics and noisy response-body logging. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Accept readable tile URL templates with {z}, {x}, {y}, {-y}, and {q} while preserving the legacy percent-template path and transform_tile_number behavior.
This takes the useful URL placeholder idea from Radioxoma's fork without migrating existing configs or adopting the larger fetcher rewrite.
Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Generate a JOSM maps-1.0 imagery XML document from configured TWMS layers and expose it through both request=GetJOSMImagery and stdlib routes at /josm/imagery.xml and /maps.xml. This keeps the useful JOSM imagery-list idea from Radioxoma's fork without taking the heavy route/config reshuffle or launching JOSM remote-control side effects. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Pass the target image format explicitly when saving through a temporary cache path, so Pillow does not infer the format from the .tmp.<pid> suffix. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Pass optional per-layer HTTP headers to tile and WMS upstream downloads while leaving layers without headers on the legacy string-url path. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Honor per-layer min_zoom in both tile and WMS fetchers while preserving the existing exclusive max_zoom behavior. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Add an opt-in cache_layout for slippy/MOBAC-style <z>/<x>/<y> cache paths while keeping TWMS' historical grouped cache layout as the default. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Allow upstream WMS remote_url templates to supply {bbox}, {width}, {height}, and {proj} while preserving the legacy base-URL append behavior when no WMS placeholders are present.
Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the stdlib threaded server tiny, but avoid waiting forever on a stalled upstream tile or WMS source. Layer configs may override the timeout when an old deployment deliberately needs different behavior. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep malformed REST-style protocol paths from falling through to the overview or disconnecting handler threads. Also fix the legacy GetTile fast-cache layer lookup exposed by WMTS REST tile requests with query strings. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the preserved WmsCanvas helper from waiting forever on remote WMS tiles while retaining an explicit timeout=None escape hatch for deployments that relied on the historical unbounded wait. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the legacy overlay/filter surface by making configured empty_color values transparent again, including per-channel empty_color_delta tolerance. This repairs the behavior instead of deleting the old rendering helpers. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Use integer cache shard directories under Python 3 and read cached GetTile responses as bytes, so the historical grouped cache layout remains usable alongside the new opt-in zxy layout. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the legacy cache_tile_responses fast path under Python 3 by reading and writing cached image responses as bytes instead of text. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest Radioxoma's HTTP retry idea without taking the larger backend rewrite: keep urllib, default to one attempt for legacy behavior, and let global or per-layer config opt in retries for transient transport errors. HTTP errors such as 404 still flow to the existing TNE/cache handling instead of being retried as generic network failures. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the legacy max_ram_cached_tiles cache, but make eviction least-recently-used instead of FIFO/off-by-one history popping. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep legacy cache_tile_responses configs working when they use MIME strings such as image/png, as shown in the packaged example, while still accepting the normalized Pillow format keys used internally. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Document the decision not to import Radioxoma's Firefox cookie-store discovery into upstream TWMS. Private deployments can still provide cookies through per-layer headers or their own Python config code without adding a browser-profile dependency to the tiny server. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the useful hygiene from Radioxoma's added ignore file, but reduce it to artifacts TWMS actually produces during tests, builds, and PyInstaller packaging. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Accept Radioxoma's readable bounds layer metadata key as an alias for historical data_bounding_box/bbox in tile filtering and generated TileJSON/WMTS metadata without replacing the legacy config contract. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Harvest Radioxoma's useful JOSM imagery metadata idea without importing the larger API/config rewrite or launching JOSM remote-control side effects. The generated maps-1.0 XML now carries layer bounds, overlays, attribution URLs, valid-georeference/default hints, and fixed per-MD5 no-tile checksums while preserving TWMS' existing URL and zoom semantics. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the legacy overview page but harvest Radioxoma's useful layer metadata behavior: accept the readable bounds alias for preview/bbox links and link configured provider URLs without reshaping the overview UI. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Preserve the small launcher deployment idea from Radioxoma's branch without installing desktop integration by default. Package Windows batch and Linux desktop-entry templates under share/twms/contrib and document them as optional manual deployment helpers. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the CI queue useful while the PR is being rebuilt commit-by-commit. Cancel older runs for the same workflow event and branch, but keep push and pull_request groups separate so the latest checks remain visible.
Treat a layer mimetype as the source for its tile extension when ext is not configured, and normalize legacy ext-only configs back to MIME types at config load time. Keep existing ext configs working while letting TileJSON, WMTS, JOSM imagery, overview HTML, GetTile cache paths, and downloaded cache writes use mimetype-only layers. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Accept image/webp as a normal WMS GetMap output format and advertise it in the legacy WMS 1.1.1 capabilities response. Keep the change tiny: it reuses Pillow's WEBP encoder through the existing format map instead of adding a new protocol or rendering path. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Accept an optional layer_defaults mapping from Python configs and expose it through a tiny dict wrapper rather than importing the fork's full config rewrite. Defaults participate in get() and [] lookups while explicit layer keys still remain distinguishable, and format metadata is normalized for defaults as well as per-layer values. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Keep the stdlib server compatible with the useful route shape from the fork by accepting /wms/<layer>/<z>/<x>/<y> tile URLs, /wms query requests with a /wms service URL, and /josm/maps.xml. This keeps the route layer tiny and delegates behavior to the existing twms_main requests instead of importing the fork's larger api.py split. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
List the useful local overview, WMS, WMTS, and JOSM imagery URLs when the stdlib server starts, and expose the TWMS version in the Server header. This harvests the helpful operator-facing part of the fork's server startup changes without importing its logging/color-output rewrite. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Allow readable config fetch values such as tms, tile, and wms to resolve to the existing legacy fetcher functions. This harvests the useful config readability part of the fork without importing its TileFetcher class rewrite or provider-specific dynamic URL discovery. Co-authored-by: Eugene Dvoretsky <radioxoma@gmail.com>
Use python -c for the import smoke check so the same workflow step works under both Bash on Linux and PowerShell on Windows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the reciprocal branch update for the semantic rebuild in Komzpa#22.
Komzpa#21 could not be merged as-is, so the useful pieces were rebuilt as reviewable commits on top of current upstream
master, while keeping TWMS tiny and general-purpose. This branch keeps TWMS' legacy WMS/GetTile/web.py surfaces, preserves the old keyboard-style public version0.07z, and credits Eugene Dvoretsky / Radioxoma for the modernization work in the copied copyright and README notes.What was carried over and reshaped:
twms.exeandtwms-webpy.exe;web.py/WSGI path;pyprojfor non-core projections;bounds,mimetype,layer_defaults, and string fetcher aliases.What was intentionally not carried over:
web.pydeployments;Current validation is green in Komzpa#22:
Both runs passed Ubuntu and Windows package smoke jobs for Python 3.11 and 3.13, plus the Windows executable artifact job.
This PR is expected to replace/rebase the fork branch rather than merge cleanly on top of the current fork history: GitHub compare reports the branches as diverged, with this branch ahead of
radioxoma/masterand also behind it.