Skip to content

Bittencourt/DESSEM2Julia

BranchesTags

Repository files navigation

DESSEM2Julia

CI codecov

Installation

Quick Install

Install the latest release from GitHub:

using Pkg
Pkg.add(url="https://github.com/Bittencourt/DESSEM2Julia.jl")

Verify Installation

Test your installation with the greet() function:

using DESSEM2Julia
DESSEM2Julia.greet()
# Output: "Hello World!"

For additional installation options (development mode, local installation, etc.), see INSTALL.md.

Convert DESSEM input files (.DAT and related text files) into structured Julia objects and persist them in JLD2 format.

DESSEM is a hydrothermal dispatch optimization model used for short-term operational planning in Brazilian power systems.

🎉 PROJECT COMPLETE (December 20, 2025) - See Completion Summary

All 32 DESSEM parsers implemented • 7,680+ tests passing • Version 1.0.0

📚 Documentation

🗺️ Complete Documentation Index - Wiki-style navigation for all documentation
📖 Documentation Hub - Organized documentation overview
🗺️ Documentation Sitemap - Visual map of all docs

Quick Links:

Quickstart

Open this folder in VS Code, then in a Julia REPL run:

using Pkg
Pkg.activate(".")
Pkg.instantiate()
Pkg.test()

Documentation

Getting Started

Planning & Progress

Architecture & Design

Parser Implementation Guides

Session Summaries

Examples

See examples/README.md for a complete list of examples.

Sample Data

The project includes two real-world DESSEM cases for testing:

  • DS_CCEE_102025_SEMREDE_RV0D28: CCEE case without network constraints (October 2025)
  • DS_ONS_102025_RV2D11: ONS case with network modeling enabled (October 2025, Rev 2)

Both samples are fully supported by the existing parsers. See docs/planning/ONS_COMPATIBILITY_SUMMARY.md for verification results.

Contributing

Tests run in CI on pushes/PRs. To run tests locally before committing, enable the pre-commit hook:

PowerShell:

./scripts/setup-hooks.ps1

Then commit as usual; the hook will run Pkg.test().

Development

  • Main module: src/DESSEM2Julia.jl
  • Type definitions:
    • Legacy types: src/types.jl
    • Core type system: src/models/core_types.jl ⭐ (40+ types, 32/32 files complete - 100%)
  • Parser infrastructure: src/parser/ (36 parser files, all DESSEM formats implemented)
    • Common utilities: src/parser/common.jl
    • Registry system: src/parser/registry.jl
    • All 32 DESSEM parsers complete ✅ (7,680+ tests passing)
    • Master file index: src/parser/dessemarq.jl ✅
    • Hydroelectric: src/parser/hidr.jl, src/parser/hidr_binary.jl ✅ (binary + text, 111 fields)
    • Thermal: src/parser/termdat.jl, src/parser/operut.jl ✅
    • System config: src/parser/entdados.jl ✅ (35+ record types)
    • Network: src/parser/desselet.jl, src/parser/network_topology.jl ✅
    • Constraints: src/parser/operuh.jl, src/parser/restseg.jl, src/parser/rstlpp.jl ✅
    • See docs/file_formats.md for complete parser list
  • JLD2 I/O: src/io.jl
  • Public API: src/api.jl
  • Tests: test/runtests.jl, test/*_tests.jl

Linting and formatting

We enforce consistent formatting in CI using JuliaFormatter. The CI lint job runs the formatter and fails if any diffs are produced.

  • Config: .JuliaFormatter.toml
  • Line endings: .gitattributes enforces LF for .jl, .toml, .yml, .yaml, .json to avoid CRLF/LF drift
  • CI: .github/workflows/ci.yml includes a "Lint (JuliaFormatter)" job that prints a unified diff when changes are needed

Run locally in a temporary environment (does not touch Project.toml):

julia --project=. scripts/format_ci.jl

Alternatively (will add JuliaFormatter to your active environment):

julia -e "using Pkg; Pkg.add(;name=\"JuliaFormatter\", version=\"1\"); using JuliaFormatter; format(\".\", verbose=true)"

Notes for Windows:

  • The pre-commit hook may not run if julia.exe isn’t on the expected path; if needed, commit with --no-verify and run the formatter manually.
  • If you see CI lint failures, open the lint job logs; it now prints a unified diff to show exactly what to change.

Current Status

✅ Core Type System - Production Ready ⭐ NEW

  • Comprehensive data model for all 32 DESSEM files
  • 40+ types organized into 11 functional subsystems
  • 15/32 files with complete type definitions (47% coverage)
  • Hierarchical design: DessemCase → Subsystems → Records
  • Full documentation in docs/type_system.md

✅ dessem.arq Parser - Production Ready

  • Master file index parser (file registry/manifest)
  • 68/68 tests passing (100%)
  • Maps all 32 DESSEM input files dynamically
  • Ordered files registry exposes every entry as DessemFileRecord for iteration

✅ TERMDAT.DAT Parser - Production Ready

  • Parses thermal plant registry (CADUSIT, CADUNIDT, CURVACOMB records)
  • 110/110 tests passing (100%)
  • Successfully parses 98 plants, 387 units from actual DESSEM files

✅ ENTDADOS.DAT Parser - Production Ready ⭐ 100% COMPLETE

  • Parses general operational data (35+ record types - complete coverage)
  • 2,362/2,362 tests passing (100%)
  • Successfully parses all real CCEE and ONS data:
    • Core data: 73 time periods, 5 subsystems, 168 hydro plants, 116 thermal plants, 293 demand records
    • Network configuration: RD (network options), RIVAR (variable restrictions), REE (energy regions)
    • Hydro modeling: TVIAG (water travel time), USIE (pump storage stations)
    • Electrical constraints: RE (constraint definitions), LU (limits with optional fields)
    • Plant adjustments: AC (variable format support), AG (aggregate groups)
    • Coefficient records: FH, FT, FI, FE, FR, FC (hydro/thermal/interchange/energy/renewable/load)
    • Advanced parameters: TX (discount rate), EZ (coupling %), R11 (gauge variations), FP (FPHA), SECR (river sections), CR (head-flow polynomials)

✅ OPERUT.DAT Parser - Production Ready

  • Parses thermal unit operational data (INIT, OPER records)
  • 62/62 tests passing (100%)
  • Successfully parses real CCEE production data:
    • 387 thermal units (47 ON, 340 OFF states)
    • 422 operational constraint records
    • Fixed-width column format based on IDESEM reference
  • INIT records publish initial_status (0=off, 1=on) for cross-parser consistency

✅ DADVAZ.DAT Parser - Production Ready ⭐ NEW

  • Parses natural inflow data and metadata
  • 13/13 tests passing (100%)
  • Successfully parses real CCEE data:
    • Header metadata (plant roster, study start, FCF configuration)
    • Daily inflow slices for 168 hydro plants
    • Handles symbolic period markers ("I"/"F") and optional hours

✅ OPERUH.DAT Parser - Production Ready ⭐ NEW

  • Parses hydro operational constraints (REST, ELEM, LIM, VAR records)
  • All tests passing (100%)
  • Successfully parses real ONS data:
    • Constraint definitions (REST records)
    • Plant participation factors (ELEM records)
    • Time-varying operational limits (LIM records)
    • Ramp/variation constraints (VAR records)
    • Constraint ID linking validated

✅ DESSELET.DAT Parser - Production Ready ⭐ NEW

  • Parses network case mapping (base cases + patamares)
  • 15/15 tests passing (100%)
  • Successfully parses real ONS data:
    • Base network cases (4 PWF files: leve, media, pesada, etc.)
    • Patamar scenarios (48 half-hourly time stages)
    • Date/time scheduling with duration
    • Base case reference linking
  • Exception: Uses split() instead of fixed-width (variable spacing in actual files)

✅ HIDR.DAT Parser - Production Ready ⭐ COMPLETE

  • Parses binary hydro plant registry (111 fields from IDESSEM specification)
  • 54/54 tests passing (100%)
  • Successfully parses 320 plants from ONS production data
  • Complete field coverage:
    • Basic identification (name, posto, subsystem, company)
    • Storage limits (min/max/spillway/diversion volumes)
    • Elevation data (min/max reservoir levels)
    • Polynomials (volume-cota, cota-area curves - 10 coefficients)
    • Evaporation (12 monthly values)
    • Machine sets (up to 5 sets: capacity, head, flow)
    • Performance (productivity, losses, tailrace polynomials - 36 coefficients)
    • Operational parameters (15 fields: TEIF, IP, regulation type, etc.)
  • Binary format: 792 bytes/plant (little-endian)
  • Special handling: 8-byte posto_bdh field, 300-byte reserved block
  • See docs/sessions/session14_hidr_complete.md for details

✅ RENOVAVEIS.DAT Parser - Production Ready ⭐ SESSION 20 - NEW!

  • Parses renewable energy plant data and relationships (EOLICA, EOLICASUBM, EOLICABARRA, EOLICA-GERACAO records)
  • 45/45 tests passing (100%)
  • Successfully parses renewable plant types (wind, solar, biomass, small hydro):
    • Plant registrations (code, name, pmax, fcap, cadastro)
    • Subsystem mappings (market region relationships)
    • Bus mappings (electrical network connections)
    • Generation forecasts (time series availability data)
  • Semicolon-delimited format (exception to DESSEM fixed-width standard)
  • Handles multiple plant types: UEE (wind), UFV (solar), UTE (biomass), PCH/CGH (small hydro)
  • Extracts electrical network topology from PDO output files
  • 1,932/1,932 tests passing (100%)
  • Successfully extracts from real ONS data:
    • 342 buses with generation/load/voltage data
    • 629 transmission lines with flow/capacity data
    • Subsystem mapping (NE, SE, S, N)
    • Connectivity analysis and graph metrics
    • Visualization capabilities:
    • Interactive network diagrams (buses & lines)
    • Color-coded by subsystem
    • Edge thickness by power flow magnitude
    • Spring layout for natural clustering
    • See examples/NETWORK_VISUALIZATION.md for guide

✅ Network Topology - Production Ready ⭐ SESSION 16

  • Extracts electrical network topology from PDO output files
  • 1,932/1,932 tests passing (100%)
  • Successfully extracts from real ONS data:
    • 342 buses with generation/load/voltage data
    • 629 transmission lines with flow/capacity data
    • Subsystem mapping (NE, SE, S, N)
    • Connectivity analysis and graph metrics
  • Visualization capabilities:
    • Interactive network diagrams (buses & lines)
    • Color-coded by subsystem
    • Edge thickness by power flow magnitude
    • Spring layout for natural clustering
    • See examples/NETWORK_VISUALIZATION.md for guide

✅ RESPOT.DAT Parser - Production Ready ⭐ SESSION 23 - NEW!

  • Parses power reserve requirement files (critical for system reliability)
  • 235/235 tests passing (100%) - Fixed from 59/80 after column position corrections
  • Successfully parses real ONS data:
    • Reserve pool definitions (RP records: area + time window + description)
    • Minimum reserve limits (LM records: half-hourly MW requirements, typically 48 per day)
    • Handles symbolic period markers ("I" for initial, "F" for final day)
    • Half-hourly time series (0-23 hours, 0-1 half-hour indicator)
  • Key lesson: Character-by-character position analysis essential for fixed-width formats
  • Fixed-width column format based on real ONS data analysis:
    • Pos 10-11: day (2 chars)
    • Pos 13-14: hour (2 chars, space-padded)
    • Pos 16: half-hour (1 char: 0 or 1)
    • Pos 18-19: day_final (2 chars)
    • Pos 26-35: limit value (F10.2)

✅ PTOPER.DAT Parser - Production Ready ⭐ NEW!

  • Parses operating point / schedule definitions
  • 17/17 tests passing (100%)
  • Successfully parses real CCEE and ONS data:
    • Fixed-width format handling element types (TERM, HIDR, etc.)
    • Parses start/end times (Day, Hour, Half-hour)
    • Handles optional fields and variable spacing
  • Validated against IDESEM column specifications

📊 Overall Parser Progress

Status: 32/32 parsers implemented (100%) ✅ 🎉 Tests: 7,680+ tests passing ✅

PROJECT COMPLETE! All DESSEM file parsers are implemented:

  • 26 production-ready parsers with full structured parsing
  • 6 appropriate placeholder parsers for binary/proprietary formats

For detailed status, see:

Production Parsers (26):

  • Core: dessem.arq, termdat.dat, entdados.dat, operut.dat, dadvaz.dat, deflant.dat, operuh.dat, desselet.dat, hidr.dat
  • Constraints: areacont.dat, respot.dat, respotele.dat, restseg.dat, rstlpp.dat, rmpflx.dat, rampas.dat
  • Renewables: renovaveis.dat (Wind/Solar/Biomass)
  • Auxiliary: cotasr11.dat, curvtviag.dat, dessopc.dat, ptoper.dat, rivar.dat
  • Placeholders: bateria.dat, ilstri.dat, tolperd.dat, metas.dat

Binary/Proprietary Placeholders (6):

  • mlt.dat (binary FPHA), modif.dat
  • DECOMP integration files (placeholder/binary parsers):
    • infofcf.dec, mapcut.dec, cortes.dec - Binary placeholders
    • ✅ cortdeco.rv2 / cortdeco.rv0 - FCF cuts binary parser (51 tests ✅)
    • mapcut.rv2 / mapcut.rv0 - Cut mapping (pending research)

🧪 Test Coverage

Total Tests: 7,680+ tests passing ✅ 🎉

  • Comprehensive unit tests for all 32 implemented parsers
  • Integration tests with real CCEE and ONS data
  • Network topology extraction and visualization tests
  • Binary file parsing validation

About

Convert DESSEM input files (.DAT and related text files) into structured Julia objects and persist them in JLD2 format. DESSEM is a hydrothermal dispatch optimization model used for short-term operational planning in Brazilian power systems.

Resources

Readme

License

MIT license
Activity

Stars

2 stars

Watchers

0 watching

Forks

1 fork
Report repository

Releases 1

v0.1.0 Latest
Jan 9, 2026

Packages

 
 
 

Contributors

Languages