Consolidate trace output into a single recorder.dat - #43
Merged
Conversation
- Remove cmake_policy(SET CMP0074 OLD) to resolve CMake deprecation warning - Guard HDF5 includes, macros, and GOTCHA bindings with RECORDER_WITH_HDF5 - Make find_package(HDF5) optional; add recorder-hdf5.c only when found - Add RECORDER_WITH_HDF5 compile definition when HDF5 is detected - Guard hdf5_tracing env var and gotcha registration with RECORDER_WITH_HDF5 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make HDF5 optional and fix CMake CMP0074 warning
- Add test/CMakeLists.txt: builds all test executables and registers test_posix, test_iopr, and test_mpi as CTest targets, each running with LD_PRELOAD=librecorder.so and an isolated RECORDER_TRACES_DIR - Wire add_subdirectory(test) into the existing BUILD_TESTING block - HDF5 tests (test_hdf5, test_phdf5) are registered automatically when HDF5 is detected at configure time - test_signal and test_hybrid are compiled but not registered: test_signal requires a manual kill signal and hits a ts_merge_files bug on normal exit; test_hybrid relies on hardcoded paths - Fix pre-existing bug in test_iopr.c: open() with O_CREAT was missing the required mode argument Run tests with: cmake --build build && cd build && ctest Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add CTest integration and fix test_iopr open() bug
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add CLAUDE.md with build, architecture, and test guidance
- Add version_{major,minor,patch} and num_funcs fields to
RecorderMetadata; define RECORDER_FUNC_NAME_LEN=64.
- recorder.mt now starts with the full struct followed by
num_funcs fixed-size 64-byte null-padded function name
entries -- no 1024-byte padding, no newline-delimited text,
no separate VERSION file.
- Writer (save_global_metadata) removes any legacy VERSION file
left in the traces directory so repeated runs on the same
directory don't confuse the reader.
- Reader (check_version / read_metadata) detects the format via
VERSION-file presence: legacy path maps old structs (v2.3 and
old v3) into the new RecorderMetadata layout; new-format path
reads the struct directly and then bulk-reads binary name entries.
- recorder-filter: rewrite save_updated_metadata to produce
new-format output; remove VERSION file copy.
- Python recorder_reader: add _RecorderMetadata ctypes struct,
switch __read_num_procs and __load_func_list to handle both
legacy and new formats.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- test_reader.c: validates recorder_init_reader() against a live trace
directory; checks version matches reader, num_funcs > 0 and consistent
with metadata.num_funcs, func_list entries non-empty, posix_tracing set,
and at least one record decoded for rank 0.
- recorder_add_check_tests() macro registers four tests per trace set:
check_reader_<name> - reader API validation (test_reader binary)
check_summary_<name> - recorder-summary exits 0, output matches
"Total processes: [0-9]+"
run_text_<name> - recorder2text exits 0
verify_text_<name> - _text/0.txt was created (cmake -E cat)
- Fixture chains ensure trace-generation tests run before tool checks,
and run_text before verify_text.
- test_posix now passes RECORDER_WITH_NON_MPI=1 so it actually produces
traces (previously the directory was empty for non-MPI runs).
- Grows the suite from 3 tests to 15.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Make recorder.mt fully binary and add postprocessing tests
Replaces the multi-file output (recorder.mt, recorder.ts, recorder.cst, ug.cfg, ug.mt, per-rank N.cst/N.cfg) with a single binary recorder.dat that embeds all sections with a compact header+index structure. Format: 16-byte RecorderFileHeader (magic "RECORDER", format_version=1, num_sections) followed by a section table of RecorderSectionEntry[] (type, rank, offset, size), then the raw section data in order. Writer changes (lib/): - Add RecorderSectionType enum and RecorderFileHeader/RecorderSectionEntry structs to recorder-logger.h - combine_output_files() uses raw POSIX fds (open/read/write/lseek) to avoid GOTCHA/FILE* interference and O_EXCL to prevent multiple-rank races on systems where MPI_Comm_rank returns 0 for all processes - Metadata is written directly into the combined file via write_metadata_section_fd(), eliminating the recorder.mt temp file and its associated truncation race - update_mpi_info() now uses PMPI_Comm_rank/PMPI_Comm_size directly instead of routing through GOTCHA wrappers Reader changes (tools/): - reader.c/reader.h: detect recorder.dat, load section table, open individual sections via open_section(); fall back to legacy multi-file layout when recorder.dat is absent - recorder-filter.cpp: write filtered output as recorder.dat - verifyio/recorder_reader.py: detect and parse recorder.dat Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All tools (recorder2text, recorder2timeline, conflict-detector, recorder-summary) and the Python verifyio reader now accept a path to recorder.dat as their first argument in addition to a trace directory. reader.c normalises the path by stripping the recorder.dat suffix; each tool derives its output directory from reader.logs_dir instead of argv[1]. CTest check tests updated to pass recorder.dat paths. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Consolidate all trace output into a single recorder.dat file
HDF5 now follows the same pattern as PnetCDF and NetCDF: pass -DRECORDER_WITH_HDF5=/path/to/hdf5/install to locate the library. Falls back to find_package(HDF5) when the option is not set. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lib/CMakeLists.txt now propagates HDF5_FOUND, HDF5_INCLUDE_DIRS, and HDF5_LIBRARIES to the parent scope so test/CMakeLists.txt can see them. Added recorder_add_check_tests for test_hdf5 and test_phdf5, bringing the total to 25 tests when built with HDF5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add RECORDER_WITH_HDF5 option and register HDF5 CTest tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR bundles several related infrastructure improvements developed on our fork, built up over a series of small, reviewed changes:
recorder.datfile per run instead of a scattered set of.mt/.ts/.cst/.cfgfiles, with a simple section-table header (magic + section entries with type/rank/offset/size). This removes the oldrecorder.mt-as-rank-0-lock race and makes trace directories self-contained and easier to move/archive.recorder.mtitself was also made fully binary (version + function-name table, no more padded text/VERSION file), and all post-processing tools (reader,recorder2text,recorder2timeline,recorder-filter,conflict-detector, the Pythonverifyioreader) were updated to read the new format, with backward-compatible parsing of legacy trace directories.find_package(HDF5)no longer fails the configure step; HDF5 support (headers, macros, GOTCHA bindings, env var) is guarded byRECORDER_WITH_HDF5, and a-DRECORDER_WITH_HDF5=/pathoption is added for pointing at local installations, with clearer not-found hints.test_posix,test_iopr,test_mpi,test_hdf5,test_phdf5and the post-processing tools now run underctest, including HDF5 tests gated onHDF5_FOUNDbeing propagated to the parent scope.CLAUDE.mdwith build/architecture/test guidance for the repo.