Repository navigation
Recipe Date Enrichment
Tier: Intermediate
Commands used: datefmt, luau, partition, apply
Anchor dataset: NYC 311 Service Requests (1M-row sample, nyc311samp.csv)
This recipe expands the original Cookbook entry. The original snippets are preserved at the bottom of Cookbook for backwards compatibility.
NYC 311 service-request records have a Created Date like 12/31/2014 11:59:45 PM. For analytics you typically want:
- ISO 8601 normalized timestamp
- a
Quartercolumn (Q1,Q2, …) for fiscal-year reporting - a
Yearcolumn for partitioning - a
Weekdaycolumn for "is this complaint a weekday or weekend phenomenon?" -
Turnaround Time(TAT) — how long betweenCreated DateandClosed Date? - partitioned output: one file per quarter for parallel downstream processing
All of this in one qsv pipeline.
curl -LO https://raw.githubusercontent.com/wiki/dathere/qsv/files/nyc311samp.csv
ls -lh nyc311samp.csv
qsv headers nyc311samp.csvKey columns: Created Date, Closed Date, Borough, Complaint Type, Status, Resolution Action Updated Date.
For larger experiments, also try the 1M-row bundled sample: resources/test/NYC_311_SR_2010-2020-sample-1M.csv.
qsv datefmt 'Created Date' nyc311samp.csv > step1.csvdatefmt recognizes 19 input formats automatically. The default output is RFC 3339 (2014-12-31T23:59:45Z).
qsv datefmt 'Created Date' --formatstr '%Y' --new-column Year step1.csv > step2.csv
qsv datefmt 'Created Date' --formatstr '%Y-%m' --new-column YearMonth step2.csv > step3.csv--formatstr takes chrono strftime specifiers.
qsv datefmt 'Created Date' --formatstr '%A' --new-column Weekday step3.csv > step4.csv%A is the full weekday name. For numeric quarter, use a small Luau script (since strftime doesn't have a quarter specifier):
-- getquarter.lua (also at docs/cookbook/lua/getquarter.lua)
local month = tonumber(string.sub(col['Created Date'], 6, 7)) -- ISO YYYY-MM
local q = math.ceil(month / 3)
return "Q" .. qqsv luau map Quarter getquarter.lua step4.csv > step5.csvColumn values are exposed as globals by default — access them with col['Created Date'] (or col.Amount / col[1], or the bare column name). Pass --no-globals to turn that off. The script file is just a positional argument (there is no -x or -f flag).
-- turnaroundtime.lua (also at docs/cookbook/lua/turnaroundtime.lua)
-- needs date.lua from https://github.com/Tieske/date
local date = require "date"
if string.len(col['Created Date']) == 0 or string.len(col['Closed Date']) == 0 then
return ""
end
local diff = date.diff(date(col['Closed Date']), date(col['Created Date']))
return diff:spandays() -- fractional daysqsv luau map TAT turnaroundtime.lua step5.csv > step6.csvBoth scripts are bundled at docs/cookbook/lua/ — use them as starting points.
qsv partition Quarter nyc311byqtr/ step6.csv
ls nyc311byqtr/
# Q1.csv Q2.csv Q3.csv Q4.csvNow you have four files for parallel downstream processing.
qsv datefmt 'Created Date' nyc311samp.csv \
| qsv datefmt 'Created Date' --formatstr '%Y' --new-column Year \
| qsv datefmt 'Created Date' --formatstr '%Y-%m' --new-column YearMonth \
| qsv datefmt 'Created Date' --formatstr '%A' --new-column Weekday \
| qsv luau map Quarter getquarter.lua \
| qsv luau map TAT turnaroundtime.lua \
> nyc311_enriched.csv
# Then partition
qsv partition Quarter nyc311byqtr/ nyc311_enriched.csvqsv search --select Borough -i brooklyn nyc311samp.csv \
| qsv luau map TAT -x -f turnaroundtime.lua \
| tee brooklyn-311-details.csv \
| qsv stats --everything > brooklyn-tat-stats.csvtee writes the intermediate to a file and pipes the same data to stats for aggregation.
qsv datefmt '/(?i)_date$/' nyc311samp.csv > all_dates_iso.csvThe regex column selector matches any column whose name ends in _date (case-insensitive).
qsv sqlp nyc311samp.csv \
"SELECT
STRFTIME(ts, '%Y') || '-Q' || CAST(EXTRACT(QUARTER FROM ts) AS VARCHAR) AS quarter,
Borough,
COUNT(*) AS complaints,
ROUND(AVG(EXTRACT(EPOCH FROM closed) - EXTRACT(EPOCH FROM ts)) / 86400.0, 2) AS avg_tat_days
FROM (
SELECT Borough,
STRPTIME(\"Created Date\", '%m/%d/%Y %H:%M') AS ts,
STRPTIME(\"Closed Date\", '%m/%d/%Y %H:%M') AS closed
FROM nyc311samp
WHERE \"Closed Date\" != ''
) t
GROUP BY quarter, Borough
ORDER BY quarter, complaints DESC"Polars SQL has no julianday, and its STRFTIME takes the value first and the format second. Its automatic date parsing also can't read US month/day/year, so the subquery parses both columns explicitly with STRPTIME. nyc311samp.csv stores 10/31/2017 8:45; the full NYC 311 export uses 12/31/2014 11:59:45 PM, which needs '%m/%d/%Y %I:%M:%S %p'. EXTRACT(EPOCH FROM …) turns each timestamp into seconds, so the difference ÷ 86,400 is the turnaround in days. See SQL & Polars.
Note
Polars SQL has bare date-part functions, so the quarter label can also be written YEAR(ts) || '-Q' || QUARTER(ts).
qsv luau filter \
"local h = tonumber(string.sub(_['Created Date'], 12, 13)); return h >= 9 and h <= 17" \
nyc311_enriched.csv > business_hours.csv-
datefmtis multithreaded (🚀 in the README legend) — no index required. -
luauis single-process but the interpreter is fast (millions of rows/sec on simple expressions). - On the full 27M-row NYC 311 export, the whole pipeline above takes ~2 minutes on an M2 Pro with an index.
- For a no-index NYC 311 1M-sample run: ~6 seconds.
- Transform & Reshape → datefmt
- Scripting (Luau / Python) — Luau details
docs/cookbook/lua/getquarter.luadocs/cookbook/lua/turnaroundtime.lua- qsv-recipes — more community Luau scripts
- Joins & Set Ops → partition
- Recipe: Stats → Insights — after enrichment, derive aggregates
- Recipe: Multi-Table Joins — join the enriched output to weather (asof join)
- Cookbook (legacy) — original short snippets
qsv — GitHub · Releases · Discussions · crates.io · qsv pro · Try it online · Benchmarks · DeepWiki · DOI · Licensed MIT · Third-party notices · AI Policy
qsv was made possible by datHere: standards-based, best-of-breed, open source solutions to make your Data Useful, Usable & Used.
Edit this page: Contributing to the Wiki
Home · Why qsv? · Tier legend
- Whirlwind Tour · Notebooks · Lessons & Exercises
- Benchmarks · Interactive Benchmarks
- Visualizations Gallery
- Neuro-symbolic, Localizable, Data Schematics
- Feature Flags · Goals/Non-goals · Testing
- Talks: NYC SOD 2022 · csv,conf,v8 · PyCon US 2025 · csv,conf,v9 · NYC SOD 2026
- "Have we achieved ACI?" series: 1 · 2 · 3 · 4
- Sponsor
- All Commands (index)
- Selection & Inspection
- Transform & Reshape
- Aggregation & Statistics
- Joins & Set Ops
- SQL & Polars
- Validation & Schema
- Metadata Profiling (profile)
- Conversion & I/O
- Geospatial
- Visualization (viz)
- HTTP & Web
- Get & Disk Cache
- Scripting (Luau / Python)
- Indexing, Compression & Diff
- AI & Documentation
- Recipes index
- Inspect an Unknown CSV
- Clean & Normalize
- Geographic Enrichment
- Date Enrichment
- CKAN Integration
- DCAT-US v3 for data.gov
- JSON Schema Validation
- Round-Trip SAS / Stata / SPSS
- Build a Data Pipeline
- Stats → Insights
- Data-Quality Signals with moarstats
- Group By
- Fetch & Cache
- Larger-than-RAM CSV
- Diff & Audit
- Multi-table Joins
- Synthesize Fake Data
- Performance Tuning
- Benchmarks
- Tuned csv Fork Benchmarks
- Security & Dependency Audits
- Environment Variables
- Stats Cache & Caching
- Lookup Tables
- MiniJinja Templating