Skip to content

Commit 4a59bd6

Browse files
committed
Additions to fileIO.jl
1 parent b8cbf14 commit 4a59bd6

2 files changed

Lines changed: 43 additions & 11 deletions

File tree

docs/comparisons/dataio/fileio.md

Lines changed: 39 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ There are many file formats, and they are good for different things. Some format
1414

1515
## Overview
1616
Because most packages work for different files, they are not in direct competition, and a comparison does not make much sense. It is more appropriate to compare options that all operate on the same file type, or with the same purpose. With that said, here is a comparison of all the packages on this page:
17-
{{ star_history FileIO ImageIO CSV Arrow Parquet MAT XLSX JSON JSON2 JSON3 JSONBase LazyJSON BSON LightBSON JSONRPC JLD2 JLD}}
17+
{{ star_history FileIO ImageIO CSV Arrow Parquet MAT XLSX JSON JSON2 JSON3 JSONBase LazyJSON BSON LightBSON JSONRPC JLD2 JLD JLSO JDF Serde}}
1818

1919
## Packages
2020
There are three types of subsections under "Packages":
@@ -36,38 +36,56 @@ Images can be loaded with [FileIO.jl](#file-io). However, there are two other al
3636
#### ImageIO.jl
3737
{{badge ImageIO}}
3838

39-
### Saving Arbitrary Julia Objects
40-
If it often useful to save variables stored in your julia session, and be able to redefine them in a new julia session. For example if one of the variables is the result of a long-running computation. There several packages that are good for this specific use case. The general recommendation is **JLD2.jl**.
39+
### Saving Arbitrary Julia Objects (Serializatoin)
40+
If it often useful to save variables stored in your julia session, and to be able to redefine them in a new julia session. For example if one of the variables is the result of a long-running computation. There several packages that are good for this specific use case. The general recommendation is **JLD2.jl**.
4141

4242
All options listed in this subsection support saving and loading just about anything you throw at it: Numbers, arrays, functions, even user-defined structs. This is generally done by saving a dictionary, where the keys are usually the variable name, and the values are the thing being saved:
43+
4344
```julia
44-
julia> using FileIO # FileIO uses JLD2 under the hood, given a .jld2 filename
45+
julia> using JLD2
4546

4647
julia> my_func(x) = x^2
4748
my_func (generic function with 1 method)
4849

4950
julia> save("test.jld2", Dict(["my_func"=>my_func]))
5051

51-
julia> loaded_func = load("test.jld2")
52-
Dict{String, Any} with 1 entry:
53-
"my_func" => my_func
54-
55-
julia> loaded_func["my_func"](3) # Computes 3^2
56-
9
52+
julia> loaded_func = load("test.jld2")["my_func"]; loaded_func(4)
53+
16 # Computed 4^2
5754
```
5855

59-
{{star_history JLD2 JLD BSON}}
56+
{{star_history JLD2 JLD JLSO BSON Serde}}
6057

6158
See also [FileIO.jl](#fileiojl), which can also save arbitrary julia objects by calling the listed packages internally.
6259

6360
#### JLD2.jl
6461
{{badge JLD2}}
6562
At at initial glance, the difference between JLD2 and JLD comes down to the fact that JLD2 is "without any dependency on the HDF5 C library".
6663

64+
JLD2 allows the user to save all variables in the current module's global scope using the syntax `@save filename`.
65+
6766
#### JLD.jl
6867
{{badge JLD}}
6968
The original package for saving arbitrary julia objects. It seems like new users should prefer to use JLD2, and that this is mostly a legacy package.
7069

70+
#### JLSO.jl
71+
> Julia Serialized Object (JLSO) file format for storing checkpoint data.
72+
```julia
73+
julia> using JLSO, Dates
74+
75+
julia> JLSO.save("breakfast.jlso", :food => "☕️🥓🍳", :cost => 11.95, :time => Time(9, 0))
76+
77+
julia> loaded = JLSO.load("breakfast.jlso")
78+
Dict{Symbol,Any} with 3 entries:
79+
:cost => 11.95
80+
:food => "☕️🥓🍳"
81+
:time => 09:00:00
82+
```
83+
84+
#### Serde.jl
85+
> Serde is a Julia library for (de)serializing data to/from various formats. The library offers a simple and concise API for defining custom (de)serialization behavior for user-defined types.
86+
87+
Inspired by the serde.rs Rust library, it supports (de)serialization of the following data formats: JSON, TOML, XML, YAML, CSV, Query. Support for MsgPack and BSON is planned.
88+
7189
### CSV and other delimited files
7290
CSV stands for comma seperated values, and comma is the most common delimiter in delimited files. Other common options include tab and semicolon. All delimited files are human readable, and use plain text encoding. This can make them especially easy to write, and read directly as plain text. The main drawback is that such delimited files are not the fastest nor the smallest option for working with data.
7391

@@ -215,5 +233,15 @@ From its README:
215233
* [Should I still be using JSON.jl?](https://discourse.julialang.org/t/should-i-still-be-using-json-jl/50809)
216234
* [So many JSON libraries; which should I use? (reddit)](https://www.reddit.com/r/Julia/comments/ni7dgk/so_many_json_libraries_which_should_i_use/)
217235

236+
### JDF.jl
237+
JDF is a DataFrames serialization format with the following goals:
238+
- Fast save and load times
239+
- Compressed storage on disk
240+
- Enable disk-based data manipulation (not yet achieved)
241+
- Supports machine learning workloads, e.g. mini-batch, sampling (not yet achieved)
242+
243+
JDF.jl is the Julia package for all things related to JDF.
244+
218245

246+
## Disclaimer
219247
This section could use some love. If you have used or developed Julia packages in this domain, we would love your help! Please visit the ["Contributing" section](https://github.com/JuliaPackageComparisons/JuliaPackageComparisons.github.io#contributing) of the [repository that hosts this website](https://github.com/JuliaPackageComparisons/JuliaPackageComparisons.github.io) for information on how to contribute.

docs/utils.jl

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,10 @@ const PKGINFOS = [
144144
PkgInfo(pkgname="DLMReader", username="sl-solution"),
145145
PkgInfo(pkgname="ReadWriteDlm2", username="strickek", docslink=nothing),
146146
PkgInfo(pkgname="uCSV", username="cjprybol", branch="master", ),
147+
PkgInfo(pkgname="JLSO", username="invenia", branch="master", ),
148+
PkgInfo(pkgname="JDF", username="xiaodaigh", docslink=nothing),
149+
PkgInfo(pkgname="Serde", username="bhftbootcamp", branch="master"),
150+
# PkgInfo(pkgname="LopShelve", username="machkouroke", docslink="https://lopuniverse.me/LopShelve.jl/"),
147151
]
148152

149153
function get_pkginfo(pkgname)

0 commit comments

Comments
 (0)