Skip to content

Conversion to dgCMatrix Causes Segmentation Fault #129

New issue
New issue
Closed
Closed
Conversion to dgCMatrix Causes Segmentation Fault#129

Description

@DarioS
DarioS
opened on Sep 10, 2026
dim(database)
    109578 2389155
class(counts(database))
    "DelayedMatrix"

After about four hours.

assay(database, "counts") <- as(assay(database, "counts"), "dgCMatrix")
     *** caught segfault ***
    address 0x7fc394337000, cause 'invalid permissions'

Activity

  1. vjcitn commented on Sep 11, 2026

    @vjcitn
    vjcitn
    on Sep 11, 2026

    are there more than 2^31 - 1 nonzero elements in the counts? Can the dgCMatrix be constructed without the involvement of DelayedArray? Would SparseArray provide a more useful representation?

  2. hpages commented on Sep 11, 2026

    @hpages
    hpages
    on Sep 11, 2026
    Contributor

    As always, we need a repreoducible example. In particular how was database obtained and what's your sessionInfo()?

    As suggested by Vince, have you tried to replace as(assay(database, "counts"), "dgCMatrix") with as(assay(database, "counts"), "SparseMatrix")?

  3. hpages commented on Sep 11, 2026

    @hpages
    hpages
    on Sep 11, 2026
    Contributor

    I'd also need to see the output of:

    is_sparse(assay(database, "counts"))

    and

    showtree(assay(database, "counts"))

    and

    show(assay(database, "counts"))

    Thanks!

  4. DarioS commented on Sep 12, 2026

    @DarioS
    DarioS
    on Sep 12, 2026
    Author

    Yes, although "invalid permissions" seems like something that needs administrator privileges.

    nnzero(counts(database)) > (2^31 - 1)
    TRUE

    SparseMatrix conversion also fails after some hours with an even less informative error message.

    50  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x157ece) [0x7f68d1557ece]
51  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x158c57) [0x7f68d1558c57]
52  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x148fb7) [0x7f68d1548fb7]
53  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x155c4b) [0x7f68d1555c4b]
54  /savona/biostat/software/R-4.6.0/lib/libR.so(Rf_eval+0x14b) [0x7f68d1555fbb]
55  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x157ece) [0x7f68d1557ece]
56  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x158c57) [0x7f68d1558c57]
57  /savona/biostat/software/R-4.6.0/lib/libR.so(Rf_eval+0x286) [0x7f68d15560f6]
58  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x159fb0) [0x7f68d1559fb0]
59  /savona/biostat/software/R-4.6.0/lib/libR.so(Rf_eval+0x530) [0x7f68d15563a0]
60  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x18f14b) [0x7f68d158f14b]
61  /savona/biostat/software/R-4.6.0/lib/libR.so(+0x18f580) [0x7f68d158f580]
=================================
 
 *** caught segfault ***
address 0xdb500160e62, cause 'unknown'
Segmentation fault

    The object looks like

    > show(assay(database, "counts"))
<109578 x 2389155> DelayedMatrix object of type "double":
                AAATCCTGTAAATAAATACAATACAAG-1 ...    Pt13Cho_TTTGTTGTCTCACTCG-1
          OR4F5                             0   .                             0
         OR4F29                             0   .                             0
         OR4F16                             0   .                             0
         SAMD11                             0   .                             0
          NOC2L                             0   .                             1
            ...                             .   .                             .
Metazoa-SRP-132                             0   .                             0
Metazoa-SRP-134                             0   .                             0
      Y-RNA-604                             0   .                             0
Metazoa-SRP-138                             0   .                             0
Metazoa-SRP-139                             0   .                             0

    Strange that it is double. The nineteen source objects used to build it by combineCols are unexpectedly a mix.

    sapply(humanList, function(x) BiocGenerics::type(counts(x)))
  D005.rds   D007.rds   D010.rds   D011.rds   D018.rds   D019.rds   D020.rds 
 "integer"   "double"  "integer"   "double"  "integer"  "integer"  "integer" 
  D021.rds   D022.rds   D023.rds D024a.rds D024b.rds   D029.rds   D030.rds 
 "integer"   "double"  "integer"   "double"   "double"  "integer"   "double" 
  D031.rds   D032.rds   D033.rds   D034.rds   D035.rds 
 "integer"  "integer"   "double"  "integer"   "double"

    Looking at one of them, D007 started as Seurat object before as.SingleCellExperiment. It was double all along.

    dataset <- readRDS("CR24.rds")
dataset
An object of class Seurat 
33538 features across 10218 samples within 1 assay 
Active assay: RNA (33538 features, 2000 variable features)
 3 layers present: counts, data, scale.data
 2 dimensional reductions calculated: pca, umap

storage.mode(dataset@assays$RNA$counts@x)
    "double"

    D011 is a set of H5AD files downloaded from Zenodo.

    library(zellkonverter)
H5ADs <- list.files(c("data 1-73 files", "data 2-73 files", "data 3-73 files"), pattern = "\\.h5ad$", full.names = TRUE)
all <- lapply(H5ADs, readH5AD, reader = 'R') 
storage.mode(assay(all[[1]], 'X')@x)
    "double"
assay(all[[1]], 'X')[1:5, 1:5]
5 x 5 sparse Matrix of class "dgCMatrix"
            AAACCCATCTTAGGAC-1 AAACGAACACGACGTC-1 AAACGAAGTTCTCGTC-1 AAACGAATCGGAGCAA-1 AAACGCTCAGCAGTAG-1
MIR1302-2HG                  .                  .                  .                  .                  .
FAM138A                      .                  .                  .                  .                  .
OR4F5                        .                  .                  .                  .                  .
AL627309.1                   .                  .                  .                  .                  .
AL627309.3                   .                  .                  .                  .                  .

    So, this Bioconductor package imports counts matrices not as integers but as doubles. But, not the problem.

    as(assay(all[[1]], 'X'), "SparseMatrix")
<36598 x 1735 SparseMatrix> of type "double" [nzcount=2004646 (3.2%)]:
            AAACCCATCTTAGGAC-1 AAACGAACACGACGTC-1 AAACGAAGTTCTCGTC-1 ... TTTGTTGCAGGTGTGA-1 TTTGTTGGTCCCACGA-1
MIR1302-2HG                  0                  0                  0   .                  0                  0
    FAM138A                  0                  0                  0   .                  0                  0
      OR4F5                  0                  0                  0   .                  0                  0
 AL627309.1                  0                  0                  0   .                  0                  0
 AL627309.3                  0                  0                  0   .                  0                  0
        ...                  .                  .                  .   .                  .                  .
 AC141272.1                  0                  0                  0   .                  0                  0
 AC023491.2                  0                  0                  0   .                  0                  0
 AC007325.1                  0                  0                  0   .                  0                  0
 AC007325.4                  0                  0                  0   .                  0                  0
 AC007325.2                  0                  0                  0   .                  0                  0

    The conversion of this small matrix worked fine. The database tree is

    109578x2389155 double: DelayedMatrix object
└─ 109578x2389155 double: Set dimnames
   └─ 109578x2389155 double: Abind (along=2)
      ├─ 109578x20590 double: Subset
      │  └─ 20392x20590 double, sparse: Abind (along=1)
      │     ├─ 20391x20590 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x20590 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x10218 double: Subset
      │  └─ 33286x10218 double, sparse: Abind (along=1)
      │     ├─ 33285x10218 double, sparse: [seed] SparseMatrix object
      │     └─ 1x10218 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x96213 double: Subset
      │  └─ 38328x96213 double, sparse: Abind (along=1)
      │     ├─ 38327x96213 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x96213 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x629187 double: Subset
      │  └─ 36347x629187 double, sparse: Abind (along=1)
      │     ├─ 36346x629187 double, sparse: [seed] SparseMatrix object
      │     └─ 1x629187 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x86253 double: Subset
      │  └─ 33416x86253 double, sparse: Abind (along=1)
      │     ├─ 33415x86253 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x86253 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x24023 double: Subset
      │  └─ 33416x24023 double, sparse: Abind (along=1)
      │     ├─ 33415x24023 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x24023 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x51645 double: Subset
      │  └─ 36350x51645 double, sparse: Abind (along=1)
      │     ├─ 36349x51645 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x51645 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x20009 double: Subset
      │  └─ 32593x20009 double, sparse: Abind (along=1)
      │     ├─ 32592x20009 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x20009 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x20091 double: Subset
      │  └─ 19544x20091 double, sparse: Abind (along=1)
      │     ├─ 19543x20091 double, sparse: [seed] SparseMatrix object
      │     └─ 1x20091 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x247520 double: Subset
      │  └─ 32496x247520 double, sparse: Abind (along=1)
      │     ├─ 32495x247520 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x247520 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x174579 double: Subset
      │  └─ 55073x174579 double, sparse: Abind (along=1)
      │     ├─ 55072x174579 double, sparse: [seed] SparseMatrix object
      │     └─ 1x174579 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x110862 double: Subset
      │  └─ 55073x110862 double, sparse: Abind (along=1)
      │     ├─ 55072x110862 double, sparse: [seed] SparseMatrix object
      │     └─ 1x110862 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x121779 double: Subset
      │  └─ 57870x121779 double, sparse: Abind (along=1)
      │     ├─ 57869x121779 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x121779 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x3248 double: Subset
      │  └─ 18117x3248 double, sparse: Abind (along=1)
      │     ├─ 18116x3248 double, sparse: [seed] SparseMatrix object
      │     └─ 1x3248 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x92385 double: Subset
      │  └─ 33416x92385 double, sparse: Abind (along=1)
      │     ├─ 33415x92385 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x92385 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x100853 double: Subset
      │  └─ 33262x100853 double, sparse: Abind (along=1)
      │     ├─ 33261x100853 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x100853 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x195248 double: Subset
      │  └─ 60289x195248 double, sparse: Abind (along=1)
      │     ├─ 60288x195248 double, sparse: [seed] SparseMatrix object
      │     └─ 1x195248 double, sparse: [seed] ConstantArraySeed object
      ├─ 109578x244474 double: Subset
      │  └─ 32398x244474 double, sparse: Abind (along=1)
      │     ├─ 32397x244474 integer, sparse: [seed] SparseMatrix object
      │     └─ 1x244474 double, sparse: [seed] ConstantArraySeed object
      └─ 109578x139978 double: Subset
         └─ 37296x139978 double, sparse: Abind (along=1)
            ├─ 37295x139978 double, sparse: [seed] SparseMatrix object
            └─ 1x139978 double, sparse: [seed] ConstantArraySeed object

    To create an example, I considered mockSCE but it only ever creates a dense matrix which is not suitable for this size.

    sessionInfo()
R version 4.6.0 (2026-04-24)
Platform: x86_64-pc-linux-gnu
Running under: Debian GNU/Linux 13 (trixie)

  5. hpages commented on Sep 22, 2026

    @hpages
    hpages
    on Sep 22, 2026
    Contributor

    Thanks for the additional details. It's very useful to know that your big 109578x2389155 SummarizedExperiment was obtained by combining 19 smaller ones with combineCols().

    A major problem here is that combineCols() delays its operations by default. For example, with 2 small artificial SummarizedExperiment objects:

    library(DelayedArray)
library(SummarizedExperiment)

counts1 <- rbind(gene1=c(1L, 0L, 0L, 1L),
                 gene2=c(0L, 0L, 2L, 0L),
                 gene3=c(0L, 1L, 1L, 0L))

counts2 <- rbind(gene4=c(0L, 0L, 1L),
                 gene2=c(3L, 0L, 0L))

SE1 <- SummarizedExperiment(list(counts=as(counts1, "SparseMatrix")))
SE2 <- SummarizedExperiment(list(counts=as(counts2, "SparseMatrix")))

SEa <- combineCols(SE1, SE2)

assay(SEa)
# <4 x 7> DelayedMatrix object of type "integer":
#       [,1] [,2] [,3] [,4] [,5] [,6] [,7]
# gene1    1    0    0    1   NA   NA   NA
# gene2    0    0    2    0    3    0    0
# gene3    0    1    1    0   NA   NA   NA
# gene4   NA   NA   NA   NA    0    0    1

showtree(assay(SEa))
# 4x7 integer: DelayedMatrix object
# └─ 4x7 integer: Set dimnames
#    └─ 4x7 integer: Abind (along=2)
#       ├─ 4x4 integer: Abind (along=1)
#       │  ├─ 3x4 integer, sparse: [seed] SparseMatrix object
#       │  └─ 1x4 logical: [seed] ConstantArraySeed object
#       └─ 4x3 integer: Subset
#          └─ 3x3 integer: Abind (along=1)
#             ├─ 2x3 integer, sparse: [seed] SparseMatrix object
#             └─ 1x3 logical: [seed] ConstantArraySeed object

    Because all operations are delayed it's VERY fast, but, unfortunately, everything you'll try to do after that is going to be HUGELY inefficient (as you've experienced).

    A much better way to go is to NOT take the delayed route by using combineCols(SE1, SE2, delayed=FALSE) instead. Unfortunately this wasn't working so far on SummarizedExperiment objects that have SparseMatrix assays. But I just repaired that so it should work now with the latest version of the SparseArray package (1.12.3 in release and 1.13.3 in devel):

    SEb <- combineCols(SE1, SE2, delayed=FALSE)

assay(SEb)
# <4 x 7 SparseMatrix> of type "integer" [nzcount=17 (61%)]:
#       [,1] [,2] [,3] [,4] [,5] [,6] [,7]
# gene1    1    0    0    1   NA   NA   NA
# gene2    0    0    2    0    3    0    0
# gene3    0    1    1    0   NA   NA   NA
# gene4   NA   NA   NA   NA    0    0    1

    This approach has 3 main advantages over the delayed route:

    1. No need to coerce the assay of the combined object to SparseMatrix because it's already a SparseMatrix.
    2. It should be much faster than doing delayed combineCols + coercion.
    3. It should work even if the combined assay has more than 2^31 nonzero values.

    Let's check performance on 4 artificial medium size objects:

    counts1 <- poissonSparseMatrix(2800, 24000, density=0.03, dimnames=list(sprintf("GENE%03d", 1:2800), NULL))
counts2 <- poissonSparseMatrix(3500, 15000, density=0.03, dimnames=list(sprintf("GENE%03d", 1:3500), NULL))
counts3 <- poissonSparseMatrix(2000, 22000, density=0.03, dimnames=list(sample(rownames(counts2), 2000), NULL))
counts4 <- poissonSparseMatrix(4500, 35000, density=0.03, dimnames=list(sprintf("GENE%03d", 1001:5500), NULL))

SE1 <- SummarizedExperiment(list(counts=counts1))
SE2 <- SummarizedExperiment(list(counts=counts2))
SE3 <- SummarizedExperiment(list(counts=counts3))
SE4 <- SummarizedExperiment(list(counts=counts4))

## Delayed combineCols + coercion:
system.time(combined_counts_A <- as(assay(combineCols(SE1, SE2, SE3, SE4)), "SparseMatrix"))
#    user  system elapsed 
#  12.795   5.518  18.272 

## Immediate combineCols (i.e. effectively combine the assays):
system.time(combined_counts_B <- assay(combineCols(SE1, SE2, SE3, SE4, delayed=FALSE)))
#    user  system elapsed 
#   0.692   0.394   1.085 

## 'combined_counts_A' and 'combined_counts_B' contain the same values but their classes
## are different flavors of the SparseArray virtual class:
class(combined_counts_A)
# [1] "COO_SparseMatrix"
# attr(,"package")
# [1] "SparseArray"
class(combined_counts_B)
# [1] "SVT_SparseMatrix"
# attr(,"package")
# [1] "SparseArray"

## Sanity check:
identical(as(combined_counts_A, class(combined_counts_B)), combined_counts_B)
# [1] TRUE

    As you can see, the speedup for this artificial example is about 17x but it might be even more with your 19 objects.

    Hope this helps,

    H.

  6. DarioS commented on Sep 23, 2026

    @DarioS
    DarioS
    on Sep 23, 2026
    Author

    Nice but upgrading the packages has created a new issue. I processed GSM8370247 (seems lots of empty droplets were uploaded).

    library(DropletUtils)
dataset3 <- read10xCounts("GSM8370247_", "B6_2", type = "prefix", mtx.class = "SVT_SparseMatrix")
dim(dataset3) # Chromium instrument can't capture this many cells.
    32286 6794880
keep3 <- colSums(counts(dataset3)) > 500
    Error in .Call2(.NAME, ..., PACKAGE = "SparseArray"): SparseArray error in get_leaf_nzoffs(): invalid SVT leaf

    The crude empty droplet filtering worked before I installed SparseArray 1.12.3 and S4Arrays 1.12.1 and restarted R.

  7. hpages commented on Sep 23, 2026

    @hpages
    hpages
    on Sep 23, 2026
    Contributor

    We don't know what versions of SparseArray or S4Arrays you were using before. Without this information we cannot deduce that the changes that happened between SparseArray 1.12.2 and 1.12.3 or between S4Arrays 1.12.0 and 1.12.1 introduced this new problem. I highly doubt it because these changes didn't touch the internal representation of SVT_SparseMatrix objects.

    BTW we still don't know what version of Bioconductor you're using (you still haven't communicated your sessionInfo()). Note that SparseArray 1.12.3 and S4Arrays 1.12.1 belong to BioC 3.23 (the current release) so make sure that you're indeed using BioC 3.23 and that all your packages are up to date.

    In particular make sure that you have the latest version of DropletUtils, which is version 1.32.0.

    Also, as always, a reproducible example is needed.

    Once we have your sessionInfo() and a reproducible example, we can start working on a solution.

  8. DarioS commented on Sep 23, 2026

    @DarioS
    DarioS
    on Sep 23, 2026
    Author

    Ah, was trying to avoid overwhelming details. A (hopefully reproducible) example:

    1. Download the trio of files from the webpage above, e.g. getGEOSuppFiles("GSM8370247"), or simply internet browser. Then,
    library(DropletUtils)
library(SparseArray)
# setwd("GSM8370247") # If getGEOSuppFiles used.
dataset <- read10xCounts("GSM8370247_Singlecell_B6_sample_2_", "B6_2", type = "prefix", mtx.class = "SVT_SparseMatrix")
colSums(counts(dataset)) > 500 # Error triggered

    The environment is:

    R version 4.6.0 (2026-04-24)
Platform: x86_64-pc-linux-gnu
Running under: Debian GNU/Linux 13 (trixie)
 
Matrix products: default
BLAS:   /dskh/nobackup/biostat/software/R-4.6.0/lib/libRblas.so 
LAPACK: /dskh/nobackup/biostat/software/R-4.6.0/lib/libRlapack.so;  LAPACK version 3.12.1
 
locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8   
 [6] LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C        
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
 
time zone: Australia/Sydney
tzcode source: system (glibc)
 
attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods   base     
 
other attached packages:
 [1] SparseArray_1.12.3          S4Arrays_1.12.1             abind_1.4-8                 Matrix_1.7-6               
 [5] DropletUtils_1.32.0         SingleCellExperiment_1.34.0 SummarizedExperiment_1.43.0 Biobase_2.72.0             
 [9] GenomicRanges_1.64.0        Seqinfo_1.2.0               IRanges_2.46.0              S4Vectors_0.50.3           
[13] BiocGenerics_0.58.1         generics_0.1.4              MatrixGenerics_1.24.0       matrixStats_1.5.0          
 
loaded via a namespace (and not attached):
 [1] limma_3.68.5              compiler_4.6.0            Rcpp_1.1.2                scuttle_1.22.0           
 [5] rhdf5filters_1.24.1       DelayedMatrixStats_1.34.0 parallel_4.6.0            BiocParallel_1.46.0      
 [9] statmod_1.5.2             lattice_0.23-1            XVector_0.52.0            DelayedArray_0.38.2      
[13] h5mread_1.4.1             R.utils_2.13.0            HDF5Array_1.40.0          Rhdf5lib_2.0.0           
[17] grid_4.6.0                locfit_1.5-9.12           edgeR_4.10.5              sparseMatrixStats_1.24.0 
[21] dqrng_0.4.1               rhdf5_2.56.1              R.methodsS3_1.8.2         R.oo_1.27.1              
[25] codetools_0.2-20          beachmat_2.28.0           tools_4.6.0

  9. vjcitn commented on Sep 23, 2026

    @vjcitn
    vjcitn
    on Sep 23, 2026

    Yes, I reproduced it on mac. What was annoying is that the session hangs. I do not get to the "recover on error" R traceback.

  10. hpages commented on Sep 23, 2026

    @hpages
    hpages
    on Sep 23, 2026
    Contributor

    I can also reproduce. This seems like a long standing bug in read10xCounts() where the function returns an SVT_SparseMatrix object with invalid SVT leaves. The bug is present since the mtx.class argument was introduced in BioC 3.22 because I can reproduce the problem with all versions of BioC since 3.22. Therefore I'm not sure how this could have ever worked for you.

    I suggest that you open an issue at https://github.com/MarioniLab/DropletUtils to report the problem.

    In the mean time a workaround is to "manually" fix the invalid SVT leaves of the SVT_SparseMatrix object:

    fix_SVT_leaves <- function(x) {
    stopifnot(is(x, "SVT_SparseMatrix"))
    if (is.null(x@SVT)) return(x)
    stopifnot(is.list(x@SVT))
    x@SVT <- lapply(x@SVT, function(leaf) {
        if (is.null(leaf)) return(leaf)  # empty leaf
        stopifnot(is.list(leaf), length(leaf) == 2L)
        nzoffs <- leaf[[2L]]
        if (is.null(nzoffs)) return(leaf)  # lacunar leaf
        stopifnot(is.integer(nzoffs))
        nzvals <- leaf[[1L]]
        stopifnot(length(nzvals) == length(nzoffs))
        if (length(nzoffs) == 0L) return(NULL)  # fix empty leaf
        if (is.list(nzvals)) return(leaf)
        stopifnot(is.atomic(nzvals))
        one <- SparseArray:::vector_of_ones(x@type, 1L)
        if (isTRUE(all(nzvals == one))) return(list(NULL, nzoffs))  # turn leaf into lacunar leaf
        leaf
    })
    if (all(lengths(x@SVT) == 0L))  # all leaves are NULLs
        x@SVT <- NULL
    x
}

    Then:

    counts(dataset3) <- fix_SVT_leaves(counts(dataset3))  # takes about 1 min

counts(dataset3)
# <32286 x 6794880 SparseMatrix> of type "integer" [nzcount=12695708 (0.0058%)]:
#                          [,1]       [,2]       [,3]       [,4] ... [,6794877] [,6794878] [,6794879] [,6794880]
# ENSMUSG00000051951          0          0          0          0   .          0          0          0          0
# ENSMUSG00000089699          0          0          0          0   .          0          0          0          0
# ENSMUSG00000102331          0          0          0          0   .          0          0          0          0
# ENSMUSG00000102343          0          0          0          0   .          0          0          0          0
# ENSMUSG00000025900          0          0          0          0   .          0          0          0          0
#                ...          .          .          .          .   .          .          .          .          .
# ENSMUSG00000095475          0          0          0          0   .          0          0          0          0
# ENSMUSG00000094855          0          0          0          0   .          0          0          0          0
# ENSMUSG00000095019          0          0          0          0   .          0          0          0          0
# ENSMUSG00000095041          0          0          0          0   .          0          0          0          0
#               eGFP          0          0          0          0   .          0          0          0          0

    Whoa, this is incredibly sparse!

    table(colSums(counts(dataset3)) > 500)
#   FALSE    TRUE 
# 6791109    3771

  11. hpages commented on Sep 23, 2026

    @hpages
    hpages
    on Sep 23, 2026
    Contributor

    FWIW another fix is to not specify mtx.class="SVT_SparseMatrix" to get a dgCMatrix object and to coerce to SVT_SparseMatrix:

    dataset3b <- read10xCounts("GSM8370247_Singlecell_B6_sample_2_", "B6_2", type="prefix")
counts(dataset3b) <- as(counts(dataset3b), "SVT_SparseMatrix")
type(counts(dataset3b)) <- "integer"

identical(counts(dataset3), counts(dataset3b))
# [1] TRUE

    This is easier/faster than the fix_SVT_leaves workaround but it only works if the number of nonzero values is < 2^31.

  12. LTLA commented on Sep 24, 2026

    @LTLA
    LTLA
    on Sep 24, 2026
    Contributor

    Basically the fix distills down to:

    counts(dataset)@SVT <- lapply(counts(dataset)@SVT, function(x) { 
    if (length(x[[1]]) > 0) { 
       x
    } else {
       NULL 
    }
})

    I presume that SVT_SparseMatrix expects all non-NULL leaves to be non-empty, which is alluded to in ?"SVT_SparseMatrix-class":

    Each list element is an SVT of depth 0 (as described above), or a NULL if the corresponding column is empty (i.e. has no nonzero data).

    Somehow, I had interpreted this as a suggestion that empty columns could be replaced with NULLs for efficiency, rather than an explicit requirement for correctness. So read10xCounts() wasn't going out of its way to detect empty columns and set the corresponding leaves to NULL. But no matter, this is easy enough to fix, see MarioniLab/DropletUtils#133.

    @hpages I'd request adding a line to the description of leaf vectors in ?"SVT_SparseMatrix-class" that mentions that a leaf vector must have a non-empty nzoffs and nzvals, otherwise the leaf should be NULL. I semi-regularly consult this page as a reference of what the SVT_SparseMatrix is meant to be, given that I sometimes need to construct them manually without going through dgCMatrix intermediates.

  13. hpages commented on Sep 24, 2026

    @hpages
    hpages
    on Sep 24, 2026
    Contributor

    @LTLA The full specs for SVT leaves are in src/leaf_utils.h:
    https://github.com/Bioconductor/SparseArray/blob/9a0e93056f05879a15057e3d597d0102068f09bb/src/leaf_utils.h#L10-L31

    I'm now referring the developer to this file in the man page: https://github.com/Bioconductor/SparseArray/blob/9a0e93056f05879a15057e3d597d0102068f09bb/man/SVT_SparseArray-class.Rd#L212-L213

    Note that the specs in src/leaf_utils.h also cover the lacunar leaf topic which my fix_SVT_leaves() utility above handles. In Dario's case, this contributes to a 20% additional memory footprint reduction after the 70% reduction obtained by replacing empty leaves with NULLs.

  14. hpages commented on Sep 24, 2026

    @hpages
    hpages
    on Sep 24, 2026
    Contributor

    Back to the original issue.

    @DarioS Does the combineCols(..., delayed=FALSE) suggestion work for you? Note that the fill argument is set to NA by default. However, depending on what you're going to do downstream with the combined object, you might want to consider setting fill to 0L. It should speed up things even more and further reduce the memory footprint of the combined object.

    Using fill=NA (the default):

    system.time(combined_counts_B <- assay(combineCols(SE1, SE2, SE3, SE4, fill=NA, delayed=FALSE)))
#    user  system elapsed 
#   0.734   0.301   1.035 

type(combined_counts_B)
# [1] "integer"

object.size(combined_counts_B)
# 1748384784 bytes

    Using fill=0L:

    ## IMPORTANT: Make sure to use 0L and not 0 so the combined object is still of type "integer". Using
## 0 would change its type to "double", which would significantly (and unnecessarily) increase its
## memory footprint.
system.time(combined_counts_B0 <- assay(combineCols(SE1, SE2, SE3, SE4, fill=0L, delayed=FALSE)))
#    user  system elapsed 
#   0.393   0.009   0.401 

type(combined_counts_B0)
# [1] "integer"

object.size(combined_counts_B0)
# 84833400 bytes

    So in this case using fill=0L reduces memory footprint by 20x (YMMV).

    But again, it all depends on what you're going to do downstream.

  15. DarioS commented on Sep 24, 2026

    @DarioS
    DarioS
    on Sep 24, 2026
    Author

    It works.

  16. DarioS
    closed this as completedon Sep 24, 2026

  17. hpages commented on Sep 24, 2026

    @hpages
    hpages
    on Sep 24, 2026
    Contributor

    you're welcome

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions