valgrind possible memory leaks from parallelFor

[MurrayEfford]

From a closed issue 3 years ago, I get the impression that valgrind 'possible memory leaks' are a routine and unavoidable result of using parallelFor. Is that the case?

I can live with that, but it makes for awkward relations with CRAN core (my revision 2.2.1 of package openCR that shows such leaks has just been accepted on the condition I try to sort it out; triggered when valgrind revealed a more serious bug, now fixed).

I would appreciate an update from those in the know.

Murray

[kevinushey]

What are the valgrind errors that you are seeing?

[MurrayEfford]

Heres one small example. 'Possibly lost' can extend to >100,000 bytes. Occurs (differently) with both TBB and tinythread. ==2807542== ==2807542== HEAP SUMMARY: ==2807542== in use at exit: 324,178,780 bytes in 59,954 blocks ==2807542== total heap usage: 323,722 allocs, 263,768 frees, 1,691,928,852 bytes allocated ==2807542== ==2807542== 2,240 bytes in 7 blocks are possibly lost in loss record 531 of 3,693 ==2807542== at 0x483BAE9: calloc (vg_replace_malloc.c:760) ==2807542== by 0x40139FB: _dl_allocate_tls (in /usr/lib64/ld-2.32.so) ==2807542== by 0x4D8714C: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.32.so) ==2807542== by 0x23FD54B8: launch (thread_monitor.h:218) ==2807542== by 0x23FD54B8: wake_or_launch (private_server.cpp:297) ==2807542== by 0x23FD54B8: tbb::internal::rml::private_server::wake_some(int) (private_server.cpp:395) ==2807542== by 0x23D4B705: spawn (task.h:1046) ==2807542== by 0x23D4B705: offer_work (parallel_for.h:118) ==2807542== by 0x23D4B705: execute<tbb::interface9::internal::start_for<tbb::blocked_range<long unsigned int>, RcppParallel::(anonymous namespace)::TBBWorker, const tbb::auto_partitioner>, tbb::blocked_range<long unsigned int> > (partitioner.h:249) ==2807542== by 0x23D4B705: tbb::interface9::internal::start_for<tbb::blocked_range<unsigned long>, RcppParallel::(anonymous namespace)::TBBWorker, tbb::auto_partitioner const>::execute() (parallel_for.h:139) ==2807542== by 0x23FE0374: tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) (custom_scheduler.h:517) ==2807542== by 0x23FDDDBF: tbb::internal::generic_scheduler::local_spawn_root_and_wait(tbb::task*, tbb::task*&) (scheduler.cpp:699) ==2807542== by 0x23D4B146: spawn_root_and_wait (task.h:779) ==2807542== by 0x23D4B146: run (parallel_for.h:92) ==2807542== by 0x23D4B146: parallel_for<tbb::blocked_range<long unsigned int>, RcppParallel::(anonymous namespace)::TBBWorker> (parallel_for.h:198) ==2807542== by 0x23D4B146: operator() (TBB.h:77) ==2807542== by 0x23D4B146: internal_run_and_wait<const RcppParallel::(anonymous namespace)::TBBParallelForExecutor> (task_group.h:104) ==2807542== by 0x23D4B146: run_and_wait<RcppParallel::(anonymous namespace)::TBBParallelForExecutor> (task_group.h:209) ==2807542== by 0x23D4B146: operator() (TBB.h:142) ==2807542== by 0x23D4B146: tbb::interface7::internal::delegated_function<RcppParallel::(anonymous namespace)::TBBArenaParallelForExecutor, void>::operator()() const (task_arena.h:90) ==2807542== by 0x23FDB9D3: tbb::interface7::internal::task_arena_base::internal_execute(tbb::interface7::internal::delegate_base&) const (arena.cpp:943) ==2807542== by 0x23D4F2D1: execute_impl<void, RcppParallel::(anonymous namespace)::TBBArenaParallelForExecutor> (task_arena.h:213) ==2807542== by 0x23D4F2D1: execute<RcppParallel::(anonymous namespace)::TBBArenaParallelForExecutor> (task_arena.h:344) ==2807542== by 0x23D4F2D1: RcppParallel::tbbParallelFor(unsigned long, unsigned long, RcppParallel::Worker&, unsigned long, int) (TBB.h:239) ==2807542== by 0x23D703C0: parallelFor (RcppParallel.h:42) ==2807542== by 0x23D703C0: allhistparallelcpp(int, int, int, int, int, int, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<13, Rcpp::PreserveStorage>, Rcpp::Vector<13, Rcpp::PreserveStorage>, Rcpp::Vector<13, Rcpp::PreserveStorage>, Rcpp::Vector<13, Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>, Rcpp::Vector<13, Rcpp::PreserveStorage>, Rcpp::Vector<13, Rcpp::PreserveStorage>) (prwiparallel.cpp:164) ==2807542== by 0x23D5705F: _openCR_allhistparallelcpp (RcppExports.cpp:240) ==2807542== ==2807542== LEAK SUMMARY: ==2807542== definitely lost: 0 bytes in 0 blocks ==2807542== indirectly lost: 0 bytes in 0 blocks ==2807542== possibly lost: 2,240 bytes in 7 blocks ==2807542== still reachable: 324,176,540 bytes in 59,947 blocks ==2807542== of which reachable via heuristic: ==2807542== newarray : 4,264 bytes in 1 blocks ==2807542== suppressed: 0 bytes in 0 blocks ==2807542== Reachable blocks (those to which a pointer was found) are not shown. ==2807542== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==2807542== ==2807542== For lists of detected and suppressed errors, rerun with: -s ==2807542== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

…

________________________________________ From: Kevin Ushey ***@***.***> Sent: 11 November 2021 10:25 To: RcppCore/RcppParallel Cc: Murray Efford; Author Subject: Re: [RcppCore/RcppParallel] valgrind possible memory leaks from parallelFor (Issue #175) What are the valgrind errors that you are seeing? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FRcppCore%2FRcppParallel%2Fissues%2F175%23issuecomment-965758070&data=04%7C01%7Cmurray.efford%40otago.ac.nz%7C262d3e58ba504c684f4b08d9a490ad1f%7C0225efc578fe4928b1579ef24809e9ba%7C1%7C0%7C637721763564960484%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tmNFcqE1Q0cZhhdpWE09pUVxa9rnO%2Fmg24kXT2Rs3R0%3D&reserved=0>, or unsubscribe<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACEB6PVXZPWJOPEYGLOD3GTULLPN5ANCNFSM5HY4EWAQ&data=04%7C01%7Cmurray.efford%40otago.ac.nz%7C262d3e58ba504c684f4b08d9a490ad1f%7C0225efc578fe4928b1579ef24809e9ba%7C1%7C0%7C637721763564970446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=HNWgfzeWWokVm4JjD1UFqH%2FScSOoZQbBbwSafNlzvBA%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cmurray.efford%40otago.ac.nz%7C262d3e58ba504c684f4b08d9a490ad1f%7C0225efc578fe4928b1579ef24809e9ba%7C1%7C0%7C637721763564970446%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aMY8%2F49kDV4x7KURo9ZFnxnc2QCIX%2FEGunI2T6oR9HA%3D&reserved=0> or Android<https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cmurray.efford%40otago.ac.nz%7C262d3e58ba504c684f4b08d9a490ad1f%7C0225efc578fe4928b1579ef24809e9ba%7C1%7C0%7C637721763564975425%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=CoTnw0CD99YTo61jbXy1mmYNCumIRNPC%2FRI91v%2Fdi6Q%3D&reserved=0>.

[kevinushey]

Occurs (differently) with both TBB and tinythread.

The fact that I haven't heard any other reports of memory leaks from other client packages of RcppParallel makes me think this could be an issue in openCR. It's also quite surprising to me that the leaks are reported when using tinythread as well.

==2807542== 2,240 bytes in 7 blocks are possibly lost in loss record 531 of 3,693
==2807542==    at 0x483BAE9: calloc (vg_replace_malloc.c:760)
==2807542==    by 0x40139FB: _dl_allocate_tls (in /usr/lib64/ld-2.32.so)

See https://stackoverflow.com/questions/5610677/valgrind-memory-leak-errors-when-using-pthread-create for a similar example.

My understanding of this output is that a small amount of thread-local memory might've been "leaked"; more likely is that some TBB threads were still alive even when the process shut down. This ultimately seems like a false positive to me (not that CRAN would be convinced by that)

[eddelbuettel]

(Side comment from a casual user of valgrind / another package provider: the

==2807542== LEAK SUMMARY:
==2807542==    definitely lost: 0 bytes in 0 blocks
==2807542==    indirectly lost: 0 bytes in 0 blocks
==2807542==      possibly lost: 2,240 bytes in 7 blocks

is actually a good outcome as "possibly lost" is simply something that can happen in a language like R that handles its own gc(). But you should follow the hint about

To see them, rerun with: --leak-check=full --show-leak-kinds=all

which I often found helpful.)

[MurrayEfford]

Thanks for your tips. Seems I cannot simply dismiss this as 'known behaviour' and will need to work on it.

P.S. For the record, here is a tinythread example as reported back to me by CRAN, and the specific reason they (Uwe) ask me to 'fix it soon':

==191526== 286,192 bytes in 577 blocks are possibly lost in loss record
9,846 of 10,047
==191526==    at 0x483CAE9: calloc
(/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:760)
==191526==    by 0x40145BA: _dl_allocate_tls (in /usr/lib64/ld-2.31.so)
==191526==    by 0x53F912E: pthread_create@@GLIBC_2.2.5 (in
/usr/lib64/libpthread-2.31.so)
==191526==    by 0x55D85C40: thread
(R-devel/site-library/RcppParallel/include/tthread/tinythread.h:901)
==191526==    by 0x55D85C40: RcppParallel::ttParallelFor(unsigned long,
unsigned long, RcppParallel::Worker&, unsigned long)
(R-devel/site-library/RcppParallel/include/RcppParallel/TinyThread.h:122)
==191526==    by 0x55DA5220: parallelFor
(R-devel/site-library/RcppParallel/include/RcppParallel.h:44)
==191526==    by 0x55DA5220: allhistparallelcpp(int, int, int, int, int,
int, Rcpp::Vector<14, Rcpp::PreserveStorage>, Rcpp::Vector<13,
Rcpp::PreserveStorage>, Rcpp::Vector<13, Rcpp::PreserveStorage>,
Rcpp::Vector<13, Rcpp::PreserveStorage>, Rcpp::Vector<13,
Rcpp::PreserveStorage>, Rcpp::Matrix<14, Rcpp::PreserveStorage>,
Rcpp::Vector<13, Rcpp::PreserveStorage>, Rcpp::Vector<13,
Rcpp::PreserveStorage>)
(/tmp/openCR.Rcheck/00_pkg_src/openCR/src/prwiparallel.cpp:164)
==191526==    by 0x55D8BF4F: _openCR_allhistparallelcpp
(/tmp/openCR.Rcheck/00_pkg_src/openCR/src/RcppExports.cpp:240)

[eddelbuettel]

That may be missing the relevant lines just above it: "possibly lost" is from the department of "nobody really knows".

[kevinushey]

These still seem relevant:

https://stackoverflow.com/questions/5610677/valgrind-memory-leak-errors-when-using-pthread-create
https://stackoverflow.com/questions/54435972/pthread-created-followed-by-pthread-join-valgrind-possible-loss-c

My guess is that pthread is creating or managing some static buffer of data used by threads, and valgrind is reporting that memory is "maybe" lost. But it's just a small static buffer, so it doesn't actually matter in the end.

I personally don't think it's fair that every package using RcppParallel gets dinged on CRAN for what is almost surely a valgrind false positive.

[MurrayEfford]

I'm glad the concensus (Kevin!) seems to be heading to 'false positive'.

To be fair, the original 'Additional issue' arose when valgrind found a genuine bug of mine that had gone unnoticed for years. The revelation of 'possible' memory leaks was collateral, and Uwe Ligges let the package through even though he could see them. Most likely, 'possible memory leaks' alone do not trigger an 'Additional issue', but I do live in fear that Ripley will revisit the test and take a harder line.

[kevinushey]

The thing that confuses me is that, presumedly, I would be able to reproduce a similar issue in RcppParallel's own tests. Here's what I see on Ubuntu 20.04:

kevin@ubuntu:~/scratch
$ R -d valgrind --debugger-args='-s --leak-check=full' --vanilla -s -e 'tools:::testInstalledPackage("RcppParallel")'
==204661== Memcheck, a memory error detector
==204661== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==204661== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==204661== Command: /usr/lib/R/bin/exec/R --vanilla -s -e tools:::testInstalledPackage("RcppParallel")
==204661== 
Testing examples for package ‘RcppParallel’
  comparing ‘RcppParallel-Ex.Rout’ to ‘RcppParallel-Ex.Rout.prev’ ... OK
Running specific tests for package ‘RcppParallel’
  Running ‘doRUnit.R’
  Running ‘runit.distance.R’
==204661== 
==204661== HEAP SUMMARY:
==204661==     in use at exit: 48,636,905 bytes in 10,548 blocks
==204661==   total heap usage: 24,359 allocs, 13,811 frees, 74,118,629 bytes allocated
==204661== 
==204661== LEAK SUMMARY:
==204661==    definitely lost: 0 bytes in 0 blocks
==204661==    indirectly lost: 0 bytes in 0 blocks
==204661==      possibly lost: 0 bytes in 0 blocks
==204661==    still reachable: 48,636,905 bytes in 10,548 blocks
==204661==                       of which reachable via heuristic:
==204661==                         newarray           : 4,264 bytes in 1 blocks
==204661==         suppressed: 0 bytes in 0 blocks
==204661== Reachable blocks (those to which a pointer was found) are not shown.
==204661== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==204661== 
==204661== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

but everything seems fine, even though I can reproduce the valgrind issues with your package. So there's something being done specifically in your package that's triggering the valgrind warning, even though (as far as I can see) it should in fact be a false positive.

[kevinushey]

I can reproduce something similar if I create a bunch of threads and then forget to join them (manually using the pthread APIs) but I can't see how this could happen when using RcppParallel... unless somehow R / RcppParallel is getting interrupted and not allowing some of those threads to complete and become joined?

[kevinushey]

Is it possible that your parallel worker threads are trying to call Rcpp::stop() when an error condition occurs?

[MurrayEfford]

Not directly from my C++ code (I've searched for 'stop'; all occurrences in 'openCR' are commented out and all occurrences in 'secr' are outside multithreaded code).

[MurrayEfford]

Minor correction: there was one stray Rcpp::stop() inside multithreaded code in 'secr', but fixing that did not fix the 'possible memory leak'.

[MurrayEfford]

Hoping to find something specific, I built a small test package with the stripped down C++ code below. Valgrind on Linux shows a 'possible' leak of 320 bytes with TBB and 7360 bytes with tinythread. Is there a problem in my coding?

#include <Rcpp.h>
#include <RcppParallel.h>
struct Hckm : public RcppParallel::Worker {
    RcppParallel::RMatrix<double> gsbval;   // input data
    RcppParallel::RMatrix<double> dist2;    // input data
    RcppParallel::RVector<double> gk;       // output vector to write to
    Hckm(
        Rcpp::NumericMatrix gsbval, 
        Rcpp::NumericMatrix dist2, 
        Rcpp::NumericVector gk
        ): gsbval(gsbval), dist2(dist2), gk(gk) {}
    double zLcpp (int c, int k, int m)
    {
        return (gsbval(c,0) * std::exp(-dist2(k,m) / 2 / gsbval(c,1)/ gsbval(c,1)));
    }    
    void operator()(std::size_t begin, std::size_t end) {
        int cc = gsbval.nrow();
        int kk = dist2.nrow();
        int c, gi,k;
        for (std::size_t m = begin; m < end; m++) {
            for (k=0; k < kk; k++) {
                for (c=0; c < cc; c++) {
                    gi = cc * (kk * m + k) + c;  
                    gk[gi] = zLcpp(c, k, m);
                }
            }
        }
    }
};

// [[Rcpp::export]]
Rcpp::NumericVector makegkPointcpp (
        int grain,
        int ncores,
        Rcpp::NumericMatrix &gsbval, 
        Rcpp::NumericMatrix &dist2
) 
{
    Rcpp::NumericVector gk(gsbval.nrow() * dist2.size()); 
    Hckm hckm (gsbval, dist2, gk);
    if (ncores>1) {
        RcppParallel::parallelFor(0, dist2.ncol(), hckm, grain, ncores);
    }
    else {
        hckm.operator()(0,dist2.ncol());  // for debugging, allows R calls
    }
    return gk;
}

//-------------------------------------------------------------------
# called from R with
testfn <- function(grain = 1, ncores = 2, parameters = c(0.5, 20)) 
{
    distmat2 <- as.matrix(dist(matrix(seq(20,1000,20), ncol=2)))
    parametermatrix <- matrix(parameters, nrow = 1)
    gk <- makegkPointcpp (
        as.integer(grain), 
        as.integer(ncores),
        parametermatrix, 
        distmat2
        )
    sum(gk)
}
testfn()

sessionInfo()

R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Fedora 33 (Thirty Three)
Matrix products: default
BLAS/LAPACK: /usr/lib64/libopenblas-r0.3.12.so
locale:
[1] LC_CTYPE=en_NZ.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_NZ.UTF-8 LC_COLLATE=en_NZ.UTF-8
[5] LC_MONETARY=en_NZ.UTF-8 LC_MESSAGES=en_NZ.UTF-8
[7] LC_PAPER=en_NZ.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_NZ.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] leaktest_0.0.13
loaded via a namespace (and not attached):
[1] compiler_4.0.3 Rcpp_1.0.7 RcppParallel_5.1.4

[MurrayEfford]

See https://github.com/MurrayEfford/leaktest for the code embedded in a mini package, with commands to test on Linux