threading.run without call stack

[rkooo567]

Firstly, thanks for the amazing tool!

I am using memray for profiling one of components in ray https://github.com/ray-project/ray/. However, as you can see from the right side of the image, there's no further call stack after self._target (and the memory leak seems to happen from here).

Does anyone know why there's no information for this thread? I tried --split-threads

[pablogsal]

Thanks for opening this discussion!

Could you tell us how are you running/profiling your application (reproducers steps) so we can take a look?

[pablogsal]

It may be some interesting problem here but is also possible that nothing under the thread allocates memory. Check out https://bloomberg.github.io/memray/python_allocators.html to understand how Python allocates memory and how that can impact the results. You can also try to run with —trace-python-allocators or the PYTHONMALLOC=malloc environment variable set to force allocations to not use the pymalloc pools.

[godlygeek]

What you're showing is exactly what I'd expect to see if the thread's target callable is implemented in C, as opposed to in Python. If that's the case, running in native mode using --native would give you more information.

[rkooo567]

@godlygeek yeah I think you are right. It seems to be from a thread started by a gRPC server (which uses native code). Unfortunately, we couldn't still see the exact code that caused it after I used --native (I guess the gRPC wheel doesn't contain symbols to print..).

@pablogsal if you are interested, I can provide a script. But it's up to you!

[pablogsal]

If you feel brave you can try to compile the gRPC wheel from source using the —no-binary option in PIP together with “CFLAGS=-g3 -Og” although that may require a ton of binary dependencies and some ancient chants 😅

[rkooo567]

Will try that out.. Btw, I also found the result reported from the profiler is a bit different from htop result. Do you know when this can happen?

[rkooo567]

In the htop, it says RES is 895MB, but the profiler says the total allocation is 109.8MB. Is it because 895MB is already allocated and Pyallocator reuses the precreated pool?

[pablogsal]

There may be multiple explanations but the most likely ones is that either malloc or pymalloc are not releasing the memory due to fragmentation or sone similar effect involving allocator hysteresis. Memray tracks heap usage (memory requested - memory released) but this doesn’t need to match the resident size.

On the top of the flamegraph you should see two lines (you can click to enlarge the plot). One line is heap size over time and the other is resident size. As you can see in your image they differ over time.

To rule out pymalloc you can run with the “PYTHONMALLOC=malloc” environment variable set and see if still happens. In that case that would point to malloc holding the memory.

[rkooo567]

Gotcha. That makes sense! Thank you so much for the quick answer :)!!