OpenNebula

Hi Boris,

I am not able to reproduce this, in order to speed up tests I've reduce the cache of PoolObjects to 150, in PoolSql.cc,

const unsigned int PoolSQL::MAX_POOL_SIZE = 15000;   

This is the max number of objects in memory. I've done several runs with 200, 500, and 1000 onevm shows through valgrind, and I get exactly the same numbers each time:

                                                                             
==20501== HEAP SUMMARY:                                                               
==20501==     in use at exit: 78,592 bytes in 17 blocks                               
==20501==   total heap usage: 380,892 allocs, 380,875 frees, 74,788,992 bytes allocated 

==23456== HEAP SUMMARY:                                                               
==23456==     in use at exit: 78,592 bytes in 17 blocks                               
==23456==   total heap usage: 901,107 allocs, 901,090 frees, 175,591,059 bytes allocated 

==26857== HEAP SUMMARY:                                                               
==26857==     in use at exit: 78,592 bytes in 17 blocks                               
==26857==   total heap usage: 1,617,203 allocs, 1,617,186 frees, 314,881,092 bytes allocated  

As you can see there are the same amount of lost bytes (because of xmlrpc library and static initialization of some pool mutex).

I've also tried for different cache sizes (50, 100, 150 and 200) objects and the memory also gets bound to a maximum no matter how many VMs are shown (tried x10 times cache of size)...

So:
- For 15K cache size, we expect no more than 800MB od RSS. Could you check oned evolution, e.g. ps -v -p `pgrep oned` and/or car /proc/`pgrep oned`/status

- Could it be other process in the server?

- If using ruby, maybe the client side is also eating a lot memory, you can try to use the shell, that terminates each ruby process each time

for i in `seq 0 30000` ; do echo $i; onevm show $i > /dev/null ; done

So far I cannot reproduce this

Hi Ruben,

thanks for taking the time to investigate this. I will take your advice with useful pointers and do a few local tests. I will get back to you with results ASAP.

Hi Ruben,

I have an update for you. I now believe this is not a memory leak, just strange caching behavior. I'm not sure why, but it looks like oned's cache is filling up faster than it can purge/replace old cache entries (or the entries themselves grow in size). I have a sanitized DB dump, several GBs in size. If you are up for it, I can send it to you via a private channel and it should help you replicate this issue in your testbed with reasonable effort.

Thanks!

Or we could give you access to a running testbed. If that would be easier for you.

Sorry Boris, Yes I'm interested in it. If you send me a link (rsmontero at opennebula dot org) I'll download the DB and take a look. Sorry for the late response, the 5.0 release took all my cycles ;)

Ruben S. Montero wrote:

Sorry Boris, Yes I'm interested in it. If you send me a link (rsmontero at opennebula dot org) I'll download the DB and take a look. Sorry for the late response, the 5.0 release took all my cycles ;)

Done. Thank you!