Memory consumption issue

[Colin Clark]

Hey Giovanni and Antranig,

Just to clarify, gathering the numbers is still hugely valuable and worth doing in the short term so that we can make general comparisons with the much less complex and less feature-rich version of the Preferences Server we last tested several years ago. Keep going!

In the medium term, I'm midway through reviewing Antranig's big pull request with major improvements in memory management and invoker performance. He's already in the midst of upgrading the GPII's base infrastructure (Kettle) to this version of the framework. From there, we'll need to find a safe time to upgrade all the GPII realtime framework components.

Once that's all done, we'll do another round of testing and start to address targeted issues. So it will help a lot, Giovanni, if you're able to automate the process so that other developers such as Antranig can also run the performance tests when they're squashing these kinds of bugs.

Seem reasonable?

Colin


> On Jul 3, 2015, at 11:40 AM, Antranig Basman <antranig.basman at colorado.edu> wrote:
> 
> Thanks for the investigation, Giovanni. This is very useful data. We're currently in the middle of upgrading all the base libraries for the GPII, before which it isn't worth doing much profiling work. We expect the memory leakage issues to ease off a bit, and the resulting simplification of the code should make it easier to hunt down the remaining leak issues later in the year.
> 
> Cheers,
> 
> Antranig.
> 
> On 03/07/2015 02:44, Tirloni, Giovanni wrote:
>> Hello,
>> 
>>  While running load tests on the preferences server (for benchmark and
>>  capacity planning purposes), the memory consumption of the node
>>  processes started to climb very quickly. To work around it and keep the
>>  tests going, we were restarting the processes when they reached
>>  100-200MB (which coincided with CPU usage nearing 85-95%). At some
>>  point the monitoring script was restarting processes every 10 seconds
>>  and it did not seem worth it anymore. This was done using the latest
>>  code from gpii/universal (2015-07-01), the dependencies specified in
>>  the package.json file and node 0.10.36.
>> 
>>  I deployed the same code on my desktop, primed the CouchDB database
>>  with the testData and tried to reproduce the issue with a simple test
>>  (curl http://localhost:8081/preferences/elaine). I ran a few tests,
>>  varying the wait time between the requests to increase the pressure, as
>>  shown below. You can see in the graph a few 30-second plateaus between
>>  tests. I then left the preferences server sitting idle for 6 minutes,
>>  and the memory usage stayed flat at around 700MB, there were no
>>  connections active at that time. The attached file contains the CSV
>>  data (in KB) and the generated chart.
>> 
>> 19:55:30 - 19:56:30 - wait 500ms
>> 19:57:00 - 19:58:00 - wait 250ms
>> 19:58:30 - 19:59:30 - wait 100ms
>> 20:00:00 - 20:01:00 - wait  50ms
>> 20:01:30 - 20:02:30 - wait  25ms
>> 20:03:00 - 20:05:00 - wait  10ms
>> 20:05:00 - 20:11:00 - no activity
>> 
>>  I tried my hand at tracing/profiling the code but unfortunately I'm
>>  closer to a JS newbie than an expert and I couldn't discover anything
>>  Earth shattering. I did notice the number of  array/string/closure
>>  objects accounted for most of the memory used. I also collected heap
>>  snapshots using the 'heapdump' library (three samples available here:
>>  http://1drv.ms/1f6Rmfp)
>> 
>> Regards,
>> Giovanni
>> 
>> 
>> 
>> _______________________________________________________
>> fluid-work mailing list - fluid-work at lists.idrc.ocadu.ca
>> To unsubscribe, change settings or access archives,
>> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
>> 
> 
> _______________________________________________________
> fluid-work mailing list - fluid-work at lists.idrc.ocadu.ca
> To unsubscribe, change settings or access archives,
> see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work