Possible Memory Leak

Posted by: justin-dd on 3 August 2017, 6:09 am EST

  • Posted 3 August 2017, 6:09 am EST

    In more testing leading to a final release, we where trying to run large datasets to see how it performs.



    We noticed however that the memory consumption in the w3wp.exe goes up with each report run and that memory never seems to be released.



    I have attached a sample report with a single column dataset with 70000 rows. This is output to a simple report with a table.



    Using the web viewer the first run consumes over 1 GB ram! Subsequent runs consume about 250MB ram. Refreshing the web viewer a few times will quickly exhaust the 4GB of ram on my development machine.



    As far as I can tell the memory is never released, so long as the worker process is not recycled the ram usage will never go down, as though the report runs are not being garbage collected.



    I tried using the VaryBy property of the viewer to adjust caching, but it seemed to have no effect, and it does not have a setting to turn off caching completely.



    The Winform viewer seems to suffer from this too, simply loading the rdlx up and refreshing the report will pretty quickly exhaust my workstation ram. Interestingly enough the Winform viewer is actually MUCH slower at rendering the report than the web viewer!



    Now this type of data set is not typical for our users, but sometimes they may need to dump large amount of data into Excel for post analysis. However I believe the issue exist on all size datasets, and in a production install a higher volume of smaller datasets will quickly exhaust web server ram.



    The sample I have attached has a dataset serialized to xml that it uses as a datasource, but the issue occurs with a sqlserver data source as well.



    Is there anything more I need to do to release memory?


    2010/06/DDReports Mem Leak Sample.zip
  • Replied 3 August 2017, 6:09 am EST

    Any status update?



    Where you able to reproduce the memory usage issue there?
  • Replied 3 August 2017, 6:09 am EST

    Justin,



    Thanks for such a detailed explanation of the issue.



    I am currently researching further on this and will post my observations soon.
  • Replied 3 August 2017, 6:09 am EST

    Hello,



    We have looked further into this on couple of machines (Windows 2003 and Windows 2008 Server) by running your sample and the maximum memory usage I am seeing is around 310 MB on both the machines. I have used a Memory profiler to test this issue.



    Which memory profiler are you using to see this memory leak?

    Which Operating system is installed on the machine?

    I have tested this issue with the latest version (1.6.1871.24). Which version is installed on your machine?
  • Replied 3 August 2017, 6:09 am EST

    Using the latest version 1.6.1871.24.



    Window 7 64bit 4gigs of ram.



    I am working on a memory profile in VS2010 but it is taking some time.



    Perhaps the web viewer example wasn't the best example to post, I am putting together a simple winform that simply loads the report and does a ReportRefresh in the RenderingCompleted event. You can simply let it run and it will exhaust all memory on the machine after a few runs (about ~450MB used per run).



    Until I get the memory profile here is a screenshot of Task Manager after refreshing the report 3 times in your pre-built Report Previewer.



    You should be able to replicate by just loading the RDLX in the previewer and refreshing it.

  • Replied 3 August 2017, 6:09 am EST

    Another thing to note if testing the webviewer in IIS is that I believe by default IIS will recycle the app pool if the process uses more than a certain percentage of main memory.



    This will appear to free the memory, which it is, but only because the whole process is shutdown and a new worker is spun up in its place.



    You have to make sure the process ID is not changing when profile the memory in that situation.



    Of course the eaiser thing to do is run the report in the winform viewer, as the process is static and under the apps control.
  • Replied 3 August 2017, 6:09 am EST

    Justin,



    (1) I ran the report with the steps you have provided on Windows 7 64 bit machine (2GB RAM) and the ReportViewer takes maximum 345 MB and on subsequent refresh of the report, the memory usage more or less remains constant. I will wait for the sample windows forms application from your side to debug this further.



    (2) You are correct and we were not using a different process ID however using the same process ID all the time.
  • Replied 3 August 2017, 6:09 am EST

    <font face="Tahoma" size="2">Hello,
    I start working on the memory problem immediately. I'll constantly post the updates in this thread.
    </font>
  • Replied 3 August 2017, 6:09 am EST

    Here is a screenshot of the heap summary, showing most bytes are in the Gen 2 heap.

  • Replied 3 August 2017, 6:09 am EST

    Here is a screenshot of ANTS Summary for completeness, notice on the timeline you can see the report run boundaries and how heap size just keeps growing.

  • Replied 3 August 2017, 6:09 am EST

    Attached is a self contained Winform solution to reproduce the issue.



    You can simply run the executable in the bin folder, or rebuild.



    It runs the report with dataset I provided before over and over.



    On all machines tested here so far, Win 7 64bit and win 2003 32bit, this application will eventually consume all available memory and crash with and OutOfMemoryException.



    I will post a csv dump of a RedGate ANTS memory profile of 5 report runs.
    2010/07/DD Mem Leak Test Winform.zip
  • Replied 3 August 2017, 6:09 am EST

    Here is the ANTS Memory Profile in CSV format, it look like most of the memory is used by a class name #Z2e which is some kind of expression evaluation form the report engine.
    2010/07/DD Memory profile.zip
  • Replied 3 August 2017, 6:09 am EST

    <font face="Tahoma" size="2">Hello,
    I've sent you the email that contains the download link for the build that has the fix for the memory leak issue on web-viewer.
    Could you please confirm that you received the e-mail?
    </font>
  • Replied 3 August 2017, 6:09 am EST

    I have been running test with the new build and it seems to properly garbage collect memory after refreshing the web report viewer.



    Ram usage on this report seems to hold around 320-500 megs of ram in the asp.net development web server and around 1 Gigs of ram running on IIS7 in its own application pool.



    So this seems to have resolved releasing memory between runs.



    We are still somewhat concerned with the amount of memory used for just one run, as the serialized data set is only 5MB, yet ends up being 100-200X that as a rendered report in ram. This seems excessive and if 3-4 users ran this report at once it could result in an out of memory exception on the web server.



    Also as you already know the Winform viewer still suffers from this problem, but our application currently only uses the web viewer.



    I believe this is satisfactory for our needs currently although less ram usage per run would be very desirable for larger datasets.



    I also noticed that exporting to Word or Excel doesn't seem to work with this large of a data set either, I think this may be a separate issue and may have always been this way, I received as stack overflow exception exporting to Excel format after it ran for a while, and Word export never returned anything.



    Thanks for the quick response.





  • Replied 3 August 2017, 6:09 am EST

    Please also send me the link which contains the fix for memory leak. I will see if my issue is solved with this fix.



    thanks,

    Wind
  • Replied 3 August 2017, 6:09 am EST

    <font face="Tahoma" size="2">Hello,
    I've sent you the download link via email.
    </font>
  • Replied 3 August 2017, 6:09 am EST

    Hi <span class="inlineLink">Sergey</span><span class="txt2"></span>,

    I ran a report which should have more than 20,000 pages with DDR 1.6.1871.41. It crashed after one and half hours with 9621 pages rendered. Here is the picture I captured. Thanks!

    <img src="file:///F:/2.JPG" alt=""><img src="file:///F:/1.JPG" alt="">

    2010/07/2.JPG
  • Replied 3 August 2017, 6:09 am EST

    <font face="Tahoma" size="2">Update: we are aggressively working on the fix. It is not 1 line improvement of the code, so we have not complete the fix yet. I'll keep the post updated.
    </font>
  • Replied 3 August 2017, 6:09 am EST

    Hello,



    Case 147883 (Refreshing WebReportViewer can causes memory leak) and Case 147874 (Rendering multiple reports in Viewer causes memory leak) has been fixed in the latest build (1.6.1871.45) of DataDynamics Reports.



    You may download this build from here:

    ftp://ftp.fpoint.com/ActiveReports/builds/1.6.1871.45.zip
  • Replied 3 August 2017, 6:09 am EST

    I just want to let everyone know what is going on. Right now we are looking into the memory consumption of the web viewer, we will be gathering some data on what is making up the bulk of the memory usage so we can decide how to best approach this issue.



    James
  • Replied 3 August 2017, 6:09 am EST

    "It is quite possible that such a large report - 20,000 pages - is simply going to require more memory than your machine has."



    I would say this is most likely the case.



    Again in my sample I only had 70,000 records with a single column which ended up being 1892 pages.



    This dataset is only 5 MB serialized as bloated XML, yet in the report runtime ends up using 300-500 MB of ram when rendered to a simple table with no borders even!



    So I think bottom line, DD Reports is currently not a good reporting package when dealing with large datasets, that generate many pages of output because of it excessive ram usage in that scenario.



    I believe there is probably quite a bit of room for memory optimization in there though, I know we came from using Active Reports ActiveX version and it could easily generate 1800 pages of more complex output with much lower memory usage.



    I think if I where to group the large dataset and say do a graph with it, it uses very little ram, so it seems to be the number of controls rendered not the number of rows in the dataset that matters very much.



    This is ok for what we are using it for so far, but if you need many rows of output you should be aware of its memory usage especially in a shared resource scenario like the web viewer in a web site with potentially many concurrent users.
  • Replied 3 August 2017, 6:09 am EST

    blett,



    Add GDI objects to the Task Manger and see if the app fails at 10K. 10K is the limit that the OS can handle. At least it used to be...
  • Replied 3 August 2017, 6:09 am EST

    It is quite possible that such a large report - 20,000 pages - is simply going to require more memory than your machine has. Just to consider a purely hypothetical example, if each page had a 500KB image on it, a 20,000 page report could consume 10GB of memory. Now, obviously this is just a contrived example and in your situation there may be a genuine problem. If you suspect a problem please reply with more information. Ideally the report attached and some sample data will enable us to investigate most effectively.

  • Replied 3 August 2017, 6:09 am EST

    Just to clarify in my testing ram usage is similar in both the web and winform viewer, so it seems related to the core engine not which viewer is being used.



    It is just exacerbated in the webviewer because more than one could be generated concurrently by different users, vs the winform which is usually just a single report at a time on a workstation.



    Also targeting x86 vs any cpu seems to make a noticeable reduction in memory usage, I would guess because you are creating a whole lot of instances of something and the pointers are 64bits instead of 32bits on any cpu on a 64bit machine.
  • Replied 3 August 2017, 6:09 am EST

    Just to keep everything related to one thread in the same thread, I have replied to your post here:

    http://www.datadynamics.com/forums/134675/ShowPost.aspx
  • Replied 3 August 2017, 6:09 am EST

    Reproduced out of memory with the latest build (1.6.1871.45) today with a large amount of data. BTW, is DDR putting all of the data queried from DB to the memory? Is it possible only to put some and re-run the query according to selection?
  • Replied 14 May 2018, 6:18 am EST

    Has there been any update on this item? I'm also using a webviewer (HTML) and the memory usage app for the app pool in IIS keeps growing as soon as I run the report in the webviewer and eventually I have to just recycle the app pool to free the memory.

    In my case I'm using a pageReport (with a FixedPage) and cannot use the CacheToDisk option.

    I also haven't come across any documentation for how to properly release the pageReport object when used in conjunction with the webviewer (If that is possible or necessary I'm not sure)
Need extra support?

Upgrade your support plan and get personal unlimited phone support with our customer engagement team

Learn More

Forum Channels