RB + Long report + Many bitmaps = Out of memory problem

rbuser

Hi,

I am still fighting with Report Builder and "Out of memory" problems.

What is the problem?
We use WPTools text component to write texts in our system. Later we need to
generate reports with Report Builder and we need to print text fields
written in WPTools in this report. WPTools has wrapper for ReportBuilder and
it works correctly as expected. But problem is, that this wrapper generates
EMF images for each text filed and page. This EMF file is then sended to
ReportBuilder engine via following draw commands:

---- Code snippet ----
lMetafile := AsMetafile;
try
lppDrawImage := TppDrawImage.Create(nil);
try
.... setting size and other properties .......
lppDrawImage.Picture.Assign(lMetafile);
aDevice.Draw(lppDrawImage);
finally
lppDrawImage.Free;
end;
finally
lMetafile.Free;
end;

But when the report contains many pages (more than 100) and there are many
metafiles (some of them with bitmap graphic within), Report Builder just
eats up more and more memory. I tried to use PageCache - an it works - it
generates files on drive, but the memory is not freed. All files has nearly
the same size as memory consumed by Report Builder. When I reset report, all
memory is correctly freed. It looks like there is no memory leak - I tested
it.

I created sample application just for demonstration - it generates 1500
pages with just many bitmaps on it. It uses plain Report Builder - no third
party components. You can download it from
http://www.dosli.cz/soubory/ruzne/RB_Bitmap_Memory.zip. Just run it together
with ProcessExplorer and look at the ram size alocated by this application.
Then press Preview Button. You will see constant grow of the allocated
memory. There is CachePage property set to True and the files are generated
within the same folder. We really need to handle longer reports with many
bitmaps. Can you help us, please?

Thanks for your answer

Petr Slípek.


---
Tato zpráva neobsahuje viry ani jiný škodlivý kód - avast! Antivirus je aktivní.
http://www.avast.com

nicocizik

Hi Petr,

1. Which version of ReportBuilder and Delphi are you using?

2. Are you using the Windows Task Manager to monitor your memory usage
or a memory manager such as FastMM or AQTime?

The CacheManager should not have to be altered. If engine memory is an
issue, the CacheManager will switch to file base caching automatically.

There is also the ThreadedPageCache at work here. Try setting
Viewer.SinglePageOnly to True and see if that helps the issue.

I tried running your application on my machine and did not receive any
errors or detect any memory leaks (using FastMM).

rbuser

Hi,


1. I am using RB 14.08 in Delphi XE2.


2. I am using Process Explorer from sysinternals.com. It allows us to see
exact ram allocated by single process.

I know, but you missed one important point. In WPTools we can edit text and
also insert OLE objects, images and other media types inside the text.
WPTools converts content to metafile (AsMetaFile routine) on the fly and
this process involves other routines which handles these media types and
converts them into bitmap objects. This bitmap is painted to metafile canvas
in required resolution. It is quite complicated process and it involves
some memory heavy bitmap operations - another story - like antialiased
resize of inner metafile bitmap objects to enhance output quality. And here
we face the memory problems - if Report Builder allocates more and more
memory for its engine, we are not able to allocate memory for our own image
processing when converting WPTools to metafile. So we need to switch on page
caching in report builder before the engine runs out of memory. I created
the demo just to show you, that RB allocates more and more memory when
printing and doesn't leave enough memory for other routines which must be
executed within report generation. I am not able to create simple demo
which shows you the memory problem - we have too many custom components
involved in the process. We were looking at our own code heavily for very
long time and figured out that this problem is related to RB engine. Because
our code has no memory leak or any other problem. It always work with
shorter reports. The problem occurs only, when report builder allocates too
much memory. When we generate report in this situation, the problem will be,
that some of inner text images are empty - no able to paint them on
metafile's canvas. And one or two pages later the application hangs with Out
of memory problem. The situation is much worse, because it happens on
different machines in different page positions. On one machine you can
generate whole report without problem. On another machine you generate just
half of the report. And sometimes you can generate the whole report, but
some images are not correctly painted.

Problem is that we need to turn on page caching before memory runs out and
when page caching is on, we need to have memory freed for unnecessary pages.
Just free memory when page is not needed. Maybe you can enhance Report
Builder to be able to setup some free memory limit when starting to use page
cache. Just tell the engine that there must be always at least some free
memory space. For example on my machine any report always crush, when RB
allocates more than ~1,2 GB RAM.


Tried that and the issue is still there.

As I told you - there is no memory leak. Just look at the allocated memory
for the process. And worse is that the memory is not freed until you call
code like ppReport1.Template.New or Load or some other method. Maybe there
is another method which we can call to tell ppReport engine to free all
allocated memory.

Best Regards,

Petr Slipek


---
Tato zpráva neobsahuje viry ani jiný škodlivý kód - avast! Antivirus je aktivní.
http://www.avast.com

nicocizik

Hi Petr,

Thanks for the further information. I downloaded Process Explorer and
took a look at some of the numbers while running your application.

1. I assume you are referring to the Private Bytes field increasing when
running your application. This however is not a very good indicator to
the amount of memory an application (or part of an application) is
actually using. Private Bytes refer to the amount of memory that the
process executable has asked for - not necessarily the amount it is
actually using. If you want to track the exact amount of memory each
object is using in an application you will need to use an advanced profiler.

2. In your code you set the CacheManger properties and the
Report.CachePages property. These are two very different concepts. The
CacheManager is used by the engine to cache report components that
descend from TppCachable to memory or file. This has nothing to do with
caching pages.

Setting CachePages to True will inform the Publisher (TppPublisher) to
cache each generated page to memory. That way, if the page is requested
again, it can obtain it from the cache rather than asking the engine to
re-generate it. Set CachePages to False to use less memory.

3. The engine itself only creates and maintains a single TppPage object.
Once a page has been generated, it's drawcommands are freed and the
page object is reused. The publisher and graphics device also should
only keep a copy of the current or viewed page(s) in memory but do not
cache all pages.

It is on our todo list to possibly enhance the publisher to allow
caching pages to file.

rbuser

Hi,

thanks for some explanation about caching. But setting CachePages to False
doesn't change anything. The memory reserved for application remains still
the same.



If I understand this correctly. Then the application I sended you should use
nearly constant ammount of memory (some memory for each page - info where it
is stored or some other information). If you look at the report definition
it paints on each page the same number of images. I would expect that if the
report generates still the same pages draw commands, there is no need for
more memory for each page. So, why is still constant grow in memory reserved
by the application and why resetting the report frees reserved memory to the
original - so there must be something in report engine which allocates more
and more memory for each generated page. And if you put bigger image in the
report (try some really big photo) the memory consumption is much bigger. So
the allocated memory doesn’t depends on number of generated pages. It
depends of the size of the printed image source - and this is still problem
we need to focus on. You can download sample PDF file generated by our
application (http://www.dosli.cz/soubory/ruzne/Course.pdf). The memory grows
from 170 MB Private Bytes to 520 MB Private Memory while generating this
report. And generating longer report (about 200 pages and allocated more
than 1.2 GB Private Bytes) makes our application fail. Remember, that each
chapter content if generated from WPTools as Metafile and the bitmaps are
painted on metafile canvas.

Petr.


---
Tato zpráva neobsahuje viry ani jiný škodlivý kód - avast! Antivirus je aktivní.
http://www.avast.com

nicocizik

Hi Petr,

On my machine, using the example you sent, I am unable to recreate the
behavior you describe with CachePages property set to False.

If I load the report, then navigate to the last page (generating every
page) the memory usage (Private Bytes) does not increase.

Are you getting different results with the demo you sent? If so, there
must be something else going on. If you trace into the TppPage
constructor and desctructor you can keep track of how may page objects
are created as the report generated. With CachePages set to False,
three TppPage objects were created (Engine, Publisher, Graphics Device).

Please alter the example you sent to reflect the behavior you are
describing and I'll take another look at it.

rbuser

Hi,

some info about our problems..... we finally be able to find the bug and I
must say sorry - no bug in Report Builder. It was a bug deep inside WPTools
components - no memory leak, but wrong way how it handles memory for
displayed images when loading different content into the same component.

Thanks for your help.

Petr Slipek


---
Tato zpráva neobsahuje viry ani jiný škodlivý kód - avast! Antivirus je aktivní.
http://www.avast.com