Handling of large RAF files?

rkaiser

Hello,

our customers are creating very large RAF files (about 20000 pages with lots of images in it). The total size of one of these files is (currently) about 3 GB but i assume that is only a matter of time that they will generate a RAF files that hit the 4GB boundary.

The position of a page in the RAF file is saved with 4 bytes (according to the sources as LongInt), so a RAF file that is larger than 4GB would need a different page table where the position values are saved as "int64". Is that correct?

Will such large RAF file be supported in the future (or are the currently supported, maybe i have missed a feature?)?
Ciao,
Ralf

nicocizik

Hi Ralf,

Thanks for the suggestion. We will add this to our list of possible enhancements to be researched and possibly implemented for a later release of the product.

rkaiser

Hello Nico,

Yesterday we had the case that there was a RAF file with a size of 2,7GB. We wanted to export it to a PDF file using the TppPDFDevice. While the export was running we got an EReadError (invalid stream format).

The RAF file itself was OK. We have a self written procedure that exports all included pages as single page by sequentially reading the RAF file, extracting a page and export it as single PDF file.

So we think that it must be an issue with the PDF device, maybe it accesses the offset for the pages as integer? That would produce an overflow when the 2GB boundary is reached and the intergers would get negative.

Ciao,
Ralf

nicocizik

Hi Ralf,

This is not a known issue. Which version of ReportBuilder and Delphi are you using? Are you able to recreate this with a simple example?

rkaiser

That would be a littl bit difficult due to the large RAF file that is needed to show the effect. (and i cannot use the RAF file that was provided by our customer because of sensitive data like addresses and invoices in that file)

We are currently using V18.03 (do not know when we will switch to V19 yet)

rkaiser

Here is a simple example. I have read the page positions inside the file and the page sizes. The page positions (stored in 4 Byte) have been read and as Cardinal and as Longint:

This small table is an excerpt taken from a big RAF file. (I did not wat to list all 7700 positions )

AsCardinal | AsLongInt | Size
2147088440 | 2147088440 | 333934
2147422374 | 2147422374 | 265375
2147687749 | -2147279547 | 333915 ***
2148021664 | -2146945632 | 333872

You can see, if you add the current size to the current position you get the position of the next page. But at the point where the value exceeds 2GB (at "***") the "LongInt-Interpretation" of the 4 byte ist getting negative and TppCustomArchiveReader.ReadPageEntry ist reading the position as LongInt into the TppPageEntryRec.Position.

If the positions would be read as Cardinal (still 4 Byte but not signed) the RAF format would work until the size exceeds 4GB. But with the current interpretation it stops working at 2GB.

We have written functions that are able to extract all pages by reading the the stream sequential, copy one page from the current read position and save it to a single file (re-stream the TppPage) but in much cases we need to use the TppArchiveReader and that seems to fail over 2GB file size.

Any idea what to do?

nicocizik

Hi Ralf,

This appears to be a design limitation of the archive feature. We have already added this to our list of items to research and enhance for a later version.

I'm a bit unclear about your description. The table becomes invalid around 2GB. This would mean that the RAF file itself would then be invalid as the ArchiveReader uses the table to determine where pages begin and end. However, in your first post you claim to have a RAF file that is 3GB in size. How can this be?

rkaiser

Yes, we have created RAF files bigger than 2GB. ans yes, all those RAF files above 2GB are invalid, they cannot be converted to PDF using the pdf devive (here the archive reader is used too). We get a ReadError (invalid stream format) for the first page above 2GB because the archive reader reads a negative page position from the page table and so is not able to find the correct start position for the requested page. The ReadError is then generated by the call to "ReadComponent" the starts reading from the wrong position.

However the single pages inside the RAF files are OK. The problem is, that the archive reader ist not able to find the correct positions if the current position is located somewhere above 2GB.

We have written some functions to extract single pages from the RAF file. In these funtions we read the page table using Cardinal-variables to read the start positions, re-position the filestream and save a single page to a single RAF file. Because the use of Cardinal we do not get an overflow at 2GB. Of course that method will only work until we reach 4GB, from that point on the page table would need to store the values as int64 and that would mean a change to the RAF-format.

nicocizik

Hi Ralf,

Thanks for the clarification. I was thrown off when you said in your second post that the "RAF file itself was OK". The pages inside the RAF file are OK but the RAF file itself is invalid when over 2GB in size.

Just to be clear, and for anyone following this thread, this is a limitation of the Archive device. The limitation is not PDF related and the issue will occur for any output device (screen, printer, or file).

We will consider enhancing the Archive device to properly handle creating files larger than 2GB for a future release.