[MAGNOLIA-5369] uncached file which is bigger than in-memory threshold does not fully download Created: 08/Oct/13  Updated: 20/Aug/14  Resolved: 31/Oct/13

Status: Closed
Project: Magnolia
Component/s: cache
Affects Version/s: 4.5.10
Fix Version/s: 4.5.13

Type: Bug Priority: Critical
Reporter: Tomáš Gregovský Assignee: Milan Divilek
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PDF File MagnoliaWhitePaperSTK.pdf    
Issue Links:
Cloners
is cloned by MGNLCACHE-25 CLONE - uncached file which is bigger... Closed
Relates
relation
is related to MAGNOLIA-5108 GZIP response when threshold for in-m... Closed
Template:
Acceptance criteria:
Empty
Task DoD:
[ ]* Doc/release notes changes? Comment present?
[ ]* Downstream builds green?
[ ]* Solution information and context easily available?
[ ]* Tests
[ ]* FixVersion filled and not yet released
[ ]  Architecture Decision Record (ADR)
Bug DoR:
[ ]* Steps to reproduce, expected, and actual results filled
[ ]* Affected version filled
Date of First Response:
Sprint: 4.5.13

 Description   

Issue found on (cache 4.5.11) but tested on demopublic45 (4.5.12) with same result.

What happens: upload pdf file with 640kB size to dms and put it into page as "download link". If you try to download it from public instance in Safari file is downloaded only with 512kB size, and because it is pdf, it cant be opened. If you download file again file is fully downloaded and all is ok. When cache if flushed then there is same problem again....



 Comments   
Comment by Tomáš Gregovský [ 10/Oct/13 ]

here is original 640kB file which sometimes was fully downloaded and sometimes cropped to 512kB

Comment by Christoph Meier [ 30/Oct/13 ]

A short summary what happens and and a guess why it happens.


When the file is requested the 1st time (1st-req.):::

Original Request (org.apache.catalina.connector.ResponseFacade) is wrapped 2x.
(a) in SecurityCallbackFilter into StatusSniffingResponseWrapper
(b) in Store into a CacheResponseWrapper
When the issue appears, from both response-wrappers getOutputStream is called and the InputStream of the File is copied into the ServletOutputStream.
=> This should take place only once; calling getOutputStream() the 2nd time i would expect to throw an exception (since the underlying response of both response-wrappers are always the same)

The 1st write write-execution (write#1) is called in DMSDownloadServlet#sendUnCompressed (using CacheResponseWrapper to write into)
The 2nd-write-execution (write#2) is called from DelegatingBlobCachedEntry#writeContent (using StatusSniffingResponseWrapper)
Both writing finally is done by IOUtils#copyLarge; write#1 writes 654411L; write#2 only 512000L bytes

At the end on StatusSniffingResponseWrapper-obj. flush() is called (in UseCache#processCachedEntry).

So .. write#1 would be sufficient, but instead there's a 2nd write process which is uncomplete and then its response becomes flushed.


When the file is requested for the 2nd time (2nd-req.), when the file is cached and no more problems appear:

Ori.-response is wrapped only once, like described in 1st-req.::(a)
Getting the ServletOutputStream and writing into it happens then in DMSDownloadServlet#sendUnCompressed#sendUnCompressed.
So, here StatusSniffingResponseWrapper is used to write successfully into its stream

>>>
i had a 1st idea some ideas for ... hm ... workarounds, but in this area, it seems risky to have "hacks".
And currently i was not able to test it, since i have troubles using DMS-Module on my intelliJ, yet.

I let this here as starting point for further discussions with somebody which has a deeper knowledge of the whole Filter-Chain than me.

Comment by Christoph Meier [ 31/Oct/13 ]

write#2 (in DelegatingBlobCachedEntry) takes the file - "restricted" to 512kB - ... so the "wrong file" is bound to the request as attribute in Store#makeCachedEntry l. 186 ...
But threre is still a kind of parallel World in DMSDownloadServlet which writes the full-file -independend from chache - to res.-wrapper (oy type StatusSniffingResponseWrapper) ... ???

Comment by Milan Divilek [ 20/Aug/14 ]

Cache is using ThresholdingCacheOutputStream. So when threshold is reached then java.io.FileOutputStream or stream response directly to user is used. And problem here was that, instead java.io.FileOutputStream or stream response directly to user, uncomplete in-memory stream was used.

Generated at Mon Feb 12 04:04:30 CET 2024 using Jira 9.4.2#940002-sha1:46d1a51de284217efdcb32434eab47a99af2938b.