Memory fix for pak export#206
Conversation
…s to help with memory management
…ack approach and uses the same reader/stream for faster execution and better memory management
|
I don't understand why it's highlighing the entire FileContainerForm.cs as changed.. Maybe different line endings? or one of those just Git things.. Relevant changes are on lines 1102 (new method being called in second commit) |
|
I definitely welcome a better implementation of right click -> Extract on folders, right now I indeed don't expect it to be usable for large folders, instead only the Extract All menu option is usable (because that option actually does use a single reader to extract all files). it's hard to tell what you changed in your second commit because of what looks to me like a change in the line endings for the whole file, would be good if you could soft reset your branch to the commit before the problematic commit, restore the line endings to what they were originally, and force-push to your branch with the amended commit so i can better see what you have changed here. once you have done that i can more thoroughly review the rest of your changes I don't think it should be necessary to set it is safe to re-use the same reader for multiple assets, the file container form can't load more than one .pak file at once |
2 participants
Pull Request
Description
Commit 6985e51
Added var for basePath for cleaner code, and res=null;
Since what the reader is reading can be large, it gets added to the large object heap, reseting the value to null flags it for GC as recouperable space.
Commit 5795288
When trying to extract Deadside's main .pak, UAssetGUI would consume every bit of memory it can and barely manage around 99% and eventually a larger export would kill the process and the debugger with it.
This linear approach which (most importantly) reuses the same stream and reader prevents memory from building up.
Additionally, in the prior game version which did extract successfully the process lasted for about 10 minutes. With this approach the export duration is 13 secionds and there's a minor memory bump which goes away as soon as the process completes.
Important note: I fully understand that my discreet experience with Deadside is far from covering any other UE 4 or 5 game. I'm PR-ing this in best faith, and in case it seems implausable, I still advise the linear stack approach as it has several benefits which outclass any downsides: a) list position can be referenced in a progress bar if it's implemented, b) easier to debug as there's a separation of concerns between assessing what needs to be exported, and actual export process.
Key takeaway is, even if a single reader/stream can't be reused (in case there's multiple paks and each exported object needs to point to it's pak for example, I don't know if this is possible even, but then again, I don't know what are possible permutations either), they should be maintained separately instead of spawning new ones for every file being exported. This causes huge memory and time overhead.
Type of change
Testing
Open UAssetGUI,
Click File > Open Containers...,
select a production ready game pak (I used game Deadside which is based on UE4.27)
Expand to Content > Blueprints,
Right click Blueprints > Export
*Monitor how long it takes and how much memory it consumes