Java Heap Space limiting more than 50ish title searchs

[NewMember12345]

Issue: When I use JAVMovieScraper to scrape for anything more than 50 titles, sometimes less, i get an error with java heap space out and it wont write any more data like poster, fanart, nfo.

This occurs as well if I scrape for 30-ish titles, write them to disk, then select the next 30. Seems like this java temp space can't handle that many jpgs. I often just close out the jar and open up another instance.

This did not occur in really old iterations of this project. But it has happened since v3.6 or so. I never thought to bring it up but I'd like to see if zuko or wizell can shed some light?

[Wizell]

I never did such big scraping session.
But i can image 2 main reasons:

• as every art is kept in memory it can be just too much compared to how much memory the application can have access to
• there is a memory leak

To rule out the first one can you do the following checks:

In the terminal you start type the command line in, you should see something like:
Heap: X GB
Free mem: Y GB
Can you give those values ?

Also can you try adding -Xms2048m -Xmx4096m to your command line.
like java -Xms2048m -Xmx4096m -jar JAVMovieScraper.jar.

If this does not help, i will profile application's memory usage when i got some time

[NewMember12345]

@Wizell Unforunately I use windows and do not code. So i have no way of knowing how to do those you mentioned.

[Wizell]

How do you start JavMovieScraper jar file ?

[NewMember12345]

I have Java runtime environment 8 installed. On Windows, I just double click on the .jar. It default to launch with JRE

[NewMember12345]

If i try to run it cmd in windows, I get this big box error:

D:\torrents\later\tmp>java -Xms2048m -Xmx4096m -jar JAVMovieScraper-v0.3.7.jar
Java(TM) SE Runtime Environment 1.8.0_261-b12(Oracle Corporation) -- 1.8(Oracle Corporation)
Heap: 3 GB
Free mem: 1 GB
Creating settings.xml since it was not found...
Initializing default jav preferences
Aug 23, 2020 5:08:35 PM moviescraper.doctord.scraper.DitzyHeadlessBrowser
INFO: Build browser with U: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
java.lang.NullPointerException
at java.io.File.(Unknown Source)
at moviescraper.doctord.scraper.DitzyHeadlessBrowser.configure(DitzyHeadlessBrowser.java:35)
at moviescraper.doctord.controller.siteparsingprofile.specific.JavLibraryParsingProfile.browserConfigure(JavLibraryParsingProfile.java:102)
at moviescraper.doctord.controller.siteparsingprofile.specific.JavLibraryParsingProfile.(JavLibraryParsingProfile.java:93)
at moviescraper.doctord.controller.amalgamation.AllAmalgamationOrderingPreferences.initializeJAVCensoredGroupDefaultPreferences(AllAmalgamationOrderingPreferences.java:129)
at moviescraper.doctord.controller.amalgamation.AllAmalgamationOrderingPreferences.initializeDefaultPreferences(AllAmalgamationOrderingPreferences.java:103)
at moviescraper.doctord.controller.amalgamation.AllAmalgamationOrderingPreferences.initializeValuesFromPreferenceFile(AllAmalgamationOrderingPreferences.java:69)
at moviescraper.doctord.view.GUIMain.reinitializeAmalgamationPreferencesFromFile(GUIMain.java:157)
at moviescraper.doctord.view.GUIMain.initialize(GUIMain.java:169)
at moviescraper.doctord.view.GUIMain.(GUIMain.java:142)
at moviescraper.doctord.view.GUIMain$1.run(GUIMain.java:126)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$500(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

[Wizell]

You got the command line right, the error is a mistake in the settings of the application.
I fixed it few hours ago. Please retry with the next release that should happen during this week.

[NewMember12345]

Thanks @Wizell

I was able to run the command on a VM that runs my primary scraping. I could only max it to 1gb heap and free mem. I'll try to see if it can do more than 50 titles. I typically scrape 100 titles every 3-4 days.

[NewMember12345]

@Wizell Do you have a quick idea how to compile your files from github into a working .jar? Maybe for future users because we see code fixes but unable to use them until a release.

[NewMember12345]

In the terminal you start type the command line in, you should see something like:
Heap: X GB
Free mem: Y GB
Can you give those values ?

java -jar JAVMovieScraper-v0.3.7.jar
Java(TM) SE Runtime Environment 1.8.0_231-b11(Oracle Corporation) -- 1.8(Oracle Corporation)
Heap: 247 MB
Free mem: 13 MB
Aug 23, 2020 5:38:06 PM moviescraper.doctord.scraper.DitzyHeadlessBrowser
INFO: Build browser with U: Mozilla/5.0 (Linux; Android 7.0; Moto G (5) Build/NPPS25.137-93-8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.137 Mobile Safari/537.36
Movie Size: 0
Gui Initialized