Hello, and welcome to the forum.
There are a couple of things you need to try out.
First: is it working in the browser ? Try logging into the administrative backend, and issue a backup command - do one trial for it compressed - and another trial for it uncompressed.
If it's working with uncompressed only, please disable compression in the backup tool also. The thing is that when the php scripts attempt to compress the database in memory - they do not use any sort of caching or something, so they first read the SQL backup in memory (let's say 50-60 Mb of data) and then they try to zip it which uses almost the same amount of memory. So if you have a limitation on how much memory a php script can use, it will already be killed - and you'll end up with an empty file.
Second: Please check the log files on the server. Java SMF Backup si just a wrapper, it does the clicks for you as you would do it in the browser. If something went wrong, I would first check the logs on the server, around the time when the backup happens.
Third: If you are using some other theme beside the default one, that might cause the issues. The backup tool tries to identify the form in which it has to put the username and password. If you use some other theme, it might also add its own forms handling language selections, or other things that might confuse the form detection.
To test this you can create a 'backup' user that is admin, and make the user's theme the default one. If backup works then from the Java SMF Backup, then it's the theme.
PS: if none of above works (I mean no errors on the server side, and it works with or without compression in the browser, but not the same with Java SMF Backup), I will have to ask you to provide me with a temporary admin username / pass so I could check it myself and see how the app behaves, and then will provide a bug fix as soon as possible.
Thank you,