You can set an output directory wherever you want as long as your web server is set up in a way which allows PHP to access it and write to it. There are two reasons why this might not be possible:
- Ownership and permissions. Depending on your server setup, your FTP/SFTP user and your web server might be running under different system users and groups. If your new output folder has 0755 permissions, the default, it is read-only to system users and groups which are different to the ones it is owned by. A quick test is to temporarily use 0777 permissions. If that works you need to talk to your host about the best way to handle the folder's permissions and ownership so it's writeable by PHP.
- open_basedir. See the official PHP documentation. Many hosts, especially low quality and/or WordPress hosts, use this architecturally incorrect approach in their server configuration. If they limit it to the site's web root you can't use an output directory outside of it. You will need to contact your host and explain your use case. If they refuse to change their configuration you'll have to create a backup output directory inside your web root.
Both of these problems are caused by server configuration we have no say to. The only thing we can do is detect when the output directory is unwritable and fall back to the default so that backups can run. An unwritable output directory not only doesn't let the backups run but also results in no log file being output since it would indeed try to use the output directory which is not writeable to begin with. This had always been a source of frustration which is why we automatically override it for the last several years.
Regarding the cannot connect error, this also comes from your server. It's possible that your host has configured the server in such a way that sending a path in the request — necessary to implement a file browser on a completely stateless CMS like WordPress — is prohibited and results in a connection error. Once again, this is something you need to talk to your host about or just enter paths manually, without using the folder browser. We obviously can't implement a folder browser if it doesn't know which folder it's meant to browse and we have no say on your server configuration.