In GnuPG 2.1.x, secret key is under control of gpg-agent. You can use
--pinentry-mode=loopback.
But, I think that --batch should imply --pinentry-mode=loopback.

When I try the following under gnupg 2.1.3 with arch linux:

$ gpg --homedir <gpg-dir> --batch --pinentry-mode=loopback --passphrase-file
<passfile> --decrypt myfile.gpg

I get the following error:

gpg: setting pinentry mode 'loopback' failed: Not supported
...
gpg: decryption failed: No secret key

Is the gnupg version of arch just missing some compile-time flag to support
--passphrase-file without manual pinentry? If this is the case, I could report
this back to the arch maintainer to get it fixed downstream.
Or is there still some work to be done on gnupg?

It seems that your gpg-agent doesn't support loopback mode.
Either, your gpg-agent is from 2.0 or the socket is hijacked by gnome-keyring.
For the latter, please see http://wiki.gnupg.org/GnomeKeyring

It seems that the gpg-agent needs to be started with --allow-loopback-pinentry
for this to work.
Because I let gpg autostart the daemon for me, this does not get passed to
gpg-agent and therefore does not work when setting --pinentry-mode=loopback in gpg.

I don't know what is to do here.
Should gpg with --pinentry-mode=loopback autostart the gpg-agent with
--allow-loopback-pinentry ?
Or should I just add some documentation to the manpages to describe what is
necessary for --pinentry-mode=loopback and --passphrase-file to work?

I checked the code and the behavior. It is confirmed that the default of
gpg-agent disables loopback-pinentry mode and user needs to enable it.

Now, we need some fixes/improvements:
(1) gpg should automatically work with gpg-agent with the option of --passphrase
(-file, -fd).
In GnuPG 2.1.x, the secret keys are under control of gpg-agent and gpg frontend
should pass the passphrase to gpg-agent in some way.
When --passphrase (-file, -fd) option is supplied, gpg frontend could use
gpg-agent feature of either loopback-pinentry mode _OR_ preset_passphrase .
The latter requires specific key identification, so, loopback-pinentry mode
would be the solution for general.
(2) Both of loopback-pinentry mode and preset_passphrase are disabled as
default. We need to fix this default of gpg-agent _AND_ we need to fix gpg
frontend error handling of this case of disabled feature of gpg-agent. Well, I
don't know the reason this features need to be disabled...
(3) When it is gpg frontend which invokes gpg-agent, it would be natural to
enable loopback-pinentry (or preset_passphrase). But, there will be existing
gpg-agent even with --batch option. I don't know how it should work in this case.

As DKG noted on the mailing list, we --batch shouldn't imply
--pinentry-mode=loop. He provides the example of a graphical tool that fills in
many of the fields when generating a key, but should not have to worry about
securely managing the password.

This suggests that if a password is somehow provided on the command line, we
should prime (i.e., preset) gpg agent so that it doesn't request a password.

In another message (<874mnnlqxn.fsf@alice.fifthhorseman.net>) DKG notes that we
shouldn't allow loopback mode or preseeding to prevent passphrase-guessing attacks.

How would the suggested method of programmatically using gpg be?

I'm maintaining a service that uses gpg as a streaming encryption/decryption
backend and we need to provide the passphrase for the keys somehow in a
convenient manner.

Priming the agent is not optimal too because it would force me to restart the
agent every time i add new keys.

Maybe give me the possibility to provide new passphrases to the agent via the
agent socket?

In 3ef0938cfd8637e9801369f142eb8dd564f2ca61 --allow-loopback-pinentry became the default.