One way would be to process the file as a stream instead of loading it all in memory. The approach would be something like:
getFile as stream
open an output stream to file **
loop until eof
…read some number of bytes from the stream (1k, 5k, ??k) **
…base64Encode those bytes
…write the encoded bytes to the output stream **
flush output **
Items marked with ** are things you’d need to write as Java services. Or you could do the whole operation in a single Java service, invoking the base64Encode using the wM API. You might want to write your own open/close services to replace pub.file:getFile, since we don’t know how it works internally and there is no close service to release the stream/underlying file handle.
Of course this doesn’t address transporting the encoded file to its destination but you can use a stream approach with the built-in services (i.e. pub.client:ftp accepts a filename for put which presumably does not load the entire file into memory).