We were recently setting up an automated data transfer for a client. Standard sort of thing with a file being periodically copied from one server to another, then processed on that server. When we tested it we found the processing was failing.
Closer investigation revealed the issue was that the file transferred from client server was using UTF-16 encoding and the processing was UTF-8. There was no need for the files to be in UTF-16 so the solution was to convert them to UTF-8. To achieve that we used the iconv utility using the following syntax
iconv --from-code UTF-16 --to-code UTF-8 input_utf16.xml > output_utf8.xml