Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8 (SFTP)

3
votes

Using the Net-SFTP gem, Ruby 2 and Rails 4

I wrote code that was working in pure ruby, but copied my code over to rails and now I get:

Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8

What can I change in my code to get this working?

def self.get_recent_file(ftp_file, local_file)
    Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
      sftp.download!(ftp_file, local_file)
    end
  end

Log

Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `write'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `write'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `on_read'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `call'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `respond_to'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:948:in `dispatch_request'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:911:in `when_channel_polled'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `call'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `block in preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `each'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:205:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
... 13 levels...
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:205:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:802:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:787:in `connect!'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp.rb:32:in `start'

Code referenced in log from GEM:

https://github.com/net-ssh/net-sftp/blob/master/lib/net/sftp/operations/download.rb#L339

# Called when a read from a file finishes. If the read was successful
  # and returned data, this will call #download_next_chunk to read the
  # next bit from the file. Otherwise the file will be closed.
  def on_read(response)
    entry = response.request[:entry]

    if response.eof?
      update_progress(:close, entry)
      entry.sink.close
      request = sftp.close(entry.handle, &method(:on_close))
      request[:entry] = entry
    elsif !response.ok?
      raise "read #{entry.remote}: #{response}"
    else
      entry.offset += response[:data].bytesize
      update_progress(:get, entry, response.request[:offset], response[:data])
      entry.sink.write(response[:data]) # <~~ Line#339
      download_next_chunk(entry)
    end
  end
5
ruby-on-railsrubynet-sftp
what was written /download.rb:339:i in line#339? I need to see it. - Arup Rakshit
@ArupRakshit I edited the question and added what you requested - user2012677
point me to line# 339 - Arup Rakshit
@ArupRakshit github.com/net-ssh/net-sftp/blob/master/lib/net/sftp/operations/… - user2012677

5 Answers

6
votes

This helps me:

def self.get_recent_file(ftp_file, local_file)
  local_io = File.new(local_file, mode: 'w', encoding: 'ASCII-8BIT')
  Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
    sftp.download!(ftp_file, local_io)
  end
  local_io.close
end
4
votes

A combination of user72136's answer and the answer to this question worked for me (my remote file wasn't even ASCII): 

def self.get_recent_file(ftp_file, local_file)
  local_io = File.new(local_file, mode: 'wb')
  Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
    sftp.download!(ftp_file, local_io)
  end
  local_io.close
end
1
votes

As line#339 is showing

entry.sink.write(response[:data])

Fix it as :

entry.sink.write(response[:data].force_encoding('ASCII-8BIT').encode('UTF-8'))
0
votes

Change the line -

sftp.download!(ftp_file, local_file)

to say

sftp.download!(ftp_file, local_file).to_s.encode('UTF-8', {:invalid => :replace, :undef => :replace, :replace => '?'})
0
votes

This problem is being produced by how Ruby opens text-files by default after Ruby 2.0 version with UTF-8 encoding. Where you open your file you can put:

local_file = Tempfile.new(encoding: 'ascii-8bit')
#or another thing to do is to switch to binary-mode
local_file = Tempfile.new
local_file.binmode

You can also open a binary-file like this:

local_file = File.open('/tmp/local_file', 'wb')

Another solution you can do is to pass to the gem-code the filepath, instead of an open file:

def self.get_recent_file(ftp_file, local_file)
  Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
    sftp.download!(ftp_file, local_file.path)
  end
end