Close
Glad You're Ready. Let's Get Started!

Let us know how we can contact you.

Thank you!

We'll respond shortly.

PIVOTAL LABS
Standup 04/27/07: Testing File Uploads

The setup:

I’m told file uploading is a pain to test. We needed to. So we cruised through the tubes over to ruby-doc.org to check out the Net::HTTP rdoc — only to find that Net:HTTP::Post does not support multipart uploading and files. What to do, what to DO?!?

The research:

Some googling later, we find this article showing how to do it. A little copy-paste, a small spike later, and we have an external script capable of uploading files into our web-apps. But, lets brain-storm a little…

  • How can we make it better?
  • What would be a nice interface?

Well, the first step is to change the script such that it can be more easily integrated into rake test:functionals: make it less script-y; more library. The interface is somewhat inspired by the basic_auth method. All you have to say is Net::HTTP::Post.new().multipart_params = {}? You give it a hash, and it takes care of the rest. Huzzah! So lets open up Net::HTTP::POST and give it some new methods. Time for some CODE!!!

The Code

require 'net/https'
require "rubygems"
require "mime/types"
require "base64"
require 'cgi'

class Net::HTTP::Post
  def multipart_params=(param_hash={})
    boundary_token = [Array.new(8) {rand(256)}].join
    self.content_type = "multipart/form-data; boundary=#{boundary_token}"
    boundary_marker = "--#{boundary_token}rn"
    self.body = param_hash.map { |param_name, param_value|
      boundary_marker + case param_value
      when String
        text_to_multipart(param_name, param_value)
      when File
        file_to_multipart(param_name, param_value)
      end
    }.join('') + "--#{boundary_token}--rn"
  end

  protected
  def file_to_multipart(key,file)
    filename = File.basename(file.path)
    mime_types = MIME::Types.of(filename)
    mime_type = mime_types.empty? ? "application/octet-stream" : mime_types.first.content_type
    part = %Q|Content-Disposition: form-data; name="#{key}"; filename="#{filename}"rn|
    part += "Content-Transfer-Encoding: binaryrn"
    part += "Content-Type: #{mime_type}rnrn#{file.read}"
  end

  def text_to_multipart(key,value)
    "Content-Disposition: form-data; name="#{key}"rnrn#{value}rn"
  end
end

Oh the utility:

Now that’s more like it. Hackish, since you have to stick headers into the request body, but effective. Notice the bit in there about MIME::Types. Did you see that? Yeah, we went there. Say it with me… Automatic mime type detection with a safe default. The absurd thing in there is that the MIME::Types gem (as of today) does not know about .rb files.

irb(main):007:0> MIME::Types.of('something.rb')
=> []

So now that you have that, it’s just a simple use of Net::HTTP with a blizzock to upload a file in a functional test.

File.open(File.expand_path('script/test.png'), 'r') do |file|
  http = Net::HTTP.new('localhost', 3000)
  begin
    http.start do |http|
      request = Net::HTTP::Post.new('/your/url/here')
      request.basic_auth 'lonely_user', 'really_long_password'
      request.multipart_params = {'file' => file, 'title' => 'title'}
      response = http.request(request)
      response.value
      puts response.body
    end
  rescue Net::HTTPServerException => e
    p e
  end
end

The questions:

So what do you think? How can this be made even better?

Comments
  1. Ron says:

    Very, very nice.

    However, I believe this line:
    part += “Content-Type: #{mime_type}rnrn#{file.read}”

    is missing a “rn” at the end.

    Again, thanks so much — this will be very helpful to me.

    Ron

  2. Ron says:

    I needed to send multipart form data to another server, since right now RoR doesn’t do very well at storing BLOB data in SQL Server. I can’t fix that, so I am using your code in a workaround.

    I added a file, “multipart.rb,” to my /lib

    It’s very derivative of your code, with a couple of exceptions:

    • My controller reads the file contents and passes them in, along with the filename, in an array. This lets me sanitize Windows filenames on the fly.
    • file_to_multipart didn’t like integer values in the param_hash, so I added “.to_s” to the call.
    • The line that adds the file_content did need “rn” stuck on the end, so that’s there.

    Here’s the contents of multipart.rb:

    require 'net/https'
    require "mime/types"
    
    class Net::HTTP::Post
      def multipart_params=(param_hash={})
        boundary_token = [Array.new(8) {rand(256)}].join
        self.content_type = "multipart/form-data; boundary=#{boundary_token}"
        boundary_marker = "--#{boundary_token}rn"
        self.body = param_hash.map { |param_name, param_value|
          boundary_marker + case param_value
          when Array
            file_to_multipart(param_name, param_value[0], param_value[1])
          else
            text_to_multipart(param_name, param_value.to_s)
          end
        }.join('') + "--#{boundary_token}--rn"
      end
    
    
      protected
      def file_to_multipart(key, file_content, filename)
        mime_types = MIME::Types.of(filename)
        mime_type = mime_types.empty? ? "application/octet-stream" : mime_types.first.content_type
        part = %Q|Content-Disposition: form-data; name="#{key}"; filename="#{filename}"rn|
        part += "Content-Transfer-Encoding: binaryrn"
        part += "Content-Type: #{mime_type}rnrn#{file_content}rn"
      end
    
      def text_to_multipart(key,value)
        "Content-Disposition: form-data; name="#{key}"rnrn#{value}rn"
      end
    end
    
  3. Ron says:

    I meant to leave my email; this ought to do the trick.

  4. Ron says:

    P.P.S. — moving the file.read operation out to the controller also lets the controller check for non-existent files.

    R

  5. Ron says:

    Oh, dear. I meant to say that “text_to_multipart” didn’t like integers.

    R

Post a Comment

Your Information (Name required. Email address will not be displayed with comment.)

* Copy This Password *

* Type Or Paste Password Here *