1berry/ruby
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

merge revision(s) 60584,62954-62959,63008:

Browse Source 
webrick: support Proc objects as body responses

	* lib/webrick/httpresponse.rb (send_body): call send_body_proc
	  (send_body_proc): new method
	  (class ChunkedWrapper): new class

	* test/webrick/test_httpresponse.rb (test_send_body_proc): new test
	  (test_send_body_proc_chunked): ditto
	  [Feature #855]

	webrick: favor .write over << method

	This will make the next change to use IO.copy_stream
	easier-to-read.  When we can drop Ruby 2.4 support in a few
	years, this will allow us to use writev(2) with multiple
	arguments for headers and chunked responses.

	* lib/webrick/cgi.rb (write): new wrapper method
	  lib/webrick/httpresponse.rb: (send_header): use socket.write
	  (send_body_io): ditto
	  (send_body_string): ditto
	  (send_body_proc): ditto
	  (_write_data): ditto
	  (ChunkedWrapper#write): ditto
	  (_send_file): ditto
	------------------------------------------------------------------------
	r62954 | normal | 2018-03-28 17:05:52 +0900 (水, 28 3 2018) | 14 lines

	webrick/httpresponse: IO.copy_stream for regular files

	Remove the redundant _send_file method since its functionality
	is unnecessary with IO.copy_stream.  IO.copy_stream also allows
	the use of sendfile under some OSes to speed up copies to
	non-TLS sockets.

	Testing with "curl >/dev/null" and "ruby -run -e httpd" to
	read a 1G file over Linux loopback reveals a reduction from
	around ~0.770 to ~0.490 seconds on the client side.

	* lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream
	  (_send_file): remove
	  [Feature #14237]
	------------------------------------------------------------------------
	r62955 | normal | 2018-03-28 17:05:57 +0900 (水, 28 3 2018) | 10 lines

	webrick: use IO.copy_stream for single range response

	This is also compatible with range responses generated
	by Rack::File (tested with rack 2.0.3).

	* lib/webrick/httpresponse.rb (send_body_io): use Content-Range
	* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
	  use File object for the single range case
	* test/webrick/test_filehandler.rb (get_res_body): use send_body
	  to test result
	------------------------------------------------------------------------
	r62956 | normal | 2018-03-28 17:06:02 +0900 (水, 28 3 2018) | 7 lines

	test/webrick/test_filehandler.rb: stricter multipart range test

	We need to ensure we generate compatibile output in
	the face of future changes

	* test/webrick/test_filehandler.rb (test_make_partial_content):
	  check response body
	------------------------------------------------------------------------
	r62957 | normal | 2018-03-28 17:06:08 +0900 (水, 28 3 2018) | 8 lines

	webrick: quiet warning for multi-part ranges

	Content-Length is ignored by WEBrick::HTTPResponse even if we
	calculate it, so instead we chunk responses to HTTP/1.1 clients
	and terminate HTTP/1.0 connections.

	* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
	  quiet warning
	------------------------------------------------------------------------
	r62958 | normal | 2018-03-28 17:06:13 +0900 (水, 28 3 2018) | 7 lines

	webrick/httpresponse: make ChunkedWrapper copy_stream-compatible

	The .write method needs to return the number of bytes written
	to avoid confusing IO.copy_stream.

	* lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written
	  (ChunkedWrapper#<<): return self
	------------------------------------------------------------------------
	r62959 | normal | 2018-03-28 17:06:18 +0900 (水, 28 3 2018) | 9 lines

	webrick: use IO.copy_stream for multipart response

	Use the new Proc response body feature to generate a multipart
	range response dynamically.  We use a flat array to minimize
	object overhead as much as possible; as many ranges may fit
	into an HTTP request header.

	* lib/webrick/httpservlet/filehandler.rb (multipart_body): new method
	  (make_partial_content): use multipart_body

	get rid of test error/failure on Windows introduced at r62955

	* lib/webrick/httpresponse.rb (send_body_io): use seek if NotImplementedError
	  is raised in IO.copy_stream with offset.

	* lib/webrick/httpservlet/filehandler.rb (multipart_body): ditto.


git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_2@63020 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
This commit is contained in:
usa 2018-03-28 14:44:20 +00:00
parent 4cd92d7b13
commit 19cb3fa9e0
6 changed files with 244 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

98
ChangeLog
View File

@ -1,3 +1,101 @@
Wed Mar 28 23:41:53 2018 NAKAMURA Usaku <usa@ruby-lang.org>
get rid of test error/failure on Windows introduced at r62955
* lib/webrick/httpresponse.rb (send_body_io): use seek if
NotImplementedError is raised in IO.copy_stream with offset.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): ditto.
Wed Mar 28 23:41:53 2018 Eric Wong <normalperson@yhbt.net>
webrick: support Proc objects as body responses
* lib/webrick/httpresponse.rb (send_body): call send_body_proc
(send_body_proc): new method
(class ChunkedWrapper): new class
* test/webrick/test_httpresponse.rb (test_send_body_proc): new test
(test_send_body_proc_chunked): ditto
[Feature #855]
webrick: favor .write over << method
This will make the next change to use IO.copy_stream
easier-to-read. When we can drop Ruby 2.4 support in a few
years, this will allow us to use writev(2) with multiple
arguments for headers and chunked responses.
* lib/webrick/cgi.rb (write): new wrapper method
lib/webrick/httpresponse.rb: (send_header): use socket.write
(send_body_io): ditto
(send_body_string): ditto
(send_body_proc): ditto
(_write_data): ditto
(ChunkedWrapper#write): ditto
(_send_file): ditto
webrick/httpresponse: IO.copy_stream for regular files
Remove the redundant _send_file method since its functionality
is unnecessary with IO.copy_stream. IO.copy_stream also allows
the use of sendfile under some OSes to speed up copies to
non-TLS sockets.
Testing with "curl >/dev/null" and "ruby -run -e httpd" to
read a 1G file over Linux loopback reveals a reduction from
around ~0.770 to ~0.490 seconds on the client side.
* lib/webrick/httpresponse.rb (send_body_io): use IO.copy_stream
(_send_file): remove
[Feature #14237]
webrick: use IO.copy_stream for single range response
This is also compatible with range responses generated
by Rack::File (tested with rack 2.0.3).
* lib/webrick/httpresponse.rb (send_body_io): use Content-Range
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
use File object for the single range case
* test/webrick/test_filehandler.rb (get_res_body): use send_body
to test result
test/webrick/test_filehandler.rb: stricter multipart range test
We need to ensure we generate compatibile output in
the face of future changes
* test/webrick/test_filehandler.rb (test_make_partial_content):
check response body
webrick: quiet warning for multi-part ranges
Content-Length is ignored by WEBrick::HTTPResponse even if we
calculate it, so instead we chunk responses to HTTP/1.1 clients
and terminate HTTP/1.0 connections.
* lib/webrick/httpservlet/filehandler.rb (make_partial_content):
quiet warning
webrick/httpresponse: make ChunkedWrapper copy_stream-compatible
The .write method needs to return the number of bytes written
to avoid confusing IO.copy_stream.
* lib/webrick/httpresponse.rb (ChunkedWrapper#write): return bytes written
(ChunkedWrapper#<<): return self
webrick: use IO.copy_stream for multipart response
Use the new Proc response body feature to generate a multipart
range response dynamically. We use a flat array to minimize
object overhead as much as possible; as many ranges may fit
into an HTTP request header.
* lib/webrick/httpservlet/filehandler.rb (multipart_body): new method
(make_partial_content): use multipart_body
Wed Mar 28 23:37:18 2018 Nobuyoshi Nakada <nobu@ruby-lang.org>
pack.c: fix underflow

68
lib/webrick/httpresponse.rb
View File

@ -302,6 +302,8 @@ module WEBrick
def send_body(socket) # :nodoc:
if @body.respond_to? :readpartial then
send_body_io(socket)
elsif @body.respond_to?(:call) then
send_body_proc(socket)
else
send_body_string(socket)
end
@ -404,9 +406,20 @@ module WEBrick
end
_write_data(socket, "0#{CRLF}#{CRLF}")
else
size = @header['content-length'].to_i
_send_file(socket, @body, 0, size)
@sent_size = size
if %r{\Abytes (\d+)-(\d+)/\d+\z} =~ @header['content-range']
offset = $1.to_i
size = $2.to_i - offset + 1
else
offset = nil
size = @header['content-length']
size = size.to_i if size
end
begin
@sent_size = IO.copy_stream(@body, socket, size, offset)
rescue NotImplementedError
@body.seek(offset, IO::SEEK_SET)
@sent_size = IO.copy_stream(@body, socket, size)
end
end
ensure
@body.close
@ -435,24 +448,41 @@ module WEBrick
end
end
def _send_file(output, input, offset, size)
while offset > 0
sz = @buffer_size < size ? @buffer_size : size
buf = input.read(sz)
offset -= buf.bytesize
def send_body_proc(socket)
if @request_method == "HEAD"
# do nothing
elsif chunked?
@body.call(ChunkedWrapper.new(socket, self))
_write_data(socket, "0#{CRLF}#{CRLF}")
else
size = @header['content-length'].to_i
@body.call(socket)
@sent_size = size
end
end
class ChunkedWrapper
def initialize(socket, resp)
@socket = socket
@resp = resp
end
if size == 0
while buf = input.read(@buffer_size)
_write_data(output, buf)
end
else
while size > 0
sz = @buffer_size < size ? @buffer_size : size
buf = input.read(sz)
_write_data(output, buf)
size -= buf.bytesize
end
def write(buf)
return 0 if buf.empty?
socket = @socket
@resp.instance_eval {
size = buf.bytesize
data = "#{size.to_s(16)}#{CRLF}#{buf}#{CRLF}"
_write_data(socket, data)
data.clear
@sent_size += size
size
}
end
def <<(*buf)
write(buf)
self
end
end

65
lib/webrick/httpservlet/filehandler.rb
View File

@ -86,6 +86,35 @@ module WEBrick
return false
end
# returns a lambda for webrick/httpresponse.rb send_body_proc
def multipart_body(body, parts, boundary, mtype, filesize)
lambda do |socket|
begin
begin
first = parts.shift
last = parts.shift
socket.write(
"--#{boundary}#{CRLF}" \
"Content-Type: #{mtype}#{CRLF}" \
"Content-Range: bytes #{first}-#{last}/#{filesize}#{CRLF}" \
"#{CRLF}"
)
begin
IO.copy_stream(body, socket, last - first + 1, first)
rescue NotImplementedError
body.seek(first, IO::SEEK_SET)
IO.copy_stream(body, socket, last - first + 1)
end
socket.write(CRLF)
end while parts[0]
socket.write("--#{boundary}--#{CRLF}")
ensure
body.close
end
end
end
def make_partial_content(req, res, filename, filesize)
mtype = HTTPUtils::mime_type(filename, @config[:MimeTypes])
unless ranges = HTTPUtils::parse_range_header(req['range'])
@ -96,37 +125,27 @@ module WEBrick
if ranges.size > 1
time = Time.now
boundary = "#{time.sec}_#{time.usec}_#{Process::pid}"
body = ''
ranges.each{|range|
first, last = prepare_range(range, filesize)
next if first < 0
io.pos = first
content = io.read(last-first+1)
body << "--" << boundary << CRLF
body << "Content-Type: #{mtype}" << CRLF
body << "Content-Range: bytes #{first}-#{last}/#{filesize}" << CRLF
body << CRLF
body << content
body << CRLF
parts = []
ranges.each {|range|
prange = prepare_range(range, filesize)
next if prange[0] < 0
parts.concat(prange)
}
raise HTTPStatus::RequestRangeNotSatisfiable if body.empty?
body << "--" << boundary << "--" << CRLF
raise HTTPStatus::RequestRangeNotSatisfiable if parts.empty?
res["content-type"] = "multipart/byteranges; boundary=#{boundary}"
res.body = body
if req.http_version < '1.1'
res['connection'] = 'close'
else
res.chunked = true
end
res.body = multipart_body(io.dup, parts, boundary, mtype, filesize)
elsif range = ranges[0]
first, last = prepare_range(range, filesize)
raise HTTPStatus::RequestRangeNotSatisfiable if first < 0
if last == filesize - 1
content = io.dup
content.pos = first
else
io.pos = first
content = io.read(last-first+1)
end
res['content-type'] = mtype
res['content-range'] = "bytes #{first}-#{last}/#{filesize}"
res['content-length'] = last - first + 1
res.body = content
res.body = io.dup
else
raise HTTPStatus::BadRequest
end

31
test/webrick/test_filehandler.rb
View File

@ -14,16 +14,10 @@ class WEBrick::TestFileHandler < Test::Unit::TestCase
end
def get_res_body(res)
body = res.body
if defined? body.read
begin
body.read
ensure
body.close
end
else
body
end
sio = StringIO.new
sio.binmode
res.send_body(sio)
sio.string
end
def make_range_request(range_spec)
@ -75,6 +69,23 @@ class WEBrick::TestFileHandler < Test::Unit::TestCase
res = make_range_response(filename, "bytes=0-0, -2")
assert_match(%r{^multipart/byteranges}, res["content-type"])
body = get_res_body(res)
boundary = /; boundary=(.+)/.match(res['content-type'])[1]
off = filesize - 2
last = filesize - 1
exp = "--#{boundary}\r\n" \
"Content-Type: text/plain\r\n" \
"Content-Range: bytes 0-0/#{filesize}\r\n" \
"\r\n" \
"#{IO.read(__FILE__, 1)}\r\n" \
"--#{boundary}\r\n" \
"Content-Type: text/plain\r\n" \
"Content-Range: bytes #{off}-#{last}/#{filesize}\r\n" \
"\r\n" \
"#{IO.read(__FILE__, 2, off)}\r\n" \
"--#{boundary}--\r\n"
assert_equal exp, body
end
def test_filehandler

33
test/webrick/test_httpresponse.rb
View File

@ -145,5 +145,38 @@ module WEBrick
}
assert_equal 0, logger.messages.length
end
def test_send_body_proc
@res.body = Proc.new { |out| out.write('hello') }
IO.pipe do |r, w|
@res.send_body(w)
w.close
r.binmode
assert_equal 'hello', r.read
end
assert_equal 0, logger.messages.length
end
def test_send_body_proc_chunked
@res.body = Proc.new { |out| out.write('hello') }
@res.chunked = true
IO.pipe do |r, w|
@res.send_body(w)
w.close
r.binmode
assert_equal "5\r\nhello\r\n0\r\n\r\n", r.read
end
assert_equal 0, logger.messages.length
end
def test_set_error
status = 400
message = 'missing attribute'
@res.status = status
error = WEBrick::HTTPStatus[status].new(message)
body = @res.set_error(error)
assert_match(/#{@res.reason_phrase}/, body)
assert_match(/#{message}/, body)
end
end
end

2
version.h
View File

@ -1,6 +1,6 @@
#define RUBY_VERSION "2.2.10"
#define RUBY_RELEASE_DATE "2018-03-28"
#define RUBY_PATCHLEVEL 486
#define RUBY_PATCHLEVEL 487
#define RUBY_RELEASE_YEAR 2018
#define RUBY_RELEASE_MONTH 3