Previously, `ureq` used `HTTP CONNECT` for *all* requests, including
plain-text HTTP requests. However, some proxy servers like Squid
only allow tunneling via the `HTTP CONNECT` method on port 443 - since
it is usually only used to proxy HTTPS requests. As a result,
it was not possible to use `ureq` with a Squid server in its default configuration.
With the changes from this commit, ureq will interact with HTTP proxies
in a more standard-conforming way, where `CONNECT` is only used for
HTTPS requests. HTTP request paths are transformed in such a way that they
comply with RFC 7230 [1].
Tested against Squid 4.13 in standard configuration, with and without basic authentication,
for HTTP and HTTPS requests.
[1] https://www.rfc-editor.org/rfc/rfc7230#section-5.3.2
This breaks a reference cycle between PoolReturner and Agent that was
causing Agents (and their contained ConnectionPool) to never be dropped
so long as there was any stream in the ConnectionPool. This cause
sockets to leak over time, particularly when the convenience functions
ureq::get(), ureq::post(), etc were used, since those functions create
a new Agent each time.
Introduce PoolReturner, a handle on an agent and a PoolKey that is
capable of returning a Stream to a Pool. Make Streams keep track of
their own PoolReturner, instead of having PoolReturnRead keep track of
that information.
For the LimitedRead code path, get rid of PoolReturnRead. Instead,
LimitedRead is responsible for returning its Stream to the Pool after
its second-to-last read. In other words, LimitedRead will return the
stream if the next read is guaranteed to return Ok(0).
Constructing a LimitedRead of size 0 is always wrong, because we could
always just return the stream immediately. Change the size argument to
NonZeroUsize to enforce that.
Remove the Done trait, which was only used for LimitedRead. It was used
to try and make sure we returned the stream to the pool on exact reads,
but was not reliable.
This does not yet move the ChunkDecoder code path away from
PoolReturnRead. That requires a little more work.
Part 1 of #559. Fixes#555.
This allows removing the hack where we create a Response with an empty `reader`,
then immediately mutate it to set the real reader. It also happens to allow us
to get rid of 3 fields of Response that were only used to pass information to
`stream_to_reader`.
I've tried to keep the structure and logic of the body calculation as close to
the same as possible, for ease of review and to avoid introducing bugs. I think
there are some followup fixes we can make to the logic, which will be made
easier by having it in a self contained function.
This is more correct, since all gzip streams can consist of multiple members. It
has the happy side effect that it causes gzipped responses to reliably return
their streams to the pool.
The mbedtls example has caused problem in the main build a number of
times. By making it a standalone `cargo new --bin`, we can keep it in
the source tree as a good example but avoid having it break the main
build.
Also, fix some clippy lints.
* Response: build reader at construction time
* Remove unwrapping/rewrapping DeadlineStream
* Let into_reader() provide Sync + 'static
* Prebuffer Vec use known capacity
Co-authored-by: Martin Algesten <martin@algesten.se>
Previously, ReadWrite had methods `is_poolable` and `written_bytes`, which
were solely for the use of unittests.
This replaces `written_bytes` and `TestStream` with a `struct Recorder`
that implements `ReadWrite` and allows unittests to access its recorded
bytes via an `Arc<Mutex<Vec<u8>>>`. It eliminates `is_poolable`; it's fine
to pool a Stream of any kind.
The new `Recorder` also has some convenience methods that abstract away
boilerplate code from many of our unittests.
I got rid of `Stream::from_vec` and `Stream::from_vec_poolable` because
they depended on `TestStream`. They've been replaced by `NoopStream` for
the pool.rs tests, and `ReadOnlyStream` for constructing `Response`s from
`&str` and some test cases.
We unwrap the stream exactly once per response, and we know that case
will be uncontended for the same reason `SyncWrapper` works:
`into_reader()` takes `self`, so it must have exclusive ownership.
Uncontended mutexes are extremely cheap. This saves us a dependency
at a trivial performance cost.
This fixes an issue where requests with a deadline would do a syscall
on every read, rather than pulling from the buffer. Fixes a problem
reported in #506 where `resp.into_json()` was unnecessarily slow.
If the user reads exactly the number of bytes in the response, then
drops the Read object, streams will never get returned to the pool since
the user never triggered a read past the end of the LimitedRead.
This fixes that by making PoolReturnRead aware of the level below it, so
it can ask if a stream is "done" without actually doing a read.
Also, a refactorign:
Previously, Response contained an Option<Box<Unit>> because the testing
method `from_str()` would construct a Response with no associated Unit.
However, this increased code complexity with no corresponding test
benefit. Instead, construct a fake Unit in from_str().
Also, instead of taking an `Option<Box<Unit>>`, PoolReturnRead now takes
a URL (to figure out host and port for the PoolKey) and an &Agent where
it will return the stream. This cuts interconnectedness somewhat:
PoolReturnRead doesn't need to know about Unit anymore.
