We were generating two whole sets of jobs, for --tests and --docs
separately. But the default invocation of cargo runs both tests and
doctests. We can save some resources and speed up our test runs by
allowing cargo to do both in one invocation.
It turns out Headers is actually an internal-only API. None of the
user-facing types use it.
Unfortunately, making it unexported also required deleting the doctests,
since doctests can only run against a public interface.
Followup to #56. At the time, doing this would have been an API
break; but we can do this as part of 2.0. This simplifies the API
nicely and creates better symmetry between sending bodies and
receiving them.
This adds a source field to keep track of upstream errors and allow
backtraces, plus a URL field to indicate what URL an error was
associated with.
The enum variants we used to use for Error are now part of a new
ErrorKind type. For convenience within ureq, ErrorKinds can be turned
into an Error with `.new()` or `.msg("some additional information")`.
Error acts as a builder, so additional information can be added after
initial construction. For instance, we return a DnsFailed error when
name resolution fails. When that error bubbles up to Request's
`do_call`, Request adds the URL.
Fixes#232.
This makes src/lib.rs the primary source for crate-level documentation.
I've generated README.md with `cargo readme > README.md`. Since links to
specific documentation items should be relative when possible, but must
be absolute in README.md, I've used the new syntax for intra-rustdoc
links
(https://github.com/rust-lang/rfcs/blob/master/text/1946-intra-rustdoc-links.md),
along with a README.tpl that sets up those links to point at the
absolute versions. `cargo readme` uses the README.tpl by default.
I've also rewritten the crate level docs, removing some TODO information
at the bottom, and moving the license information to CONTRIBUTING.md.
This makes src/lib.rs the primary source for crate-level documentation.
I've generated README.md with `cargo readme > README.md`. Since links to
specific documentation items should be relative when possible, but must
be absolute in README.md, I've used the new syntax for intra-rustdoc
links
(https://github.com/rust-lang/rfcs/blob/master/text/1946-intra-rustdoc-links.md),
along with a README.tpl that sets up those links to point at the
absolute versions. `cargo readme` uses the README.tpl by default.
I've also rewritten the crate level docs, removing some TODO information
at the bottom, and moving the license information to CONTRIBUTING.md.
I missed these in my previous doctest PR.
The doctests all now run without accessing the network. Tested by
turning off networking and running them.
Request.call's doctest wouldn't run because it relied on making a custom
AgentBuilder and building it, which bypasses the test_agent. I concluded
that this doctest was mostly illustrating behavior of AgentBuilder, not
call(), and simplified it to be more like the other calling methods on
request.
Doctests run against a normally-built copy of the crate, i.e. one
without #[cfg(test)] set, so we can't use the conditional compilation
feature.
Instead, define a static var that indicates whether the library is
running in test mode or not. For each doctest, insert a hidden call that
sets this var to true. Then, when ureq::agent() is called, it returns a
test_agent instead.
This required moving testserver out of the test mod and into src/, so
that it can be included unconditionally (i.e. when cfg(test) is false).
This PR converts one doctest as an example. If we land this PR, I'll
send a followup to convert the rest.
Instead, rely on Url's built-in query parameter handling. A Request now
accumulates a list of query param pairs, and joins them with a parsed
URL at the time do_call is called.
In the process, remove some getters that rely on parsing the URL.
Adapting these getters was going to be awkward, and they mostly
duplicate things people can readily get by parsing the URL.
Replace `this` idiom with `mut self`.
Move idle connections constants from pool.rs to agent.rs.
Remove Agent.set and some convenience request methods
(leaving get, post, and put).
Move max_idle_connections setting from AgentConfig to AgentBuilder
(since the builder passes these to ConnectionPool and the Agent
doesn't subsequently need them).
Eliminate duplicate copy of proxy in AgentState; use the one in
AgentConfig.
* Remove Request::build
* All mutations on Request follow builder pattern
The previous `build()` on request was necessary because mutating
functions did not follow a proper builder pattern (taking `&mut self`
instead of `mut self`). With a proper builder pattern, the need for
`.build()` goes away.
* All Request body and call methods consume self
Anything which "executes" the request will now consume the `Request`
to produce a `Result<Response>`.
* Move all config from request to agent builder
Timeouts, redirect config, proxy settings and TLS config are now on
`AgentBuilder`.
* Rename max_pool_connections -> max_idle_connections
* Rename max_pool_connections_per_host -> max_idle_connections_per_host
Consistent internal and external naming.
* Introduce new AgentConfig for static config created by builder.
`Agent` can be seen as having two parts. Static config and a mutable
shared state between all states. The static config goes into
`AgentConfig` and the mutable shared state into `AgentState`.
* Replace all use of `Default` for `new`.
Deriving or implementing `Default` makes for a secondary instantiation
API. It is useful in some cases, but gets very confusing when there
is both `new` _and_ a `Default`. It's especially devious for derived
values where a reasonable default is not `0`, `false` or `None`.
* Remove feature native_tls, we want only native rustls.
This feature made for very clunky handling throughout the code. From a
security point of view, it's better to stick with one single TLS API.
Rustls recently got an official audit (very positive).
https://github.com/ctz/rustls/tree/master/audit
Rustls deliberately omits support for older, insecure TLS such as TLS
1.1 or RC4. This might be a problem for a user of ureq, but on balance
not considered important enough to keep native_tls.
* Remove auth and support for basic auth.
The API just wasn't enough. A future reintroduction should at least
also provide a `Bearer` mechanism and possibly more.
* Rename jar -> cookie_store
* Rename jar -> cookie_tin
Just make some field names sync up with the type.
* Drop "cookies" as default feature
The need for handling cookies is probably rare, let's not enable it by
default.
* Change all feature checks for "cookie" to "cookies"
The outward facing feature is "cookies" and I think it's better form
that the code uses the official feature name instead of the optional
library "cookies".
* Keep `set` on Agent level as well as AgentBuilder.
The idea is that an auth exchange might result in a header that need
to be set _after_ the agent has been built.
* Change ordering on TestServer's atomic bool to SeqCst
This feature was broken in #67, which reset timeouts on the
stream before passing it to set_stream.
As part of this change, refactor the internal storage of
timeouts on the Request object to use Option<Duration>.
Remove the deadline field on Response. It wasn't used. The
deadline field on unit was used instead.
Add a unittest.
Previously, Agent stored most of its state in one big
Arc<Mutex<AgentState>>. This separates the Arc from the Mutexes.
Now, Agent is a thin wrapper around an Arc<AgentState>. The individual
components that need locking, ConnectionPool and CookieStore, now are
responsible for their own locking.
There were a couple of reasons for this. Internal components that needed
an Agent were often instead carrying around an Arc<Mutex<AgentState>>.
This felt like the components were too intertwined: those other
components shouldn't have to care quite so much about how Agent is
implemented. Also, this led to compromises of convenience: the Proxy on
Agent wound up stored inside the `Arc<Mutex<AgentState>>` even though it
didn't need locking. It was more convenient that way because that was
what Request and Unit had access too.
The other reason to push things down like this is that it can reduce
lock contention. Mutations to the cookie store don't need to lock the
connection pool, and vice versa. This was a secondary concern, since I
haven't actually profiled these things and found them to be a problem,
but it's a happy result of the refactoring.
Now all the components outside of Agent take an Agent instead of
AgentState.
In the process I removed `Agent.cookie()`. Its API was hard to use
correctly, since it didn't distinguish between cookies on different
hosts. And it would have required updates as part of this refactoring.
I'm open to reinstating some similar functionality with a refreshed API.
I kept `Agent.set_cookie`, but updated its method signature to take a
URL as well as a cookie.
Many of ConnectionPool's methods went from `&mut self` to `&self`,
because ConnectionPool is now using interior mutability.
This is a step towards allowing our tests to run without network access,
which will make them more resilient and faster.
Replace the URL in one instance of an HTTPS test that didn't need HTTPS.
In the process, rename set_foo methods to just foo, since methods on the
builder will always be setters.
Adds a new() method on ConnectionPool so it can be constructed directly
with the desired limits. Removes the setter methods on ConnectionPool
for those limits. This means that connection limits can only be set when
an Agent is built.
There were two tests that verify Send and Sync implementations, one for
Agent and one for Request. This PR moves the Request test to request.rs,
and changes both tests to more directly verify the traits. There may be
another way to do this, I'm not sure.
This adds validation of header values on receive, and of both header
names and header values on send. This doesn't change the return
type of set to be a Result, it just validates when the request is
sent. Also removes the section in the README describing handling
of invalid headers, and updates a test that verified acceptance of
non-ASCII headers so that it verifies rejection of them instead.
Gets rid of synthetic_error, and makes the various send_* methods return `Result<Response, Error>`.
Introduces a new error type "HTTP", which represents an error due to status codes 4xx or 5xx.
The HTTP error type contains a boxed Response, so users can read the actual response if they want.
Adds an `error_for_status` setting to disable the functionality of treating 4xx and 5xx as errors.
Adds .unwrap() to a lot of tests.
Fixes#128.
CONNECT, specified at https://tools.ietf.org/html/rfc7231#section-4.3.6,
says:
> if successful, thereafter restrict its behavior to blind forwarding
> of packets, in both directions, until the tunnel is closed.
ureq doesn't actually support the semantics of CONNECT, since it doesn't
offer a bidirectional channel on a Response. So I'm fairly confident no
one is using these methods.
CookieJar doesn't support the path-match and domain-match algorithms from [RFC 6265](https://tools.ietf.org/html/rfc6265#section-5.1.3), while cookie_store does.
This fixes some issues with the cookie matching algorithm currently in ureq. For instance,
the domain-match uses substring matching rather than the RFC 6265 algorithm.
This deletes two tests:
match_cookies_returns_nothing_when_no_cookies didn't test much
agent_cookies was failing because cookie_store rejects cookies on the `test:` scheme.
The way around this is to set up a testserver - but it turns out cookies_on_redirect already
does that, and covers the same cases and more.
This changes some cookie-related behavior:
- Cookies could previously be sent to a wrong domain - e.g. a cookie set on `example.com`
could go to `example.com.evil.com` or `evilexample.com`. Probably no one was relying on
this, since it's quite broken.
- A cookie with a path of `/foo` could be sent on a request to `/foobar`, but now it can't.
- Cookies could previously be set on IP addresses, but now they can't.
- Cookies could previously be set for domains other than the one on the request (or its
parents), but now they can't.
- When a cookie had no domain attribute, it would previously get the domain from the
request, and subsequently be sent to that domain and all subdomains. Now, it will only
be sent to that exact domain (host-only).
That last one is probably the most likely to break people, since someone could depend
on it without realizing it was broken behavior.
This emulates the test matrix that gets run in CI, making it easier to
find failures locally.
There was one conflict in the matrix: when JSON is enabled and TLS is
disabled, two of the doctests would fail. This was previous worked
around as an exclude in the github workflow. I changed the JSON doctest
to use HTTP instead.
Previously we had a special case for BadStatusRead that would happen
only when we got a ConnectionAborted error reading the status line.
However, sometimes we get ConnectionReset instead. Also the HTTP
spec says that idempotent requests may be retried anytime a connection
is closed prematurely.
The change treats as retryable any ConnectionAborted OR ConnectionReset
error while reading the status line and headers. It removes the special
case BadStatusRead error.
Fixes#165 (I think).
Previously any io::Error on reading a status line or response headers
would be unconditionally mapping into BadStatus or BadHeader. I think
it's better to pass through the actual io::Error.
BadStatusRead is still kept, since it has special status when dealing
with timed out connections, and BadStatus is still used when the status
line is malformed.
In a few places we relied on "localhost" as a default if a URL's host
was not set, but I think it's better to error out in these cases.
In general, there are a few places in Unit that assumed there is a
host as part of the URL. I've made that explicit by doing a check
at the beginning of `connect()`. I've also tried to plumb through
the semantics of "host is always present" by changing the parameter
types of some of the functions that use the hostname.
I considered a more thorough way to express this with types - for
instance implementing an `HttpUrl` struct that embeds a `Url`, and
exports most of the same methods, but guarantees that host is always
present. However, that was more invasive than this so I did a smaller
change to start.
We only want streams in the pool that are ready for the next request,
and don't have any remaining bytes of the previous request. That's the
case when we return a stream due to EOF on the underlying `Read`.
However, there was also an `impl Drop` that returned the stream to the
pool on drop of a PoolReturnRead. That resulted in a bug: If a user
called `response.into_reader()`, but did not read the entire body,
a dirty stream would be returned to the pool. The next time a request
was made to the same host, that stream would be reused, and instead of
reading off "HTTP/1.1 200 OK", ureq would read a remnant of the previous
request, resulting in a BadStatus error.
This broke during some recent refactorings. The special case branch for
max_connections == 0 wasn't actually setting the max_idle_connections
field.
This change simply deletes that branch, since the existing code that
whittles down the pool size one element at a time suffices. In the
common case this will be set on an empty pool anyhow.
Now that we allow multiple connections for the same PoolKey, update our
invariants accordingly. Also provide a couple of helper functions for
removing the first or last match of an entry in a VecDeque.
This also changes which entry from `lru` gets removed when a stream is
removed from the pool. Previously it was the oldest matching one. Now it's
the newest matching one, which matches the semantics we are applying to
`recycle[K]`.
Followup to #133
/cc @cfal
Some helper functions introduced as part of the cookies_on_redirect
test are only used by that test, which means they are only used when the
cookie feature is enabled. This causes clippy warnings for some of the
other feature combinations in CI.
Previously this test relied on a specific ordering of cookies, which was
not guaranteed. This makes the test robust to random ordering.
Also, fix the check for the `second` cookie to actually check the right
cookie.
* Put the crate version in the User-Agent header.
https://tools.ietf.org/html/rfc7231#section-5.5.3
Each product identifier consists of a name and optional version.
product = token ["/" product-version]
product-version = token
* User-Agent of rust-ureq
* Update src/unit.rs
Co-authored-by: Martin Algesten <martin@algesten.se>
This doesn't change the API at all, since it delays any errors until do_call.
Since there was already an `Into<Response> for Error`, and do_call already
had a `.unwrap_or_else(|e| e.into())`, nothing in do_call needed to change.
The only visible effect of this is that code that was previously relying on
`get("/path")` to fetch something from localhost will now get an error. I think
most of those cases would probably actually be accidents, possibly caused
by typos, e.g. `get("example.com/path")` (forgetting the https: prefix).
Fixes#105
