1.5. 3.0.x Branch¶
1.5.1. Upgrade Notes¶
#2228: The default maximum document size has been reduced to 8MB. This means that databases with larger documents will not be able to replicate into CouchDB 3.0 correctly without modification. This change has been made in preparation for anticipated hard upper limits on document size imposed by CouchDB 4.0. For 3.x, the max document size setting can be relaxed via the
[couchdb] max_document_size
config setting.#2228: The default database sharding factor
q
has been reduced to 2 by default. This, combined with automated database resharding (see below), is a better starting place for new CouchDB databases. As in CouchDB 2.x, specify?q=#
to change the value upon database creation if desired. The default can be changed via the config[cluster] q
setting.#1523, #2092, #2336, #2475: The “node-local” HTTP interface, by default exposed on port 5986, has been removed. All functionality previously available at that port is now available on the main, clustered interface (by default, port 5984). Examples:
GET /_node/{nodename}/_stats GET /_node/{nodename}/_system GET /_node/{nodename}/_all_dbs GET /_node/{nodename}/_uuids GET /_node/{nodename}/_config GET /_node/{nodename}/_config/couchdb/uuid POST /_node/{nodename}_config/_reload GET /_node/{nodename}/_nodes/_changes?include_docs=true PUT /_node/{nodename}/_dbs/{dbname} POST /_node/{nodename}/_restart GET /_node/{nodename}/{db-shard} GET /_node/{nodename}/{db-shard}/{doc} GET /_node/{nodename}/{db-shard}/{ddoc}/_info
…and so on. Documentation has been updated to reflect this change.
Warning
The _node endpoint is for adminstrative purposes it is NOT intended as an alternative to the regular endpoints (“GET /dbname”, “PUT /dbname/docid” and so on)
#2389: CouchDB 3.0 now requires a server admin user to be defined at startup, or will print an error message and exit. If you do not have one, be sure to create an admin user. (The Admin Party is now over.)
#2576: CouchDB 3.0 now requires admin-level access for the
/_all_dbs
endpoint.#2339: All databases are now created by default as admin-only. That is, the default new database
_security
object is now:{ "members" : { "roles" : [ "_admin" ] }, "admins" : { "roles" : [ "_admin" ] } }
This can be changed after database creation.
Due to code changes in #2324, it is not possible to upgrade transparently from CouchDB 1.x to 3.x. In addition, the
couchup
utility has been removed from CouchDB 3.0 by #2399. If you are upgrading from CouchDB 1.x, you must first upgrade to CouchDB 2.3.1 to convert your database and indexes, usingcouchup
if desired. You can then upgrade to CouchDB 3.0. Or, you can start a new CouchDB 3.0 installation and replicate directly from 1.x to 3.0.#1833, #2358, #1871, #1857: CouchDB 3.0 supports running only under the following Erlang/OTP versions:
19.x - “soft” support only. No longer tested, but should work.
20.x - must be newer than 20.3.8.11 (20.0, 20.1, 20.2 versions all invalid)
21.x - for 21.2, must be newer than 21.2.3
22.x - for 22.0, must be newer than 22.0.5
#1804: By default, views are limited to return a maximum of 2**28 (268435456) results. This limit can be configured separately for views and partitioned views via the
query_limit
andpartition_query_limit
values in the ini file[query_server_config]
section.After upgrading all nodes in a cluster to 3.0, add
[rexi] use_kill_all = true
tolocal.ini
to save some intra-cluster network bandwidth.
1.5.1.1. Deprecated feature removal¶
The following features, deprecated in CouchDB 2.x, have been removed or replaced in CouchDB 3.0:
#2089, #2128, #2251: Local endpoints for replication targets, which never functioned as expected in CouchDB 2.x, have been completely removed. When replicating databases, always specify a full URL for the source and target. In addition, the node local
_replicator
database is no longer automatically created.#2163: The
disk_size
anddata_size
fields have been retired from the database info object returned byGET /{db}/
. These were deprecated in CouchDB 2.x and replaced by thesizes
object, which contains the improvedfile
,active
andexternal
size metrics. Fauxton has been updated to match.#2173: The ability to submit multiple queries against a view using the POST to
/{db}/_design/{ddoc}/_view/{view}
with the?queries=
option has been replaced by the new queries endpoint. The same is true of the _all_docs, _design_docs, and _local_docs endpoints. Specify akeys
object when POST-ing to these endpoints.#2248: CouchDB externals (
_external/
) have been removed entirely.#2208: CouchDB no longer supports the
delayed_commits
option in the configuration file. All writes are now full commits. The/_ensure_full_commit
API endpoint has been retained (as a no-op) for backwards compatibility with old CouchDB replicators.#2395: The security object in the
_users
database cannot be edited by default. A setting exists in the configuration file to revert this behaviour. The ability to override the disable setting is expected to be removed in CouchDB 4.0.
1.5.1.2. Deprecated feature warnings¶
The following features are deprecated in CouchDB 3.0 and will be removed in CouchDB 4.0:
Show functions (
/{db}/{ddoc}/_show
)List functions (
/{db}/{ddoc}/_list
)Update functions (
/{db}/{ddoc}/_update
)Virtual hosts and ini-file rewrites
Rewrite functions (
/{db}/{ddoc}/_rewrite
)
1.5.2. Version 3.0.1¶
1.5.2.1. Features and Enhancements¶
Fauxton was updated to version v1.2.3.
1.5.2.2. Bugfixes¶
#2441: A memory leak when encoding large binary content was patched. This should resolve a long-standing gradual memory increase bug in CouchDB.
#2613: Simultaneous attempts to create the same new database should no longer result in a 500 Internal Server Error error.
#2678: Defaults for the
smoosh
compaction daemon are now consistent with the shippeddefault.ini
file.#2680: The Windows CouchDB startup batch file will no longer fail to start CouchDB if incompatible versions of OpenSSL are on the
PATH
.#2741: A small performance improvement in the
couch_server
process was made.#2745: The
require_valid_user
exception logic was corrected.#2643: The
users_db_security_editable
setting is now in the correct section of thedefault.ini
file.#2654: Filtered changes feeds that need to rewind partially should no longer rewind all the way to the beginning of the feed.
#2655: When deleting a session cookie, CouchDB should now respect the operator-specified cookie domain, if set.
#2690: Nodes that re-enter a cluster after a database was created (while the node was offline or in maintenance mode) should more correctly handle creating local replicas of that database.
#2805: Mango operators more correctly handle being passed empty arrays.
#2716, #2738: The
remsh
utility will now try and guess the node name and Erlang cookie of the local installation. It will also respect theCOUCHDB_ARGS_FILE
environment variable.#2797: The cluster setup workflow now uses the correct logging module.
#2818: Mango now uses a safer method of bookmark creation that prevents unexpectedly creating new Erlang atoms.
#2756: SpiderMonkey 60+ will no longer corrupt UTF-8 strings when various JS functions are applied to them.
Multiple test case improvements, including more ports of JS tests to Elixir.
1.5.3. Version 3.0.0¶
1.5.3.1. Features and Enhancements¶
#1789: User-defined partitioned databases.
These special databases support user-driven placement of documents into the same shard range. JavaScript views and Mango indexes have specific optimizations for partitioned databases as well.
Two tweakable configuration parameters exist:
#1972, #2012: Automated shard splitting. Databases can now be re-sharded while online to increase the
q
factor to a larger number. This can be configured to require specific node and range parameters upon execution.#1910: Automatic background indexing, internally known as
ken
. This subsystem ensures secondary indexes (such as JavaScript, Mango, and text search) are kept up to date, without requiring an external query to trigger building them. Many configuration parameters are available.#1904: Completely rewritten automatic compaction daemon, internally known as
smoosh
. This subsystem automatically triggers background compaction jobs for both databases and views, based on configurable thresholds.#1889, #2408: New IO Queue subsystem implementation. This is highly configurable and well-documented.
#2436, #2455: CouchDB now regression tests against, and officially supports, running on the
arm64v8
(aarch64
) andppc64le
(ppc64el
) machine architectures. Convenience binaries are generated on these architectures for Debian 10.x (“buster”) packages, and for the Docker containers.#1875, #2437, #2423: CouchDB now supports linking against SpiderMonkey 60 or SpiderMonkey 1.8.5. SpiderMonkey 60 provides enhanced support for ES5, ES6, and ES2016+. Full compatibility information is available at the ECMAScript compatibility table. Click on “Show obsolete platforms”, then look for “FF 60 ESR” in the list of engine types.
However, it was discovered that on some ARM 64-bit distributions, SM 60 segfaults frequently, including the SM 60 packages on CentOS 8 and Debian 10.
As a result, CouchDB’s convenience binaries only link against SM 60 on the ``x86_64`` and ``ppc64le`` architectures. This includes the Docker image for these architectures.
At present, CouchDB ships with SM 60 linked in on the following binary distributions:
Debian buster (10.x)
CentOS / RedHat 8.x
macOS (10.10+)
Windows (7+)
Docker (3.0.0)
FreeBSD (CURRENT)
We expect to add SM 60 support to Ubuntu with Focal Fossa (20.04 LTS) when it ships in April 2020.
It is unlikely we will backport SM 60 packages to older versions of Debian, CentOS, RedHat, or Ubuntu.
The Windows installer has many improvements, including:
Prompts for an admin user/password as CouchDB 3.0 requires * Will not overwrite existing credentials if in place
No longer remove user-modified config files, closing #1989 * Also will not overwrite them on install.
Checkbox to disable installation of the Windows service
Friendly link to these online release notes in the exit dialog
Higher resolution icon for HiDPI (500x500)
Warning
Windows 8, 8.1, and 10 require the .NET Framework v3.5 to be installed.
#2037: Dreyfus, the CouchDB side of the Lucene-powered search solution, is now shipped with CouchDB. When one or more Clouseau Java nodes are joined to the cluster, text-based indexes can be enabled in CouchDB. It is recommended to have as many Clouseau nodes as you have CouchDB nodes. Search is advertised in the feature list present at
GET /
if configured correctly (#2206). Configuration and installation documentation is available.#2411: The
/_up
endpoint no longer requires authentication, even whenrequire_valid_user
istrue
.#2392: A new
_metrics
role can be given to a user. This allows that user access only to the/_node/{node}/_stats
and/_node/{node}/_system
endpoints.#1912: A new alternative
systemd-journald logging
backend has been added, and can be enabled through the ini file. The new backend does not include CouchDB’s microsecond-accurate timestamps, and uses thesd-daemon(3)
logging levels.#2296, #1977: If the configuration file setting
[couchdb]
single_node
is set totrue
, CouchDB will automatically create the system databases on startup if they are not present.#2338, #2343: POST request to CouchDB views and the
/{db}/_all_docs
,/{db}/_local_docs
and/{db}/_design_docs
endpoints now support the same functionality as GET. Parameters are passed in the body as a JSON object, rather than in the URL when using POST.#2292: The
_scheduler/docs
and_scheduler/info
endpoints now return detailed replication stats for running and pending jobs.#2282, #2272, #2290: CouchDB now supports specifying separate proxies for both the
source
andtarget
in a replication viasource_proxy
andtarget_proxy
keys. The API documentation has been updated.#2240: Headers are now returned from the
/{db}/_changes
feed immediately, even when there are no changes available. This avoids client blocking.#2005, #2006: The name of any node can now be retrieved through the new API endpoint
GET /_node/{node-name}
.#1766: Timeouts for requests,
all_docs
, attachments, views, and partitioned view requests can all be specified separately in the ini file under the[fabric]
section. Seedefault.ini
for more detail.#1963: Metrics are now kept on the number of partition and global view queries, along with the number of timeouts that occur.
#2452, #2221: A new configuration field
[couch_httpd_auth] same_site
has been added to set the value of the CouchDB auth cookie’sSameSite
attribute. It may be necessary to set this tostrict
for compatibility with future versions of Google Chrome. If CouchDB CORS support is enabled, set this toNone
.
1.5.3.2. Performance¶
#2277: The
couch_server
process has been highly optimized, supporting significantly more load than before.#2360: It is now possible to make the rexi interface’s unacked message limit configurable. A new, more optimized default (5, lowered from 10) has been set. This results in a ~50% improvement on view queries on large clusters with
q ≥ 8
.#2280: Connection sharing for replication now functions correctly when replicating through a forward proxy. Closes #2271.
#2195, #2207: Metrics aggregation now supports CouchDB systems that sleep or hibernate, ensuring that on wakeup does not trigger thousands of unnecessary function calls.
#1795: Avoid calling
fabric:update_docs
with empty doc lists.#2497: The setup wizard no longer automatically creates the
_global_changes
database, as the majority of users do not need this functionality. This reduces overall CouchDB load.
1.5.3.3. Bugfixes¶
#1752, #2398, #1803: The cluster setup wizard now ensures a consistent UUID and http secret across all nodes in a cluster. CouchDB admin passwords are also synced when the cluster setup wizard is used. This prevents being logged out when using Fauxton as a server admin user through a load balancer.
#2388: A compatibility change has been made to support replication with future databases containing per-document access control fields.
#2379: Any replicator error messages will provide an object in the response, or null, but never a string.
#2244, #2310: CouchDB will no longer send more data than is requested when retrieving partial attachment data blocks.
#2138: Manual operator updates to a database’s shard map will not corrupt additional database properties, such as partitioning values.
#1877: The
_purge
and_purged_infos_limit
endpoints are now correctly restricted to server admin only.#1794: The minimum purge sequence value for a database is now gathered without a clustered
_all_docs
lookup.#2351: A timeout case clause in
fabric_db_info
has been normalised to match other case clauses.#1897: The
/{db}/_bulk_docs
endpoint now correctly catches invalid (i.e., non-hexadecimal)_rev_
values and responds with a 400 Bad Request error.#2321: CouchDB no longer requires Basic auth credentials to reach the
/_session
endpoint for login, even whenrequire_valid_user
is enabled.#2295: CouchDB no longer marks a job as failed permanently if the internal doc processor crashes.
#2178: View compaction files are now removed on view cleanup.
#2179: The error message logged when CouchDB does not have a
_users
database is now less scary.#2153: CouchDB no longer may return a
badmatch
error when queryingall_docs
with a passedkeys
array.#2137: If search is not available, return a 400 Bad Request instead of a 500 Internal Server Error status code.
#2077: Any failed
fsync(2)
calls are now correctly raised to avoid data corruption arising from retry attempts.#2027: Handle epoch mismatch when duplicate UUIDs are created through invalid operator intervention.
#2019: If a database is deleted and re-created while internal cluster replication is still active, CouchDB will no longer retry to delete it continuously.
#2003, #2438: CouchDB will no longer automatically reset an index file if any attempt to read its header fails (such as when the couch_file process terminates unexpectedly). CouchDB now also handles the case when a view file lacks a proper header.
#1983: Improve database “external” size calcuation to be more precise.
#1971: Correctly compare ETags using weak comparison methods to support W/ prefix added by some load balancer configurations.
#1901: Invalid revision specified for a document update will no longer result in a
badarg
crash.#1845: The
end_time
field in/_replicate
now correctly converts time to UTC.#1824:
rexi
stream workers are now cleaned up when the coordinator process is killed, such as when the ddoc cache is refreshed.#1770: Invalid database
_security
objects no longer return afunction_clause
error and stack trace.#2412: Mango execution stats now correctly count documents read which weren’t followed by a match within a given shard.
#2393, #2143: It is now possible to override the query server environment variables
COUCHDB_QUERY_SERVER_JAVASCRIPT
andCOUCHDB_QUERY_SERVER_COFFEESCRIPT
without overwriting thecouchdb
/couchdb.cmd
startup scripts.#2426, #2415: The replicator now better handles the situation where design document writes to the target fail when replicating with non-admin credentials.
#2444, #2413: Replicator error messages are now significantly improved, reducing
function_clause
responses.#2454: The replication auth session plugin now ignores other cookies it may receive without logging an error.
#2458: Partitioned queries and dreyfus search functions no longer fail if there is a single failed node or rexi worker error.
#1783: Mango text indexes no longer error when given an empty selector or operators with empty arrays.
#2466: Mango text indexes no longer error if the indexed document revision no longer exists in the primary index.
#2486: The
$lt
,$lte
,$gt
, and$gte
Mango operators are correctly quoted internally when used in conjunction with a text index search.#2493: The
couch_auth_cache
no longer has a runaway condition in which it creates millions of monitors on the_users
database.
1.5.3.4. Other¶
The 3.0.0 release also includes the following minor improvements:
#2472: CouchDB now logs the correct, clustered URI at startup (by default: port
5984
.)#2034, #2416: The path to the Fauxton installation can now be specified via the
COUCHDB_FAUXTON_DOCROOT
environment variable.#2447: Replication stats are both persisted when jobs are re-created, as well as properly handled when bulk document batches are split.
#2410, #2390, #1913: Many metrics were added for Mango use, including counts of unindexed queries, invalid index queries, docs examined that do and don’t meet cluster quorum, query time, etc.
#2152, #2504: CouchDB can now be started via a symlink to the binary on UNIX-based platforms.
#1844: A new internal API has been added to write custom Erlang request-level metrics reporting plugins.
#2293, #1095: The
-args_file
,-config
and-couch_ini
parameters may now be overridden via theCOUCHDB_INI_FILES
environment variable on UNIX-based systems.#2352: The
remsh
utility now searches for the Erlang cookie inERL_FLAGS
as well asvm.args
.#2324: All traces of the (never fully functional) view-based
_changes
feed have been expunged from the code base.#2337: The md5 shim (introduced to support FIPS-compliance) is now used consistently throughout the code base.
#2270: Negative and non-integer
heartbeat
values now return 400 Bad Request.#2268: When rescheduling jobs, CouchDB now stops sufficient running jobs to make room for the pending jobs.
#2186: CouchDB plugin writers have a new field in which endpoint credentials may be stashed for later use.
#2183:
dev/run
now supports an--extra-args
flag to modify the Erlang runtime environment during development.#2105:
dev/run
no longer fails on unexpected remote end connection close during cluster setup.#2118: Improve
couch_epi
process replacement mechanism using map childspecs functionality in modern Erlang.#2111: When more than
MaxJobs
replication jobs are defined, CouchDB now correctly handles job rotation when some jobs crash.#2020: Fix full ring assertion in fabric stream shard replacements
#1925: Support list for docid when using
couch_db:purge_docs/3
.#1642:
io_priority
is now set properly on view update and compaction processes.#1865: Purge now supports >100 document IDs in a single request.
#1861: The
vm.args
file has improved commentary.#1808: Pass document update type for additional checks in
before_doc_update
.#1835: Module lists are no longer hardcoded in
.app
files.#1798, #1933: Multiple compilation warnings were eliminated.
#1826: The
couch_replicator_manager
shim has been fully removed.#1820: After restarting CouchDB, JS and Elixir tests now wait up to 30s for it to be ready before timing out.
#1800:
make elixir
supports specifying individual tests to run withtests=
.#1805:
dev/run
supports--with-haproxy
again.#1774:
dev/run
now supports more than 3 nodes.#1779: Refactor Elixir test suite initialization.
#1769: The Elixir test suite uses Credo for static analysis.
#1776: All Python code is now formatted using Python black.
#1786:
dev/run
: do not create needlessdev/data/
directory.#2482: A redundant
get_ring_opts
call has been removed fromdreyfus_fabric_search
.#2506: CouchDB’s release candidates no longer propagate the RC tags into each Erlang application’s version string.
#2511: recon, the Erlang diagnostic toolkit, has been added to CouchDB’s build process and ships in the release + convenience binaries.
Fauxton updated to v1.2.3, which includes:
Support multiple server-generated warnings when running queries
Partitioned database support
Search index support
Remove references to deprecated dbinfo fields
Improve accessibility for screen readers
Numerous CSS fixes
Improved test cases:
Many, many test race conditions and bugs have been removed (PR list too long to include here!)
More test cases were ported to Elixir, including:
#2431:
chttpd_purge_tests
have been improved in light of CI failures.#2432: Address flaky test failure on
t_invalid_view/1
.#2363: Elixir tests now run against a single node cluster, in line with the original design of the JavaScript test suite. This is a permanent change.
#1893: Add “w:3” for lots of doc tests.
#1939, #1931: Multiple fixes to improve support in constrained CI environments.
#2346: Big-endian support for the
couch_compress
tests.#2314: Do not auto-index when testing
update=false
in Mango.#2141: Fix
couch_views
encoding test.#2123: Timeout added for
fold_docs-with_different_keys
test.#2114: EUnit tests now correctly inherit necessary environment variables.
#2122: :meck.unload() is now called automatically after every test.
#2098: Fix
cpse_test_purge_replication
eunit test.#2084: Increase timeouts on two slow btree tests.
#1922: Tests added for shard splitting functionality.
#1869: New test added for doc reads with etag
If-None-Match
header.#1831: Re-introduced cpse_test_purge_seqs test.
#1790: Reorganise
couch_flag_config_tests
into a proper suite.#1785: Use
devclean
on elixir target for consistency of Makefile.#2476: For testing,
Triq
has been replaced withPropEr
as an optional dependency.
External dependency updates:
A llama! OK, no, not really. If you got this far…thank you for reading.