Misc¶
ACLs¶
- swift.common.middleware.acl.acls_from_account_info(info)¶
Extract the account ACLs from the given account_info, and return the ACLs.
- Parameters:
info – a dict of the form returned by get_account_info
- Returns:
None (no ACL system metadata is set), or a dict of the form:: {‘admin’: […], ‘read-write’: […], ‘read-only’: […]}
- Raises:
ValueError – on a syntactically invalid header
- swift.common.middleware.acl.clean_acl(name, value)¶
Returns a cleaned ACL header value, validating that it meets the formatting requirements for standard Swift ACL strings.
The ACL format is:
[item[,item...]]
Each item can be a group name to give access to or a referrer designation to grant or deny based on the HTTP Referer header.
The referrer designation format is:
.r:[-]value
The
.r
can also be.ref
,.referer
, or.referrer
; though it will be shortened to just.r
for decreased character count usage.The value can be
*
to specify any referrer host is allowed access, a specific host name likewww.example.com
, or if it has a leading period.
or leading*.
it is a domain name specification, like.example.com
or*.example.com
. The leading minus sign-
indicates referrer hosts that should be denied access.Referrer access is applied in the order they are specified. For example, .r:.example.com,.r:-thief.example.com would allow all hosts ending with .example.com except for the specific host thief.example.com.
Example valid ACLs:
.r:* .r:*,.r:-.thief.com .r:*,.r:.example.com,.r:-thief.example.com .r:*,.r:-.thief.com,bobs_account,sues_account:sue bobs_account,sues_account:sue
Example invalid ACLs:
.r: .r:-
By default, allowing read access via .r will not allow listing objects in the container – just retrieving objects from the container. To turn on listings, use the .rlistings directive.
Also, .r designations aren’t allowed in headers whose names include the word ‘write’.
ACLs that are “messy” will be cleaned up. Examples:
Original
Cleaned
bob, sue
bob,sue
bob , sue
bob,sue
bob,,,sue
bob,sue
.referrer : *
.r:*
.ref:*.example.com
.r:.example.com
.r:*, .rlistings
.r:*,.rlistings
- Parameters:
name – The name of the header being cleaned, such as X-Container-Read or X-Container-Write.
value – The value of the header being cleaned.
- Returns:
The value, cleaned of extraneous formatting.
- Raises:
ValueError – If the value does not meet the ACL formatting requirements; the error message will indicate why.
- swift.common.middleware.acl.format_acl(version=1, **kwargs)¶
Compatibility wrapper to help migrate ACL syntax from version 1 to 2. Delegates to the appropriate version-specific format_acl method, defaulting to version 1 for backward compatibility.
- Parameters:
kwargs – keyword args appropriate for the selected ACL syntax version (see
format_acl_v1()
orformat_acl_v2()
)
- swift.common.middleware.acl.format_acl_v1(groups=None, referrers=None, header_name=None)¶
Returns a standard Swift ACL string for the given inputs.
Caller is responsible for ensuring that :referrers: parameter is only given if the ACL is being generated for X-Container-Read. (X-Container-Write and the account ACL headers don’t support referrers.)
- Parameters:
groups – a list of groups (and/or members in most auth systems) to grant access
referrers – a list of referrer designations (without the leading .r:)
header_name – (optional) header name of the ACL we’re preparing, for clean_acl; if None, returned ACL won’t be cleaned
- Returns:
a Swift ACL string for use in X-Container-{Read,Write}, X-Account-Access-Control, etc.
- swift.common.middleware.acl.format_acl_v2(acl_dict)¶
Returns a version-2 Swift ACL JSON string.
- HTTP headers for Version 2 ACLs have the following form:
Header-Name: {“arbitrary”:”json”,”encoded”:”string”}
JSON will be forced ASCII (containing six-char uNNNN sequences rather than UTF-8; UTF-8 is valid JSON but clients vary in their support for UTF-8 headers), and without extraneous whitespace.
Advantages over V1: forward compatibility (new keys don’t cause parsing exceptions); Unicode support; no reserved words (you can have a user named .rlistings if you want).
- Parameters:
acl_dict – dict of arbitrary data to put in the ACL; see specific auth systems such as tempauth for supported values
- Returns:
a JSON string which encodes the ACL
- swift.common.middleware.acl.parse_acl(*args, **kwargs)¶
Compatibility wrapper to help migrate ACL syntax from version 1 to 2. Delegates to the appropriate version-specific parse_acl method, attempting to determine the version from the types of args/kwargs.
- Parameters:
args – positional args for the selected ACL syntax version
kwargs – keyword args for the selected ACL syntax version (see
parse_acl_v1()
orparse_acl_v2()
)
- Returns:
the return value of
parse_acl_v1()
orparse_acl_v2()
- swift.common.middleware.acl.parse_acl_v1(acl_string)¶
Parses a standard Swift ACL string into a referrers list and groups list.
See
clean_acl()
for documentation of the standard Swift ACL format.- Parameters:
acl_string – The standard Swift ACL string to parse.
- Returns:
A tuple of (referrers, groups) where referrers is a list of referrer designations (without the leading .r:) and groups is a list of groups to allow access.
- swift.common.middleware.acl.parse_acl_v2(data)¶
Parses a version-2 Swift ACL string and returns a dict of ACL info.
- Parameters:
data – string containing the ACL data in JSON format
- Returns:
A dict (possibly empty) containing ACL info, e.g.: {“groups”: […], “referrers”: […]}
- Returns:
None if data is None, is not valid JSON or does not parse as a dict
- Returns:
empty dictionary if data is an empty string
- swift.common.middleware.acl.referrer_allowed(referrer, referrer_acl)¶
Returns True if the referrer should be allowed based on the referrer_acl list (as returned by
parse_acl()
).See
clean_acl()
for documentation of the standard Swift ACL format.- Parameters:
referrer – The value of the HTTP Referer header.
referrer_acl – The list of referrer designations as returned by
parse_acl()
.
- Returns:
True if the referrer should be allowed; False if not.
Buffered HTTP¶
Monkey Patch httplib.HTTPResponse to buffer reads of headers. This can improve performance when making large numbers of small HTTP requests. This module also provides helper functions to make HTTP connections using BufferedHTTPResponse.
Warning
If you use this, be sure that the libraries you are using do not access the socket directly (xmlrpclib, I’m looking at you :/), and instead make all calls through httplib.
- class swift.common.bufferedhttp.BufferedHTTPConnection(host, port=None, timeout=<object object>, source_address=None)¶
Bases:
HTTPConnection
HTTPConnection class that uses BufferedHTTPResponse
- connect()¶
Connect to the host and port specified in __init__.
- getresponse()¶
Get the response from the server.
If the HTTPConnection is in the correct state, returns an instance of HTTPResponse or of whatever object is returned by the response_class variable.
If a request has not been sent or if a previous response has not be handled, ResponseNotReady is raised. If the HTTP response indicates that the connection should be closed, then it will be closed before the response is returned. When the connection is closed, the underlying socket is closed.
- putheader(header, value)¶
Send a request header line to the server.
For example: h.putheader(‘Accept’, ‘text/html’)
- putrequest(method, url, skip_host=0, skip_accept_encoding=0)¶
Send a request to the server.
- Parameters:
method – specifies an HTTP request method, e.g. ‘GET’.
url – specifies the object being requested, e.g. ‘/index.html’.
skip_host – if True does not add automatically a ‘Host:’ header
skip_accept_encoding – if True does not add automatically an ‘Accept-Encoding:’ header
- response_class¶
alias of
BufferedHTTPResponse
- class swift.common.bufferedhttp.BufferedHTTPResponse(sock, debuglevel=0, strict=0, method=None)¶
Bases:
HTTPResponse
HTTPResponse class that buffers reading of headers
- close()¶
Flush and close the IO object.
This method has no effect if the file is already closed.
- nuke_from_orbit()¶
Terminate the socket with extreme prejudice.
Closes the underlying socket regardless of whether or not anyone else has references to it. Use this when you are certain that nobody else you care about has a reference to this socket.
- read(amt=None)¶
Read and return up to n bytes.
If the argument is omitted, None, or negative, reads and returns all data until EOF.
If the argument is positive, and the underlying raw stream is not ‘interactive’, multiple raw reads may be issued to satisfy the byte count (unless EOF is reached first). But for interactive raw streams (as well as sockets and pipes), at most one raw read will be issued, and a short result does not imply that EOF is imminent.
Returns an empty bytes object on EOF.
Returns None if the underlying raw stream was open in non-blocking mode and no data is available at the moment.
- readline(size=1024)¶
Read and return a line from the stream.
If size is specified, at most size bytes will be read.
The line terminator is always b’n’ for binary files; for text files, the newlines argument to open can be used to select the line terminator(s) recognized.
- swift.common.bufferedhttp.http_connect(ipaddr, port, device, partition, method, path, headers=None, query_string=None, ssl=False)¶
Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
- Parameters:
ipaddr – IPv4 address to connect to
port – port to connect to
device – device of the node to query
partition – partition on the device
method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.)
path – request path
headers – dictionary of headers
query_string – request query string
ssl – set True if SSL should be used (default: False)
- Returns:
HTTPConnection object
- swift.common.bufferedhttp.http_connect_raw(ipaddr, port, method, path, headers=None, query_string=None, ssl=False)¶
Helper function to create an HTTPConnection object. If ssl is set True, HTTPSConnection will be used. However, if ssl=False, BufferedHTTPConnection will be used, which is buffered for backend Swift services.
- Parameters:
ipaddr – IPv4 address to connect to
port – port to connect to
method – HTTP method to request (‘GET’, ‘PUT’, ‘POST’, etc.)
path – request path
headers – dictionary of headers
query_string – request query string
ssl – set True if SSL should be used (default: False)
- Returns:
HTTPConnection object
Constraints¶
- swift.common.constraints.check_account_format(req, name, *, target_type='Account')¶
Validate that the header contains valid account or container name.
- Parameters:
req – HTTP request object
name – header value to validate
target_type – which header is being validated (Account or Container)
- Returns:
A properly encoded account name or container name
- Raises:
HTTPPreconditionFailed – if account header is not well formatted.
- swift.common.constraints.check_container_format(req, name, *, target_type='Container')¶
Validate that the header contains valid account or container name.
- Parameters:
req – HTTP request object
name – header value to validate
target_type – which header is being validated (Account or Container)
- Returns:
A properly encoded account name or container name
- Raises:
HTTPPreconditionFailed – if account header is not well formatted.
- swift.common.constraints.check_delete_headers(request)¶
Check that ‘x-delete-after’ and ‘x-delete-at’ headers have valid values. Values should be positive integers and correspond to a time greater than the request timestamp.
If the ‘x-delete-after’ header is found then its value is used to compute an ‘x-delete-at’ value which takes precedence over any existing ‘x-delete-at’ header.
- Parameters:
request – the swob request object
- Raises:
HTTPBadRequest in case of invalid values
- Returns:
the swob request object
- swift.common.constraints.check_dir(root, drive)¶
Verify that the path to the device is a directory and is a lesser constraint that is enforced when a full mount_check isn’t possible with, for instance, a VM using loopback or partitions.
- Parameters:
root – base path where the dir is
drive – drive name to be checked
- Returns:
full path to the device
- Raises:
ValueError – if drive fails to validate
- swift.common.constraints.check_drive(root, drive, mount_check)¶
Validate the path given by root and drive is a valid existing directory.
- Parameters:
root – base path where the devices are mounted
drive – drive name to be checked
mount_check – additionally require path is mounted
- Returns:
full path to the device
- Raises:
ValueError – if drive fails to validate
- swift.common.constraints.check_float(string)¶
Helper function for checking if a string can be converted to a float.
- Parameters:
string – string to be verified as a float
- Returns:
True if the string can be converted to a float, False otherwise
- swift.common.constraints.check_metadata(req, target_type)¶
Check metadata sent in the request headers. This should only check that the metadata in the request given is valid. Checks against account/container overall metadata should be forwarded on to its respective server to be checked.
- Parameters:
req – request object
target_type – str: one of: object, container, or account: indicates which type the target storage for the metadata is
- Returns:
HTTPBadRequest with bad metadata otherwise None
- swift.common.constraints.check_mount(root, drive)¶
Verify that the path to the device is a mount point and mounted. This allows us to fast fail on drives that have been unmounted because of issues, and also prevents us for accidentally filling up the root partition.
- Parameters:
root – base path where the devices are mounted
drive – drive name to be checked
- Returns:
full path to the device
- Raises:
ValueError – if drive fails to validate
- swift.common.constraints.check_name_format(req, name, target_type)¶
Validate that the header contains valid account or container name.
- Parameters:
req – HTTP request object
name – header value to validate
target_type – which header is being validated (Account or Container)
- Returns:
A properly encoded account name or container name
- Raises:
HTTPPreconditionFailed – if account header is not well formatted.
- swift.common.constraints.check_object_creation(req, object_name)¶
Check to ensure that everything is alright about an object to be created.
- Parameters:
req – HTTP request object
object_name – name of object to be created
- Returns:
HTTPRequestEntityTooLarge – the object is too large
- Returns:
HTTPLengthRequired – missing content-length header and not a chunked request
- Returns:
HTTPBadRequest – missing or bad content-type header, or bad metadata
- Returns:
HTTPNotImplemented – unsupported transfer-encoding header value
- swift.common.constraints.check_utf8(string, internal=False)¶
Validate if a string is valid UTF-8 str or unicode and that it does not contain any reserved characters.
- Parameters:
string – string to be validated
internal – boolean, allows reserved characters if True
- Returns:
True if the string is valid utf-8 str or unicode and contains no null characters, False otherwise
- swift.common.constraints.reload_constraints()¶
Parse SWIFT_CONF_FILE and reset module level global constraint attrs, populating OVERRIDE_CONSTRAINTS AND EFFECTIVE_CONSTRAINTS along the way.
- swift.common.constraints.valid_api_version(version)¶
Checks if the requested version is valid.
Currently Swift only supports “v1” and “v1.0”.
- swift.common.constraints.valid_timestamp(request)¶
Helper function to extract a timestamp from requests that require one.
- Parameters:
request – the swob request object
- Returns:
a valid Timestamp instance
- Raises:
HTTPBadRequest – on missing or invalid X-Timestamp
Container Sync Realms¶
- class swift.common.container_sync_realms.ContainerSyncRealms(conf_path, logger)¶
Bases:
object
Loads and parses the container-sync-realms.conf, occasionally checking the file’s mtime to see if it needs to be reloaded.
- clusters(realm)¶
Returns a list of clusters for the realm.
- endpoint(realm, cluster)¶
Returns the endpoint for the cluster in the realm.
- get_sig(request_method, path, x_timestamp, nonce, realm_key, user_key)¶
Returns the hexdigest string of the HMAC-SHA1 (RFC 2104) for the information given.
- Parameters:
request_method – HTTP method of the request.
path – The path to the resource (url-encoded).
x_timestamp – The X-Timestamp header value for the request.
nonce – A unique value for the request.
realm_key – Shared secret at the cluster operator level.
user_key – Shared secret at the user’s container level.
- Returns:
hexdigest str of the HMAC-SHA1 for the request.
- key(realm)¶
Returns the key for the realm.
- key2(realm)¶
Returns the key2 for the realm.
- realms()¶
Returns a list of realms.
- reload()¶
Forces a reload of the conf file.
Digest¶
- swift.common.digest.extract_digest_and_algorithm(value)¶
Returns a tuple of (digest_algorithm, hex_encoded_digest) from a client-provided string of the form:
<hex-encoded digest>
or:
<algorithm>:<base64-encoded digest>
Note that hex-encoded strings must use one of sha1, sha256, or sha512.
- Raises:
ValueError on parse failures
- swift.common.digest.get_allowed_digests(conf_digests, logger=None)¶
Pulls out ‘allowed_digests’ from the supplied conf. Then compares them with the list of supported and deprecated digests and returns whatever remain.
When something is unsupported or deprecated it’ll log a warning.
- Parameters:
conf_digests – iterable of allowed digests. If empty, defaults to DEFAULT_ALLOWED_DIGESTS.
logger – optional logger; if provided, use it issue deprecation warnings
- Returns:
A set of allowed digests that are supported and a set of deprecated digests.
- Raises:
ValueError, if there are no digests left to return.
- swift.common.digest.get_hmac(request_method, path, expires, key, digest='sha1', ip_range=None)¶
Returns the hexdigest string of the HMAC (see RFC 2104) for the request.
- Parameters:
request_method – Request method to allow.
path – The path to the resource to allow access to.
expires – Unix timestamp as an int for when the URL expires.
key – HMAC shared secret.
digest – constructor or the string name for the digest to use in calculating the HMAC Defaults to SHA1
ip_range – The ip range from which the resource is allowed to be accessed. We need to put the ip_range as the first argument to hmac to avoid manipulation of the path due to newlines being valid in paths e.g. /v1/a/c/on127.0.0.1
- Returns:
hexdigest str of the HMAC for the request using the specified digest algorithm.
Direct Client¶
Internal client library for making calls directly to the servers rather than through the proxy.
- exception swift.common.direct_client.DirectClientException(stype, method, node, part, path, resp, host=None)¶
Bases:
ClientException
- exception swift.common.direct_client.DirectClientReconException(method, node, path, resp)¶
Bases:
ClientException
- swift.common.direct_client.direct_delete_account(node, part, account, conn_timeout=5, response_timeout=15, headers=None)¶
- swift.common.direct_client.direct_delete_container(node, part, account, container, conn_timeout=5, response_timeout=15, headers=None)¶
Delete container directly from the container server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
headers – dict to be passed into HTTPConnection headers
- Raises:
ClientException – HTTP DELETE request failed
- swift.common.direct_client.direct_delete_container_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶
- swift.common.direct_client.direct_delete_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶
Delete object directly from the object server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
obj – object name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
- Raises:
ClientException – HTTP DELETE request failed
- swift.common.direct_client.direct_get_account(node, part, account, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15, end_marker=None, reverse=None, headers=None)¶
Get listings directly from the account server.
- Parameters:
node – node dictionary from the ring
part – partition the account is on
account – account name
marker – marker query
limit – query limit
prefix – prefix query
delimiter – delimiter for the query
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
end_marker – end_marker query
reverse – reverse the returned listing
- Returns:
a tuple of (response headers, a list of containers) The response headers will HeaderKeyDict.
- swift.common.direct_client.direct_get_container(node, part, account, container, marker=None, limit=None, prefix=None, delimiter=None, conn_timeout=5, response_timeout=15, end_marker=None, reverse=None, headers=None, extra_params=None)¶
Get container listings directly from the container server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
marker – marker query
limit – query limit
prefix – prefix query
delimiter – delimiter for the query
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
end_marker – end_marker query
reverse – reverse the returned listing
headers – headers to be included in the request
extra_params – a dict of extra parameters to be included in the request. It can be used to pass additional parameters, e.g, {‘states’:’updating’} can be used with shard_range/namespace listing. It can also be used to pass the existing keyword args, like ‘marker’ or ‘limit’, but if the same parameter appears twice in both keyword arg (not None) and extra_params, this function will raise TypeError.
- Returns:
a tuple of (response headers, a list of objects) The response headers will be a HeaderKeyDict.
- swift.common.direct_client.direct_get_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, resp_chunk_size=None, headers=None)¶
Get object directly from the object server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
obj – object name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
resp_chunk_size – if defined, chunk size of data to read.
headers – dict to be passed into HTTPConnection headers
- Returns:
a tuple of (response headers, the object’s contents) The response headers will be a HeaderKeyDict.
- Raises:
ClientException – HTTP GET request failed
- swift.common.direct_client.direct_get_recon(node, recon_command, conn_timeout=5, response_timeout=15, headers=None)¶
Get recon json directly from the storage server.
- Parameters:
node – node dictionary from the ring
recon_command – recon string (post /recon/)
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
headers – dict to be passed into HTTPConnection headers
- Returns:
deserialized json response
- Raises:
DirectClientReconException – HTTP GET request failed
- swift.common.direct_client.direct_get_suffix_hashes(node, part, suffixes, conn_timeout=5, response_timeout=15, headers=None)¶
Get suffix hashes directly from the object server.
Note that unlike other
direct_client
functions, this one defaults to using the replication network to make requests.- Parameters:
node – node dictionary from the ring
part – partition the container is on
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
headers – dict to be passed into HTTPConnection headers
- Returns:
dict of suffix hashes
- Raises:
ClientException – HTTP REPLICATE request failed
- swift.common.direct_client.direct_head_container(node, part, account, container, conn_timeout=5, response_timeout=15, headers=None)¶
Request container information directly from the container server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
- Returns:
a dict containing the response’s headers in a HeaderKeyDict
- Raises:
ClientException – HTTP HEAD request failed
- swift.common.direct_client.direct_head_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶
Request object information directly from the object server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
obj – object name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
headers – dict to be passed into HTTPConnection headers
- Returns:
a dict containing the response’s headers in a HeaderKeyDict
- Raises:
ClientException – HTTP HEAD request failed
- swift.common.direct_client.direct_post_container(node, part, account, container, conn_timeout=5, response_timeout=15, headers=None)¶
Make a POST request to a container server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
headers – additional headers to include in the request
- Raises:
ClientException – HTTP PUT request failed
- swift.common.direct_client.direct_post_object(node, part, account, container, name, headers, conn_timeout=5, response_timeout=15)¶
Direct update to object metadata on object server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
name – object name
headers – headers to store as metadata
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
- Raises:
ClientException – HTTP POST request failed
- swift.common.direct_client.direct_put_container(node, part, account, container, conn_timeout=5, response_timeout=15, headers=None, contents=None, content_length=None, chunk_size=65535)¶
Make a PUT request to a container server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
headers – additional headers to include in the request
contents – an iterable or string to send in request body (optional)
content_length – value to send as content-length header (optional)
chunk_size – chunk size of data to send (optional)
- Raises:
ClientException – HTTP PUT request failed
- swift.common.direct_client.direct_put_container_object(node, part, account, container, obj, conn_timeout=5, response_timeout=15, headers=None)¶
- swift.common.direct_client.direct_put_object(node, part, account, container, name, contents, content_length=None, etag=None, content_type=None, headers=None, conn_timeout=5, response_timeout=15, chunk_size=65535)¶
Put object directly from the object server.
- Parameters:
node – node dictionary from the ring
part – partition the container is on
account – account name
container – container name
name – object name
contents – an iterable or string to read object data from
content_length – value to send as content-length header
etag – etag of contents
content_type – value to send as content-type header
headers – additional headers to include in the request
conn_timeout – timeout in seconds for establishing the connection
response_timeout – timeout in seconds for getting the response
chunk_size – if defined, chunk size of data to send.
- Returns:
etag from the server response
- Raises:
ClientException – HTTP PUT request failed
- swift.common.direct_client.gen_headers(hdrs_in=None, add_ts=True)¶
Get the headers ready for a request. All requests should have a User-Agent string, but if one is passed in don’t over-write it. Not all requests will need an X-Timestamp, but if one is passed in do not over-write it.
- Parameters:
headers – dict or None, base for HTTP headers
add_ts – boolean, should be True for any “unsafe” HTTP request
- Returns:
HeaderKeyDict based on headers and ready for the request
- swift.common.direct_client.retry(func, *args, **kwargs)¶
Helper function to retry a given function a number of times.
- Parameters:
func – callable to be called
retries – number of retries
error_log – logger for errors
args – arguments to send to func
kwargs – keyward arguments to send to func (if retries or error_log are sent, they will be deleted from kwargs before sending on to func)
- Returns:
result of func
- Raises:
ClientException – all retries failed
Exceptions¶
- exception swift.common.exceptions.APIVersionError¶
Bases:
SwiftException
- exception swift.common.exceptions.ChunkReadError¶
Bases:
SwiftException
- exception swift.common.exceptions.ChunkReadTimeout(seconds=None, exception=None)¶
Bases:
Timeout
- exception swift.common.exceptions.ChunkWriteTimeout(seconds=None, exception=None)¶
Bases:
Timeout
- exception swift.common.exceptions.ClientException(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=None, http_reason='', http_device='', http_response_content='', http_headers=None)¶
Bases:
Exception
- exception swift.common.exceptions.ConnectionTimeout(seconds=None, exception=None)¶
Bases:
Timeout
- exception swift.common.exceptions.DatabaseAuditorException¶
Bases:
SwiftException
Bases:
SwiftException
- exception swift.common.exceptions.DiskFileBadMetadataChecksum¶
Bases:
DiskFileError
- exception swift.common.exceptions.DiskFileCollision¶
Bases:
DiskFileError
- exception swift.common.exceptions.DiskFileDeleted(metadata=None)¶
Bases:
DiskFileNotExist
Bases:
DiskFileError
- exception swift.common.exceptions.DiskFileError¶
Bases:
SwiftException
- exception swift.common.exceptions.DiskFileExpired(metadata=None)¶
Bases:
DiskFileDeleted
- exception swift.common.exceptions.DiskFileNoSpace¶
Bases:
DiskFileError
- exception swift.common.exceptions.DiskFileNotExist¶
Bases:
DiskFileError
- exception swift.common.exceptions.DiskFileNotOpen¶
Bases:
DiskFileError
- exception swift.common.exceptions.DiskFileQuarantined¶
Bases:
DiskFileError
- exception swift.common.exceptions.DiskFileXattrNotSupported¶
Bases:
DiskFileError
- exception swift.common.exceptions.DriveNotMounted¶
Bases:
SwiftException
- exception swift.common.exceptions.DuplicateDeviceError¶
Bases:
RingBuilderError
- exception swift.common.exceptions.EmptyRingError¶
Bases:
RingBuilderError
- exception swift.common.exceptions.EncryptionException¶
Bases:
SwiftException
- exception swift.common.exceptions.FileNotFoundError¶
Bases:
SwiftException
Bases:
SwiftException
- exception swift.common.exceptions.InsufficientStorage¶
Bases:
SwiftException
- exception swift.common.exceptions.InvalidAccountInfo¶
Bases:
DatabaseAuditorException
- exception swift.common.exceptions.InvalidPidFileException¶
Bases:
Exception
- exception swift.common.exceptions.InvalidTimestamp¶
Bases:
SwiftException
- exception swift.common.exceptions.LinkIterError¶
Bases:
SwiftException
- exception swift.common.exceptions.ListingIterError¶
Bases:
SwiftException
- exception swift.common.exceptions.ListingIterNotAuthorized(aresp)¶
Bases:
ListingIterError
- exception swift.common.exceptions.ListingIterNotFound¶
Bases:
ListingIterError
- exception swift.common.exceptions.LockTimeout(seconds=None, msg=None)¶
Bases:
MessageTimeout
- exception swift.common.exceptions.MemcacheConnectionError¶
Bases:
Exception
- exception swift.common.exceptions.MemcacheIncrNotFoundError¶
Bases:
MemcacheConnectionError
- exception swift.common.exceptions.MemcachePoolTimeout(seconds=None, exception=None)¶
Bases:
Timeout
- exception swift.common.exceptions.MessageTimeout(seconds=None, msg=None)¶
Bases:
Timeout
- exception swift.common.exceptions.MimeInvalid¶
Bases:
SwiftException
- exception swift.common.exceptions.MultiphasePUTNotSupported¶
Bases:
SwiftException
- exception swift.common.exceptions.PartitionLockTimeout(seconds=None, msg=None)¶
Bases:
LockTimeout
- exception swift.common.exceptions.PathNotDir¶
Bases:
OSError
- exception swift.common.exceptions.PermissionError¶
Bases:
SwiftException
- exception swift.common.exceptions.PutterConnectError(status=None)¶
Bases:
Exception
- exception swift.common.exceptions.QuarantineRequest¶
Bases:
SwiftException
- exception swift.common.exceptions.RangeAlreadyComplete¶
Bases:
SwiftException
- exception swift.common.exceptions.ReplicationException¶
Bases:
Exception
- exception swift.common.exceptions.ReplicationLockTimeout(seconds=None, msg=None)¶
Bases:
LockTimeout
- exception swift.common.exceptions.ResponseTimeout(seconds=None, exception=None)¶
Bases:
Timeout
- exception swift.common.exceptions.RingBuilderError¶
Bases:
SwiftException
- exception swift.common.exceptions.RingLoadError¶
Bases:
SwiftException
- exception swift.common.exceptions.RingValidationError¶
Bases:
RingBuilderError
- exception swift.common.exceptions.SegmentError¶
Bases:
SwiftException
- exception swift.common.exceptions.ShortReadError¶
Bases:
SwiftException
- exception swift.common.exceptions.SuffixSyncError¶
Bases:
SwiftException
- exception swift.common.exceptions.SwiftException¶
Bases:
Exception
- exception swift.common.exceptions.UnPicklingError¶
Bases:
SwiftException
- exception swift.common.exceptions.UnknownSecretIdError¶
Bases:
EncryptionException
Internal Client¶
- class swift.common.internal_client.CompressingFileReader(file_obj, compresslevel=9, chunk_size=4096)¶
Bases:
object
Wrapper for file object to compress object while reading.
Can be used to wrap file objects passed to InternalClient.upload_object().
Used in testing of InternalClient.
- Parameters:
file_obj – File object to wrap.
compresslevel – Compression level, defaults to 9.
chunk_size – Size of chunks read when iterating using object, defaults to 4096.
- next()¶
- read(*a, **kw)¶
Reads a chunk from the file object.
Params are passed directly to the underlying file object’s read().
- Returns:
Compressed chunk from file object.
- seek(offset, whence=0)¶
- set_initial_state()¶
Sets the object to the state needed for the first read.
- class swift.common.internal_client.InternalClient(conf_path, user_agent, request_tries, use_replication_network=False, global_conf=None, app=None, **kwargs)¶
Bases:
object
An internal client that uses a swift proxy app to make requests to Swift.
This client will exponentially slow down for retries.
- Parameters:
conf_path – Full path to proxy config.
user_agent – User agent to be sent to requests to Swift.
request_tries – Number of tries before InternalClient.make_request() gives up.
use_replication_network – Force the client to use the replication network over the cluster.
global_conf – a dict of options to update the loaded proxy config. Options in
global_conf
will override those inconf_path
except where theconf_path
option is preceded byset
.app – Optionally provide a WSGI app for the internal client to use.
- static check_gatekeeper_not_loaded(app)¶
- container_exists(account, container)¶
Checks to see if a container exists.
- Parameters:
account – The container’s account.
container – Container to check.
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- Returns:
True if container exists, false otherwise.
- create_account(account)¶
Creates an account.
- Parameters:
account – Account to create.
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- create_container(account, container, headers=None, acceptable_statuses=(2,))¶
Creates container.
- Parameters:
account – The container’s account.
container – Container to create.
headers – Defaults to empty dict.
acceptable_statuses – List of status for valid responses, defaults to (2,).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- delete_account(account, acceptable_statuses=(2, 404))¶
Deletes an account.
- Parameters:
account – Account to delete.
acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- delete_container(account, container, headers=None, acceptable_statuses=(2, 404))¶
Deletes a container.
- Parameters:
account – The container’s account.
container – Container to delete.
acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- delete_object(account, container, obj, acceptable_statuses=(2, 404), headers=None)¶
Deletes an object.
- Parameters:
account – The object’s account.
container – The object’s container.
obj – The object.
acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
headers – extra headers to send with request
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- get_account_info(account, acceptable_statuses=(2, 404))¶
Returns (container_count, object_count) for an account.
- Parameters:
account – Account on which to get the information.
acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- get_account_metadata(account, metadata_prefix='', acceptable_statuses=(2,), params=None)¶
Gets account metadata.
- Parameters:
account – Account on which to get the metadata.
metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’.
acceptable_statuses – List of status for valid responses, defaults to (2,).
- Returns:
Returns dict of account metadata. Keys will be lowercase.
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- get_container_metadata(account, container, metadata_prefix='', acceptable_statuses=(2,), params=None)¶
Gets container metadata.
- Parameters:
account – The container’s account.
container – Container to get metadata on.
metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’.
acceptable_statuses – List of status for valid responses, defaults to (2,).
- Returns:
Returns dict of container metadata. Keys will be lowercase.
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- get_object(account, container, obj, headers=None, acceptable_statuses=(2,), params=None)¶
Gets an object.
- Parameters:
account – The object’s account.
container – The object’s container.
obj – The object name.
headers – Headers to send with request, defaults to empty dict.
acceptable_statuses – List of status for valid responses, defaults to (2,).
params – A dict of params to be set in request query string, defaults to None.
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- Returns:
A 3-tuple (status, headers, iterator of object body)
- get_object_metadata(account, container, obj, metadata_prefix='', acceptable_statuses=(2,), headers=None, params=None)¶
Gets object metadata.
- Parameters:
account – The object’s account.
container – The object’s container.
obj – The object.
metadata_prefix – Used to filter values from the headers returned. Will strip that prefix from the keys in the dict returned. Defaults to ‘’.
acceptable_statuses – List of status for valid responses, defaults to (2,).
headers – extra headers to send with request
- Returns:
Dict of object metadata.
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- handle_request(*args, **kwargs)¶
- iter_containers(account, marker='', end_marker='', prefix='', acceptable_statuses=(2, 404))¶
Returns an iterator of containers dicts from an account.
- Parameters:
account – Account on which to do the container listing.
marker – Prefix of first desired item, defaults to ‘’.
end_marker – Last item returned will be ‘less’ than this, defaults to ‘’.
prefix – Prefix of containers
acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- iter_object_lines(account, container, obj, headers=None, acceptable_statuses=(2,))¶
Returns an iterator of object lines from an uncompressed or compressed text object.
Uncompress object as it is read if the object’s name ends with ‘.gz’.
- Parameters:
account – The object’s account.
container – The object’s container.
obj – The object.
acceptable_statuses – List of status for valid responses, defaults to (2,).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- iter_objects(account, container, marker='', end_marker='', prefix='', acceptable_statuses=(2, 404))¶
Returns an iterator of object dicts from a container.
- Parameters:
account – The container’s account.
container – Container to iterate objects on.
marker – Prefix of first desired item, defaults to ‘’.
end_marker – Last item returned will be ‘less’ than this, defaults to ‘’.
prefix – Prefix of objects
acceptable_statuses – List of status for valid responses, defaults to (2, HTTP_NOT_FOUND).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- make_path(account, container=None, obj=None)¶
Returns a swift path for a request quoting and utf-8 encoding the path parts as need be.
- Parameters:
account – swift account
container – container, defaults to None
obj – object, defaults to None
- Raises:
ValueError – Is raised if obj is specified and container is not.
- make_request(method, path, headers, acceptable_statuses, body_file=None, params=None)¶
Makes a request to Swift with retries.
- Parameters:
method – HTTP method of request.
path – Path of request.
headers – Headers to be sent with request.
acceptable_statuses – List of acceptable statuses for request.
body_file – Body file to be passed along with request, defaults to None.
params – A dict of params to be set in request query string, defaults to None.
- Returns:
Response object on success.
- Raises:
UnexpectedResponse – Exception raised when make_request() fails to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- set_account_metadata(account, metadata, metadata_prefix='', acceptable_statuses=(2,))¶
Sets account metadata. A call to this will add to the account metadata and not overwrite all of it with values in the metadata dict. To clear an account metadata value, pass an empty string as the value for the key in the metadata dict.
- Parameters:
account – Account on which to get the metadata.
metadata – Dict of metadata to set.
metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’.
acceptable_statuses – List of status for valid responses, defaults to (2,).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- set_container_metadata(account, container, metadata, metadata_prefix='', acceptable_statuses=(2,))¶
Sets container metadata. A call to this will add to the container metadata and not overwrite all of it with values in the metadata dict. To clear a container metadata value, pass an empty string as the value for the key in the metadata dict.
- Parameters:
account – The container’s account.
container – Container to set metadata on.
metadata – Dict of metadata to set.
metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’.
acceptable_statuses – List of status for valid responses, defaults to (2,).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- set_object_metadata(account, container, obj, metadata, metadata_prefix='', acceptable_statuses=(2,))¶
Sets an object’s metadata. The object’s metadata will be overwritten by the values in the metadata dict.
- Parameters:
account – The object’s account.
container – The object’s container.
obj – The object.
metadata – Dict of metadata to set.
metadata_prefix – Prefix used to set metadata values in headers of requests, used to prefix keys in metadata when setting metadata, defaults to ‘’.
acceptable_statuses – List of status for valid responses, defaults to (2,).
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- upload_object(fobj, account, container, obj, headers=None, acceptable_statuses=(2,), params=None)¶
- Parameters:
fobj – File object to read object’s content from.
account – The object’s account.
container – The object’s container.
obj – The object.
headers – Headers to send with request, defaults to empty dict.
acceptable_statuses – List of acceptable statuses for request.
params – A dict of params to be set in request query string, defaults to None.
- Raises:
UnexpectedResponse – Exception raised when requests fail to get a response with an acceptable status
Exception – Exception is raised when code fails in an unexpected way.
- class swift.common.internal_client.SimpleClient(url=None, token=None, starting_backoff=1, max_backoff=5, retries=5)¶
Bases:
object
Simple client that is used in bin/swift-dispersion-* and container sync
- base_request(method, container=None, name=None, prefix=None, headers=None, proxy=None, contents=None, full_listing=None, logger=None, additional_info=None, timeout=None, marker=None)¶
- get_account(*args, **kwargs)¶
- get_container(container, **kwargs)¶
- put_container(container, **kwargs)¶
- put_object(container, name, contents, **kwargs)¶
- retry_request(method, **kwargs)¶
- exception swift.common.internal_client.UnexpectedResponse(message, resp)¶
Bases:
Exception
Exception raised on invalid responses to InternalClient.make_request().
- Parameters:
message – Exception message.
resp – The unexpected response.
- swift.common.internal_client.delete_object(url, **kwargs)¶
For usage with container sync
- swift.common.internal_client.get_auth(url, user, key, auth_version='1.0', **kwargs)¶
- swift.common.internal_client.head_object(url, **kwargs)¶
For usage with container sync
- swift.common.internal_client.put_object(url, **kwargs)¶
For usage with container sync
Manager¶
- class swift.common.manager.Manager(servers, run_dir='/var/run/swift')¶
Bases:
object
Main class for performing commands on groups of servers.
- Parameters:
servers – list of server names as strings
- force_reload(**kwargs)¶
alias for reload
- get_command(cmd)¶
Find and return the decorated method named like cmd
- Parameters:
cmd – the command to get, a string, if not found raises UnknownCommandError
- kill(**kwargs)¶
stop a server (no error if not running)
- kill_child_pids(**kwargs)¶
kill child pids, optionally servicing accepted connections
- classmethod list_commands()¶
Get all publicly accessible commands
- Returns:
a list of string tuples (cmd, help), the method names who are decorated as commands
- no_daemon(**kwargs)¶
start a server interactively
- no_wait(**kwargs)¶
spawn server and return immediately
- once(**kwargs)¶
start server and run one pass on supporting daemons
- reload(**kwargs)¶
graceful shutdown then restart on supporting servers
- reload_seamless(**kwargs)¶
seamlessly re-exec, then shutdown of old listen sockets on supporting servers
- restart(**kwargs)¶
stops then restarts server
- run_command(cmd, **kwargs)¶
Find the named command and run it
- Parameters:
cmd – the command name to run
- shutdown(**kwargs)¶
allow current requests to finish on supporting servers
- start(**kwargs)¶
starts a server
- status(**kwargs)¶
display status of tracked pids for server
- stop(**kwargs)¶
stops a server
- class swift.common.manager.Server(server, run_dir='/var/run/swift')¶
Bases:
object
Manage operations on a server or group of servers of similar type
- Parameters:
server – name of server
- conf_files(**kwargs)¶
Get conf files for this server
- Parameters:
number – if supplied will only lookup the nth server
- Returns:
list of conf files
- get_conf_file_name(pid_file)¶
Translate pid_file to a corresponding conf_file
- Parameters:
pid_file – a pid_file for this server, a string
- Returns:
the conf_file for this pid_file
- get_pid_file_name(conf_file)¶
Translate conf_file to a corresponding pid_file
- Parameters:
conf_file – an conf_file for this server, a string
- Returns:
the pid_file for this conf_file
- get_running_pids(**kwargs)¶
Get running pids
- Returns:
a dict mapping pids (ints) to pid_files (paths)
- interact(**kwargs)¶
wait on spawned procs to terminate
- iter_pid_files(**kwargs)¶
Generator, yields (pid_file, pids)
- kill_child_pids(**kwargs)¶
Kill child pids, leaving server overseer to respawn them
- Parameters:
graceful – if True, attempt SIGHUP on supporting servers
seamless – if True, attempt SIGUSR1 on supporting servers
- Returns:
a dict mapping pids (ints) to pid_files (paths)
- kill_running_pids(**kwargs)¶
Kill running pids
- Parameters:
graceful – if True, attempt SIGHUP on supporting servers
seamless – if True, attempt SIGUSR1 on supporting servers
- Returns:
a dict mapping pids (ints) to pid_files (paths)
- launch(**kwargs)¶
Collect conf files and attempt to spawn the processes for this server
- pid_files(**kwargs)¶
Get pid files for this server
- Parameters:
number – if supplied will only lookup the nth server
- Returns:
list of pid files
- signal_children(sig, **kwargs)¶
Send a signal to child pids for this server
- Parameters:
sig – signal to send
- Returns:
a dict mapping pids (ints) to pid_files (paths)
- signal_pids(sig, **kwargs)¶
Send a signal to pids for this server
- Parameters:
sig – signal to send
- Returns:
a dict mapping pids (ints) to pid_files (paths)
- spawn(conf_file, once=False, wait=True, daemon=True, additional_args=None, **kwargs)¶
Launch a subprocess for this server.
- Parameters:
conf_file – path to conf_file to use as first arg
once – boolean, add once argument to command
wait – boolean, if true capture stdout with a pipe
daemon – boolean, if false ask server to log to console
additional_args – list of additional arguments to pass on the command line
- Returns:
the pid of the spawned process
- status(pids=None, **kwargs)¶
Display status of server
- Parameters:
pids – if not supplied pids will be populated automatically
number – if supplied will only lookup the nth server
- Returns:
1 if server is not running, 0 otherwise
- stop(**kwargs)¶
Send stop signals to pids for this server
- Returns:
a dict mapping pids (ints) to pid_files (paths)
- wait(**kwargs)¶
wait on spawned procs to start
- exception swift.common.manager.UnknownCommandError¶
Bases:
Exception
- swift.common.manager.command(func)¶
Decorator to declare which methods are accessible as commands, commands always return 1 or 0, where 0 should indicate success.
- Parameters:
func – function to make public
- swift.common.manager.format_server_name(servername)¶
Formats server name as swift compatible server names E.g. swift-object-server
- Parameters:
servername – server name
- Returns:
swift compatible server name and its binary name
- swift.common.manager.get_child_pids(pid)¶
Get the current set of all child PIDs for a PID.
- Parameters:
pid – process id
- swift.common.manager.kill_group(pid, sig)¶
Send signal to process group
: param pid: process id : param sig: signal to send
- swift.common.manager.safe_kill(pid, sig, name)¶
Send signal to process and check process name
: param pid: process id : param sig: signal to send : param name: name to ensure target process
- swift.common.manager.setup_env()¶
Try to increase resource limits of the OS. Move PYTHON_EGG_CACHE to /tmp
- swift.common.manager.verify_server(server)¶
Check whether the server is among swift servers or not, and also checks whether the server’s binaries are installed or not.
- Parameters:
server – name of the server
- Returns:
True, when the server name is valid and its binaries are found. False, otherwise.
- swift.common.manager.watch_server_pids(server_pids, interval=1, **kwargs)¶
Monitor a collection of server pids yielding back those pids that aren’t responding to signals.
- Parameters:
server_pids – a dict, lists of pids [int,…] keyed on Server objects
MemCacheD¶
Why our own memcache client? By Michael Barton
python-memcached doesn’t use consistent hashing, so adding or removing a memcache server from the pool invalidates a huge percentage of cached items.
If you keep a pool of python-memcached client objects, each client object has its own connection to every memcached server, only one of which is ever in use. So you wind up with n * m open sockets and almost all of them idle. This client effectively has a pool for each server, so the number of backend connections is hopefully greatly reduced.
python-memcache uses pickle to store things, and there was already a huge stink about Swift using pickles in memcache (http://osvdb.org/show/osvdb/86581). That seemed sort of unfair, since nova and keystone and everyone else use pickles for memcache too, but it’s hidden behind a “standard” library. But changing would be a security regression at this point.
Also, pylibmc wouldn’t work for us because it needs to use python sockets in order to play nice with eventlet.
Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:
http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt
- class swift.common.memcached.MemcacheCommand(method, key)¶
Bases:
object
Helper class that encapsulates common parameters of a command.
- Parameters:
method – the name of the MemcacheRing method that was called.
key – the memcached key.
- class swift.common.memcached.MemcacheConnPool(server, size, connect_timeout, tls_context=None)¶
Bases:
Pool
Connection pool for Memcache Connections
The server parameter can be a hostname, an IPv4 address, or an IPv6 address with an optional port. See
swift.common.utils.parse_socket_string()
for details.- create()¶
Generate a new pool item. In order for the pool to function, either this method must be overriden in a subclass or the pool must be constructed with the create argument. It accepts no arguments and returns a single instance of whatever thing the pool is supposed to contain.
In general,
create()
is called whenever the pool exceeds its previous high-water mark of concurrently-checked-out-items. In other words, in a new pool with min_size of 0, the very first call toget()
will result in a call tocreate()
. If the first caller callsput()
before some other caller callsget()
, then the first item will be returned, andcreate()
will not be called a second time.
- get()¶
Return an item from the pool, when one is available. This may cause the calling greenthread to block.
- class swift.common.memcached.MemcacheRing(servers, connect_timeout=0.3, io_timeout=2.0, pool_timeout=1.0, tries=3, max_conns=2, tls_context=None, logger=None, error_limit_count=10, error_limit_time=60, error_limit_duration=60, item_size_warning_threshold=-1)¶
Bases:
object
Simple, consistent-hashed memcache client.
- decr(key, delta=1, time=0)¶
Decrements a key which has a numeric value by delta. Calls incr with -delta.
- Parameters:
key – key
delta – amount to subtract to the value of key (or set the value to 0 if the key is not found) will be cast to an int
time – the time to live
- Returns:
result of decrementing
- Raises:
- delete(key, server_key=None)¶
Deletes a key/value pair from memcache.
- Parameters:
key – key to be deleted
server_key – key to use in determining which server in the ring is used
- get(key, raise_on_error=False)¶
Gets the object specified by key. It will also unserialize the object before returning if it is serialized in memcache with JSON.
- Parameters:
key – key
raise_on_error – if True, propagate Timeouts and other errors. By default, errors are treated as cache misses.
- Returns:
value of the key in memcache
- get_multi(keys, server_key)¶
Gets multiple values from memcache for the given keys.
- Parameters:
keys – keys for values to be retrieved from memcache
server_key – key to use in determining which server in the ring is used
- Returns:
list of values
- incr(key, delta=1, time=0)¶
Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta or 0 if delta < 0. If passed a negative number, will use memcached’s decr. Returns the int stored in memcached Note: The data memcached stores as the result of incr/decr is an unsigned int. decr’s that result in a number below 0 are stored as 0.
- Parameters:
key – key
delta – amount to add to the value of key (or set as the value if the key is not found) will be cast to an int
time – the time to live
- Returns:
result of incrementing
- Raises:
- set(key, value, serialize=True, time=0, min_compress_len=0, raise_on_error=False)¶
Set a key/value pair in memcache
- Parameters:
key – key
value – value
serialize – if True, value is serialized with JSON before sending to memcache
time – the time to live
min_compress_len – minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it.
raise_on_error – if True, propagate Timeouts and other errors. By default, errors are ignored.
- set_multi(mapping, server_key, serialize=True, time=0, min_compress_len=0)¶
Sets multiple key/value pairs in memcache.
- Parameters:
mapping – dictionary of keys and values to be set in memcache
server_key – key to use in determining which server in the ring is used
serialize – if True, value is serialized with JSON before sending to memcache.
time – the time to live
- Min_compress_len:
minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it
- swift.common.memcached.load_memcache(conf, logger)¶
Build a MemcacheRing object from the given config. It will also use the passed in logger.
- Parameters:
conf – a dict, the config options
logger – a logger
- swift.common.memcached.sanitize_timeout(timeout)¶
Sanitize a timeout value to use an absolute expiration time if the delta is greater than 30 days (in seconds). Note that the memcached server translates negative values to mean a delta of 30 days in seconds (and 1 additional second), client beware.
Middleware Registry¶
- swift.common.registry.get_sensitive_headers()¶
Returns the set of registered sensitive headers.
Used by
swift.common.middleware.proxy_logging
to perform redactions prior to logging.
- swift.common.registry.get_sensitive_params()¶
Returns the set of registered sensitive query parameters.
Used by
swift.common.middleware.proxy_logging
to perform redactions prior to logging.
- swift.common.registry.get_swift_info(admin=False, disallowed_sections=None)¶
Returns information about the swift cluster that has been previously registered with the register_swift_info call.
- Parameters:
admin – boolean value, if True will additionally return an ‘admin’ section with information previously registered as admin info.
disallowed_sections – list of section names to be withheld from the information returned.
- Returns:
dictionary of information about the swift cluster.
- swift.common.registry.register_sensitive_header(header)¶
Register a header as being “sensitive”.
Sensitive headers are automatically redacted when logging. See the
reveal_sensitive_prefix
option in the proxy-server sample config for more information.- Parameters:
header – The (case-insensitive) header name which, if present, may contain sensitive information. Examples include
X-Auth-Token
and (if s3api is enabled)Authorization
. Limited to ASCII characters.
- swift.common.registry.register_sensitive_param(query_param)¶
Register a query parameter as being “sensitive”.
Sensitive query parameters are automatically redacted when logging. See the
reveal_sensitive_prefix
option in the proxy-server sample config for more information.- Parameters:
query_param – The (case-sensitive) query parameter name which, if present, may contain sensitive information. Examples include
temp_url_signature
and (if s3api is enabled)X-Amz-Signature
. Limited to ASCII characters.
- swift.common.registry.register_swift_info(name='swift', admin=False, **kwargs)¶
Registers information about the swift cluster to be retrieved with calls to get_swift_info.
- NOTE: Do not use “.” in the param: name or any keys in kwargs. “.” is used
in the disallowed_sections to remove unwanted keys from /info.
- Parameters:
name – string, the section name to place the information under.
admin – boolean, if True, information will be registered to an admin section which can optionally be withheld when requesting the information.
kwargs – key value arguments representing the information to be added.
- Raises:
ValueError – if name or any of the keys in kwargs has “.” in it
Request Helpers¶
Miscellaneous utility functions for use in generating responses.
Why not swift.common.utils, you ask? Because this way we can import things from swob in here without creating circular imports.
- class swift.common.request_helpers.SegmentedIterable(req, app, listing_iter, max_get_time, logger, ua_suffix, swift_source, name='<not specified>', response_body_length=None)¶
Bases:
object
Iterable that returns the object contents for a large object.
- Parameters:
req – original request object
app – WSGI application from which segments will come
listing_iter –
iterable yielding the object segments to fetch, along with the byte sub-ranges to fetch. Each yielded item should be a dict with the following keys:
path
orraw_data
,first-byte
,last-byte
,hash
(optional),bytes
(optional).If
hash
is None, no MD5 verification will be done.If
bytes
is None, no length verification will be done.If
first-byte
andlast-byte
are None, then the entire object will be fetched.max_get_time – maximum permitted duration of a GET request (seconds)
logger – logger object
swift_source – value of swift.source in subrequest environ (just for logging)
ua_suffix – string to append to user-agent.
name – name of manifest (used in logging only)
response_body_length – optional response body length for the response being sent to the client.
- app_iter_range(*a, **kw)¶
swob.Response will only respond with a 206 status in certain cases; one of those is if the body iterator responds to .app_iter_range().
However, this object (or really, its listing iter) is smart enough to handle the range stuff internally, so we just no-op this out for swob.
- app_iter_ranges(ranges, content_type, boundary, content_size)¶
This method assumes that iter(self) yields all the data bytes that go into the response, but none of the MIME stuff. For example, if the response will contain three MIME docs with data “abcd”, “efgh”, and “ijkl”, then iter(self) will give out the bytes “abcdefghijkl”.
This method inserts the MIME stuff around the data bytes.
- close()¶
Called when the client disconnect. Ensure that the connection to the backend server is closed.
- validate_first_segment()¶
Start fetching object data to ensure that the first segment (if any) is valid. This is to catch cases like “first segment is missing” or “first segment’s etag doesn’t match manifest”.
Note: this does not validate that you have any segments. A zero-segment large object is not erroneous; it is just empty.
- swift.common.request_helpers.append_log_info(environ, log_info)¶
- swift.common.request_helpers.check_path_header(req, name, length, error_msg)¶
Validate that the value of path-like header is well formatted. We assume the caller ensures that specific header is present in req.headers.
- Parameters:
req – HTTP request object
name – header name
length – length of path segment check
error_msg – error message for client
- Returns:
A tuple with path parts according to length
- Raise:
HTTPPreconditionFailed if header value is not well formatted.
- swift.common.request_helpers.constrain_req_limit(req, constrained_limit)¶
- swift.common.request_helpers.copy_header_subset(from_r, to_r, condition)¶
Will copy desired subset of headers from from_r to to_r.
- Parameters:
from_r – a swob Request or Response
to_r – a swob Request or Response
condition – a function that will be passed the header key as a single argument and should return True if the header is to be copied.
- swift.common.request_helpers.get_container_update_override_key(key)¶
Returns the full X-Object-Sysmeta-Container-Update-Override-* header key.
- Parameters:
key – the key you want to override in the container update
- Returns:
the full header key
- swift.common.request_helpers.get_ip_port(node, headers)¶
Get the ip address and port that should be used for the given
node
. The normal ip address and port are returned unless thenode
orheaders
indicate that the replication ip address and port should be used.If the
headers
dict has an item with keyx-backend-use-replication-network
and a truthy value then the replication ip address and port are returned. Otherwise if thenode
dict has an item with keyuse_replication
and truthy value then the replication ip address and port are returned. Otherwise the normal ip address and port are returned.- Parameters:
node – a dict describing a node
headers – a dict of headers
- Returns:
a tuple of (ip address, port)
- swift.common.request_helpers.get_log_info(environ)¶
- swift.common.request_helpers.get_name_and_placement(request, minsegs=1, maxsegs=None, rest_with_last=False)¶
Utility function to split and validate the request path and storage policy. The storage policy index is extracted from the headers of the request and converted to a StoragePolicy instance. The remaining args are passed through to
split_and_validate_path()
.- Returns:
a list, result of
split_and_validate_path()
with the BaseStoragePolicy instance appended on the end- Raises:
HTTPServiceUnavailable – if the path is invalid or no policy exists with the extracted policy_index.
- swift.common.request_helpers.get_object_transient_sysmeta(key)¶
Returns the Object Transient System Metadata header for key. The Object Transient System Metadata namespace will be persisted by backend object servers. These headers are treated in the same way as object user metadata i.e. all headers in this namespace will be replaced on every POST request.
- Parameters:
key – metadata key
- Returns:
the entire object transient system metadata header for key
- swift.common.request_helpers.get_param(req, name, default=None)¶
Get a parameter from an HTTP request ensuring proper handling UTF-8 encoding.
- Parameters:
req – request object
name – parameter name
default – result to return if the parameter is not found
- Returns:
HTTP request parameter value, as a native string (in py2, as UTF-8 encoded str, not unicode object)
- Raises:
HTTPBadRequest – if param not valid UTF-8 byte sequence
- swift.common.request_helpers.get_reserved_name(*parts)¶
Generate a valid reserved name that joins the component parts.
- Returns:
a string
- swift.common.request_helpers.get_sys_meta_prefix(server_type)¶
Returns the prefix for system metadata headers for given server type.
This prefix defines the namespace for headers that will be persisted by backend servers.
- Parameters:
server_type – type of backend server i.e. [account|container|object]
- Returns:
prefix string for server type’s system metadata headers
- swift.common.request_helpers.get_user_meta_prefix(server_type)¶
Returns the prefix for user metadata headers for given server type.
This prefix defines the namespace for headers that will be persisted by backend servers.
- Parameters:
server_type – type of backend server i.e. [account|container|object]
- Returns:
prefix string for server type’s user metadata headers
- swift.common.request_helpers.get_valid_part_num(req)¶
Any non-range GET or HEAD request for a SLO object may include a part-number parameter in query string. If the passed in request includes a part-number parameter it will be parsed into a valid integer and returned. If the passed in request does not include a part-number param we will return None. If the part-number parameter is invalid for the given request we will raise the appropriate HTTP exception
- Parameters:
req – the request object
- Returns:
validated part-number value or None
- Raises:
HTTPBadRequest – if request or part-number param is not valid
- swift.common.request_helpers.http_response_to_document_iters(response, read_chunk_size=4096)¶
Takes a successful object-GET HTTP response and turns it into an iterator of (first-byte, last-byte, length, headers, body-file) 5-tuples.
The response must either be a 200 or a 206; if you feed in a 204 or something similar, this probably won’t work.
- Parameters:
response – HTTP response, like from bufferedhttp.http_connect(), not a swob.Response.
- swift.common.request_helpers.is_backend_open_expired(request)¶
Helper function to check if a request has either the headers ‘x-backend-open-expired’ or ‘x-backend-replication’ for the backend to access expired objects.
- Parameters:
request – request object
- swift.common.request_helpers.is_object_transient_sysmeta(key)¶
Tests if a header key starts with and is longer than the prefix for object transient system metadata.
- Parameters:
key – header key
- Returns:
True if the key satisfies the test, False otherwise
- swift.common.request_helpers.is_open_expired(app, req)¶
Helper function to check if a request with the header ‘x-open-expired’ can access an object that has not yet been reaped by the object-expirer based on the allow_open_expired global config.
- Parameters:
app – the application instance
req – request object
- swift.common.request_helpers.is_sys_meta(server_type, key)¶
Tests if a header key starts with and is longer than the system metadata prefix for given server type.
- Parameters:
server_type – type of backend server i.e. [account|container|object]
key – header key
- Returns:
True if the key satisfies the test, False otherwise
- swift.common.request_helpers.is_sys_or_user_meta(server_type, key)¶
Tests if a header key starts with and is longer than the user or system metadata prefix for given server type.
- Parameters:
server_type – type of backend server i.e. [account|container|object]
key – header key
- Returns:
True if the key satisfies the test, False otherwise
- swift.common.request_helpers.is_use_replication_network(headers=None)¶
Determine if replication network should be used.
- Parameters:
headers – a dict of headers
- Returns:
the value of the
x-backend-use-replication-network
item fromheaders
. If noheaders
are given or the item is not found then False is returned.
- swift.common.request_helpers.is_user_meta(server_type, key)¶
Tests if a header key starts with and is longer than the user metadata prefix for given server type.
- Parameters:
server_type – type of backend server i.e. [account|container|object]
key – header key
- Returns:
True if the key satisfies the test, False otherwise
- swift.common.request_helpers.remove_items(headers, condition)¶
Removes items from a dict whose keys satisfy the given condition.
- Parameters:
headers – a dict of headers
condition – a function that will be passed the header key as a single argument and should return True if the header is to be removed.
- Returns:
a dict, possibly empty, of headers that have been removed
- swift.common.request_helpers.resolve_etag_is_at_header(req, metadata)¶
Helper function to resolve an alternative etag value that may be stored in metadata under an alternate name.
The value of the request’s X-Backend-Etag-Is-At header (if it exists) is a comma separated list of alternate names in the metadata at which an alternate etag value may be found. This list is processed in order until an alternate etag is found.
The left most value in X-Backend-Etag-Is-At will have been set by the left most middleware, or if no middleware, by ECObjectController, if an EC policy is in use. The left most middleware is assumed to be the authority on what the etag value of the object content is.
The resolver will work from left to right in the list until it finds a value that is a name in the given metadata. So the left most wins, IF it exists in the metadata.
By way of example, assume the encrypter middleware is installed. If an object is not encrypted then the resolver will not find the encrypter middleware’s alternate etag sysmeta (X-Object-Sysmeta-Crypto-Etag) but will then find the EC alternate etag (if EC policy). But if the object is encrypted then X-Object-Sysmeta-Crypto-Etag is found and used, which is correct because it should be preferred over X-Object-Sysmeta-Ec-Etag.
- Parameters:
req – a swob Request
metadata – a dict containing object metadata
- Returns:
an alternate etag value if any is found, otherwise None
- swift.common.request_helpers.resolve_ignore_range_header(req, metadata)¶
Helper function to remove Range header from request if metadata matching the X-Backend-Ignore-Range-If-Metadata-Present header is found.
- Parameters:
req – a swob Request
metadata – dictionary of object metadata
- swift.common.request_helpers.split_and_validate_path(request, minsegs=1, maxsegs=None, rest_with_last=False)¶
Utility function to split and validate the request path.
- Returns:
result of
split_path()
if everything’s okay, as native strings- Raises:
HTTPBadRequest – if something’s not okay
- swift.common.request_helpers.split_reserved_name(name)¶
Separate a valid reserved name into the component parts.
- Returns:
a list of strings
- swift.common.request_helpers.strip_object_transient_sysmeta_prefix(key)¶
Removes the object transient system metadata prefix from the start of a header key.
- Parameters:
key – header key
- Returns:
stripped header key
- swift.common.request_helpers.strip_sys_meta_prefix(server_type, key)¶
Removes the system metadata prefix for a given server type from the start of a header key.
- Parameters:
server_type – type of backend server i.e. [account|container|object]
key – header key
- Returns:
stripped header key
- swift.common.request_helpers.strip_user_meta_prefix(server_type, key)¶
Removes the user metadata prefix for a given server type from the start of a header key.
- Parameters:
server_type – type of backend server i.e. [account|container|object]
key – header key
- Returns:
stripped header key
- swift.common.request_helpers.update_etag_is_at_header(req, name)¶
Helper function to update an X-Backend-Etag-Is-At header whose value is a list of alternative header names at which the actual object etag may be found. This informs the object server where to look for the actual object etag when processing conditional requests.
Since the proxy server and/or middleware may set alternative etag header names, the value of X-Backend-Etag-Is-At is a comma separated list which the object server inspects in order until it finds an etag value.
- Parameters:
req – a swob Request
name – name of a sysmeta where alternative etag may be found
- swift.common.request_helpers.update_ignore_range_header(req, name)¶
Helper function to update an X-Backend-Ignore-Range-If-Metadata-Present header whose value is a list of header names which, if any are present on an object, mean the object server should respond with a 200 instead of a 206 or 416.
- Parameters:
req – a swob Request
name – name of a header which, if found, indicates the proxy will want the whole object
- swift.common.request_helpers.validate_container_params(req)¶
- swift.common.request_helpers.validate_internal_account(account)¶
Validate internal account name.
- Raises:
HTTPBadRequest
- swift.common.request_helpers.validate_internal_container(account, container)¶
Validate internal account and container names.
- Raises:
HTTPBadRequest
- swift.common.request_helpers.validate_internal_obj(account, container, obj)¶
Validate internal account, container and object names.
- Raises:
HTTPBadRequest
- swift.common.request_helpers.validate_params(req, names)¶
Get list of parameters from an HTTP request, validating the encoding of each parameter.
- Parameters:
req – request object
names – parameter names
- Returns:
a dict mapping parameter names to values for each name that appears in the request parameters
- Raises:
HTTPBadRequest – if any parameter value is not a valid UTF-8 byte sequence
Swob¶
Implementation of WSGI Request and Response objects.
This library has a very similar API to Webob. It wraps WSGI request environments and response values into objects that are more friendly to interact with.
Why Swob and not just use WebOb? By Michael Barton
We used webob for years. The main problem was that the interface wasn’t stable. For a while, each of our several test suites required a slightly different version of webob to run, and none of them worked with the then-current version. It was a huge headache, so we just scrapped it.
This is kind of a ton of code, but it’s also been a huge relief to not have to scramble to add a bunch of code branches all over the place to keep Swift working every time webob decides some interface needs to change.
- class swift.common.swob.Accept(headerval)¶
Bases:
object
Wraps a Request’s Accept header as a friendly object.
- Parameters:
headerval – value of the header as a str
- best_match(options)¶
Returns the item from “options” that best matches the accept header. Returns None if no available options are acceptable to the client.
- Parameters:
options – a list of content-types the server can respond with
- Raises:
ValueError – if the header is malformed
- class swift.common.swob.HeaderEnvironProxy(environ)¶
Bases:
MutableMapping
A dict-like object that proxies requests to a wsgi environ, rewriting header keys to environ keys.
For example, headers[‘Content-Range’] sets and gets the value of headers.environ[‘HTTP_CONTENT_RANGE’]
- keys() a set-like object providing a view on D's keys ¶
- class swift.common.swob.Match(headerval)¶
Bases:
object
Wraps a Request’s If-[None-]Match header as a friendly object.
- Parameters:
headerval – value of the header as a str
- class swift.common.swob.Range(headerval)¶
Bases:
object
Wraps a Request’s Range header as a friendly object. After initialization, “range.ranges” is populated with a list of (start, end) tuples denoting the requested ranges.
If there were any syntactically-invalid byte-range-spec values, the constructor will raise a ValueError, per the relevant RFC:
“The recipient of a byte-range-set that includes one or more syntactically invalid byte-range-spec values MUST ignore the header field that includes that byte-range-set.”
According to the RFC 2616 specification, the following cases will be all considered as syntactically invalid, thus, a ValueError is thrown so that the range header will be ignored. If the range value contains at least one of the following cases, the entire range is considered invalid, ValueError will be thrown so that the header will be ignored.
value not starts with bytes=
range value start is greater than the end, eg. bytes=5-3
range does not have start or end, eg. bytes=-
range does not have hyphen, eg. bytes=45
range value is non numeric
any combination of the above
Every syntactically valid range will be added into the ranges list even when some of the ranges may not be satisfied by underlying content.
- Parameters:
headerval – value of the header as a str
- ranges_for_length(length)¶
This method is used to return multiple ranges for a given length which should represent the length of the underlying content. The constructor method __init__ made sure that any range in ranges list is syntactically valid. So if length is None or size of the ranges is zero, then the Range header should be ignored which will eventually make the response to be 200.
If an empty list is returned by this method, it indicates that there are unsatisfiable ranges found in the Range header, 416 will be returned.
if a returned list has at least one element, the list indicates that there is at least one range valid and the server should serve the request with a 206 status code.
The start value of each range represents the starting position in the content, the end value represents the ending position. This method purposely adds 1 to the end number because the spec defines the Range to be inclusive.
The Range spec can be found at the following link: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1
- Parameters:
length – length of the underlying content
- class swift.common.swob.Request(environ)¶
Bases:
object
WSGI Request object.
- property accept¶
Retrieve and set the accept property in the WSGI environ, as a Accept object
- property acl¶
Get and set the swob.ACL property in the WSGI environment
- classmethod blank(path, environ=None, headers=None, body=None, **kwargs)¶
Create a new request object with the given parameters, and an environment otherwise filled in with non-surprising default values.
- Parameters:
path – encoded, parsed, and unquoted into PATH_INFO
environ – WSGI environ dictionary
headers – HTTP headers
body – stuffed in a WsgiBytesIO and hung on wsgi.input
kwargs – any environ key with an property setter
- property body¶
Get and set the request body str
- property body_file¶
Get and set the wsgi.input property in the WSGI environment
- call_application(application)¶
Calls the application with this request’s environment. Returns the status, headers, and app_iter for the response as a tuple.
- Parameters:
application – the WSGI application to call
- property content_length¶
Retrieve and set the content-length header as an int
- copy_get()¶
Makes a copy of the request, converting it to a GET.
- ensure_x_timestamp()¶
Similar to
timestamp
, but theX-Timestamp
header will be set if not present.- Raises:
HTTPBadRequest – if X-Timestamp is already set but not a valid
Timestamp
- Returns:
the request’s X-Timestamp header, as a
Timestamp
- get_response(application)¶
Calls the application with this request’s environment. Returns a Response object that wraps up the application’s result.
- Parameters:
application – the WSGI application to call
- property host¶
Get and set the HTTP_HOST property in the WSGI environment
- property host_url¶
Get url for request/response up to path
- property if_match¶
Retrieve and set the if-match property in the WSGI environ, as a Match object
- property if_modified_since¶
Retrieve and set the if-modified-since header as a datetime, set it with a datetime, int, or str
- property if_none_match¶
Retrieve and set the if-none-match property in the WSGI environ, as a Match object
- property if_unmodified_since¶
Retrieve and set the if-unmodified-since header as a datetime, set it with a datetime, int, or str
- message_length()¶
Properly determine the message length for this request. It will return an integer if the headers explicitly contain the message length, or None if the headers don’t contain a length. The ValueError exception will be raised if the headers are invalid.
- Raises:
ValueError – if either transfer-encoding or content-length headers have bad values
AttributeError – if the last value of the transfer-encoding header is not “chunked”
- property method¶
Get and set the REQUEST_METHOD property in the WSGI environment
- property params¶
Provides QUERY_STRING parameters as a dictionary
- property path¶
Provides the full path of the request, excluding the QUERY_STRING
- property path_info¶
Get and set the PATH_INFO property in the WSGI environment
- path_info_pop()¶
Takes one path portion (delineated by slashes) from the path_info, and appends it to the script_name. Returns the path segment.
- property path_qs¶
The path of the request, without host but with query string.
- property query_string¶
Get and set the QUERY_STRING property in the WSGI environment
- property range¶
Retrieve and set the range property in the WSGI environ, as a Range object
- property referer¶
Get and set the HTTP_REFERER property in the WSGI environment
- property referrer¶
Get and set the HTTP_REFERER property in the WSGI environment
- property remote_addr¶
Get and set the REMOTE_ADDR property in the WSGI environment
- property remote_user¶
Get and set the REMOTE_USER property in the WSGI environment
- property script_name¶
Get and set the SCRIPT_NAME property in the WSGI environment
- split_path(minsegs=1, maxsegs=None, rest_with_last=False)¶
Validate and split the Request’s path.
Examples:
['a'] = split_path('/a') ['a', None] = split_path('/a', 1, 2) ['a', 'c'] = split_path('/a/c', 1, 2) ['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)
- Parameters:
minsegs – Minimum number of segments to be extracted
maxsegs – Maximum number of segments to be extracted
rest_with_last – If True, trailing data will be returned as part of last segment. If False, and there is trailing data, raises ValueError.
- Returns:
list of segments with a length of maxsegs (non-existent segments will return as None)
- Raises:
ValueError – if given an invalid path
- property str_params¶
Provides QUERY_STRING parameters as a dictionary
- property swift_entity_path¶
Provides the (native string) account/container/object path, sans API version.
This can be useful when constructing a path to send to a backend server, as that path will need everything after the “/v1”.
- property timestamp¶
Provides HTTP_X_TIMESTAMP as a
Timestamp
- property url¶
Provides the full url of the request
- property user_agent¶
Get and set the HTTP_USER_AGENT property in the WSGI environment
- class swift.common.swob.Response(body=None, status=200, headers=None, app_iter=None, request=None, conditional_response=False, conditional_etag=None, **kw)¶
Bases:
object
WSGI Response object.
- __call__(env, start_response)¶
Respond to the WSGI request.
Warning
This will translate any relative Location header value to an absolute URL using the WSGI environment’s HOST_URL as a prefix, as RFC 2616 specifies.
However, it is quite common to use relative redirects, especially when it is difficult to know the exact HOST_URL the browser would have used when behind several CNAMEs, CDN services, etc. All modern browsers support relative redirects.
To skip over RFC enforcement of the Location header value, you may set
env['swift.leave_relative_location'] = True
in the WSGI environment.
- absolute_location()¶
Attempt to construct an absolute location.
- property accept_ranges¶
Retrieve and set the accept-ranges header
- property app_iter¶
Retrieve and set the response app_iter
- property body¶
Retrieve and set the Response body str
- property charset¶
Retrieve and set the response charset
- property conditional_etag¶
The conditional_etag keyword argument for Response will allow the conditional match value of a If-Match request to be compared to a non-standard value.
This is available for Storage Policies that do not store the client object data verbatim on the storage nodes, but still need support conditional requests.
It’s most effectively used with X-Backend-Etag-Is-At which would define the additional Metadata key(s) where the original ETag of the clear-form client request data may be found.
- property content_length¶
Retrieve and set the content-length header as an int
- property content_range¶
Retrieve and set the content-range header
- property content_type¶
Retrieve and set the response Content-Type header
- property etag¶
Retrieve and set the response Etag header
- fix_conditional_response()¶
You may call this once you have set the content_length to the whole object length and body or app_iter to reset the content_length properties on the request.
It is ok to not call this method, the conditional response will be maintained for you when you __call__ the response.
- property host_url¶
Get url for request/response up to path
- property last_modified¶
Retrieve and set the last-modified header as a datetime, set it with a datetime, int, or str
- property location¶
Retrieve and set the location header
- property status¶
Retrieve and set the Response status, e.g. ‘200 OK’
- www_authenticate()¶
Construct a suitable value for WWW-Authenticate response header
If we have a request and a valid-looking path, the realm is the account; otherwise we set it to ‘unknown’.
- class swift.common.swob.StatusMap¶
Bases:
object
A dict-like object that returns HTTPException subclasses/factory functions where the given key is the status code.
- class swift.common.swob.WsgiBytesIO(initial_bytes=b'')¶
Bases:
BytesIO
This class adds support for the additional wsgi.input methods defined on eventlet.wsgi.Input to the BytesIO class which would otherwise be a fine stand-in for the file-like object in the WSGI environment.
- swift.common.swob.wsgify(func)¶
A decorator for translating functions which take a swob Request object and return a Response object into WSGI callables. Also catches any raised HTTPExceptions and treats them as a returned Response.
Utils¶
Miscellaneous utility functions for use with Swift.
- swift.common.utils.ATTRIBUTES_RE = re.compile('(\\w+)=(".*?"|[^";]+)(; ?|$)')¶
Regular expression to match form attributes.
- class swift.common.utils.CloseableChain(*iterables)¶
Bases:
ClosingIterator
Like itertools.chain, but with a close method that will attempt to invoke its sub-iterators’ close methods, if any.
- class swift.common.utils.ClosingIterator(iterable, other_closeables=None)¶
Bases:
object
Wrap another iterator and close it, if possible, on completion/exception.
If other closeable objects are given then they will also be closed when this iterator is closed.
This is particularly useful for ensuring a generator properly closes its resources, even if the generator was never started.
This class may be subclassed to override the behavior of
_get_next_item
.- Parameters:
iterable – iterator to wrap.
other_closeables – other resources to attempt to close.
- class swift.common.utils.ClosingMapper(function, iterable, other_closeables=None)¶
Bases:
ClosingIterator
A closing iterator that yields the result of
function
as it is applied to each item ofiterable
.Note that while this behaves similarly to the built-in
map
function,other_closeables
does not have the same semantic as theiterables
argument ofmap
.- Parameters:
function – a function that will be called with each item of
iterable
before yielding its result.iterable – iterator to wrap.
other_closeables – other resources to attempt to close.
- class swift.common.utils.ContextPool(size=1000)¶
Bases:
GreenPool
GreenPool subclassed to kill its coros when it gets gc’ed
- class swift.common.utils.CooperativeIterator(iterable, period=5)¶
Bases:
ClosingIterator
Wrapper to make a deliberate periodic call to
sleep()
while iterating over wrapped iterator, providing an opportunity to switch greenthreads.This is for fairness; if the network is outpacing the CPU, we’ll always be able to read and write data without encountering an EWOULDBLOCK, and so eventlet will not switch greenthreads on its own. We do it manually so that clients don’t starve.
The number 5 here was chosen by making stuff up. It’s not every single chunk, but it’s not too big either, so it seemed like it would probably be an okay choice.
Note that we may trampoline to other greenthreads more often than once every 5 chunks, depending on how blocking our network IO is; the explicit sleep here simply provides a lower bound on the rate of trampolining.
- Parameters:
iterable – iterator to wrap.
period – number of items yielded from this iterator between calls to
sleep()
; a negative value or 0 mean that cooperative sleep will be disabled.
- class swift.common.utils.Everything¶
Bases:
object
A container that contains everything. If “e” is an instance of Everything, then “x in e” is true for all x.
- class swift.common.utils.GreenAsyncPile(size_or_pool)¶
Bases:
object
Runs jobs in a pool of green threads, and the results can be retrieved by using this object as an iterator.
This is very similar in principle to eventlet.GreenPile, except it returns results as they become available rather than in the order they were launched.
Correlating results with jobs (if necessary) is left to the caller.
- spawn(func, *args, **kwargs)¶
Spawn a job in a green thread on the pile.
- waitall(timeout)¶
Wait timeout seconds for any results to come in.
- Parameters:
timeout – seconds to wait for results
- Returns:
list of results accrued in that time
- waitfirst(timeout)¶
Wait up to timeout seconds for first result to come in.
- Parameters:
timeout – seconds to wait for results
- Returns:
first item to come back, or None
- exception swift.common.utils.GreenAsyncPileWaitallTimeout(seconds=None, exception=None)¶
Bases:
Timeout
- class swift.common.utils.GreenthreadSafeIterator(unsafe_iterable)¶
Bases:
object
Wrap an iterator to ensure that only one greenthread is inside its next() method at a time.
This is useful if an iterator’s next() method may perform network IO, as that may trigger a greenthread context switch (aka trampoline), which can give another greenthread a chance to call next(). At that point, you get an error like “ValueError: generator already executing”. By wrapping calls to next() with a mutex, we avoid that error.
- class swift.common.utils.InputProxy(wsgi_input)¶
Bases:
object
File-like object that counts bytes read. To be swapped in for wsgi.input for accounting purposes.
- read(*args, **kwargs)¶
Pass read request to the underlying file-like object and add bytes read to total.
- readline(*args, **kwargs)¶
Pass readline request to the underlying file-like object and add bytes read to total.
- exception swift.common.utils.InvalidHashPathConfigError¶
Bases:
ValueError
- class swift.common.utils.LRUCache(maxsize=1000, maxtime=3600)¶
Bases:
object
Decorator for size/time bound memoization that evicts the least recently used members.
- class swift.common.utils.Namespace(name, lower, upper)¶
Bases:
object
A Namespace encapsulates parameters that define a range of the object namespace.
- Parameters:
name – the name of the
Namespace
; this SHOULD take the form of a path to a container i.e. <account_name>/<container_name>.lower – the lower bound of object names contained in the namespace; the lower bound is not included in the namespace.
upper – the upper bound of object names contained in the namespace; the upper bound is included in the namespace.
- class MaxBound¶
Bases:
NamespaceOuterBound
- class MinBound¶
Bases:
NamespaceOuterBound
- entire_namespace()¶
Returns True if this namespace includes the entire namespace, False otherwise.
- expand(donors)¶
Expands the bounds as necessary to match the minimum and maximum bounds of the given donors.
- Parameters:
donors – A list of
Namespace
- Returns:
True if the bounds have been modified, False otherwise.
- class swift.common.utils.NamespaceOuterBound¶
Bases:
object
A custom singleton type to be subclassed for the outer bounds of Namespaces.
- class swift.common.utils.OverrideOptions(devices, partitions, policies)¶
Bases:
tuple
- devices¶
Alias for field number 0
- partitions¶
Alias for field number 1
- policies¶
Alias for field number 2
- class swift.common.utils.RateLimitedIterator(iterable, elements_per_second, limit_after=0, ratelimit_if=<function RateLimitedIterator.<lambda>>)¶
Bases:
object
Wrap an iterator to only yield elements at a rate of N per second.
- Parameters:
iterable – iterable to wrap
elements_per_second – the rate at which to yield elements
limit_after – rate limiting kicks in only after yielding this many elements; default is 0 (rate limit immediately)
- class swift.common.utils.ShardName(account, root_container, parent_container_hash, timestamp, index)¶
Bases:
object
Encapsulates the components of a shard name.
Instances of this class would typically be constructed via the create() or parse() class methods.
Shard names have the form:
<account>/<root_container>-<parent_container_hash>-<timestamp>-<index>
Note: some instances of
ShardRange
have names that will NOT parse as aShardName
; e.g. a root container’s own shard range will have a name format of <account>/<root_container> which will raise ValueError if passed to parse.- classmethod create(account, root_container, parent_container, timestamp, index)¶
Create an instance of
ShardName
.- Parameters:
account – the hidden internal account to which the shard container belongs.
root_container – the name of the root container for the shard.
parent_container – the name of the parent container for the shard; for initial first generation shards this should be the same as
root_container
; for shards of shards this should be the name of the sharding shard container.timestamp – an instance of
Timestamp
index – a unique index that will distinguish the path from any other path generated using the same combination of
account
,root_container
,parent_container
andtimestamp
.
- Returns:
an instance of
ShardName
.- Raises:
ValueError – if any argument is None
- classmethod hash_container_name(container_name)¶
Calculates the hash of a container name.
- Parameters:
container_name – name to be hashed.
- Returns:
the hexdigest of the md5 hash of
container_name
.- Raises:
ValueError – if
container_name
is None.
- class swift.common.utils.ShardRange(name, timestamp=0, lower=MinBound, upper=MaxBound, object_count=0, bytes_used=0, meta_timestamp=None, deleted=False, state=None, state_timestamp=None, epoch=None, reported=False, tombstones=-1, **kwargs)¶
Bases:
Namespace
A ShardRange encapsulates sharding state related to a container including lower and upper bounds that define the object namespace for which the container is responsible.
Shard ranges may be persisted in a container database. Timestamps associated with subsets of the shard range attributes are used to resolve conflicts when a shard range needs to be merged with an existing shard range record and the most recent version of an attribute should be persisted.
- Parameters:
name – the name of the shard range; this MUST take the form of a path to a container i.e. <account_name>/<container_name>.
timestamp – a timestamp that represents the time at which the shard range’s
lower
,upper
ordeleted
attributes were last modified.lower – the lower bound of object names contained in the shard range; the lower bound is not included in the shard range namespace.
upper – the upper bound of object names contained in the shard range; the upper bound is included in the shard range namespace.
object_count – the number of objects in the shard range; defaults to zero.
bytes_used – the number of bytes in the shard range; defaults to zero.
meta_timestamp – a timestamp that represents the time at which the shard range’s
object_count
andbytes_used
were last updated; defaults to the value oftimestamp
.deleted – a boolean; if True the shard range is considered to be deleted.
state – the state; must be one of ShardRange.STATES; defaults to CREATED.
state_timestamp – a timestamp that represents the time at which
state
was forced to its current value; defaults to the value oftimestamp
. This timestamp is typically not updated with every change ofstate
because in general conflicts instate
attributes are resolved by choosing the largerstate
value. However, when this rule does not apply, for example when changing state fromSHARDED
toACTIVE
, thestate_timestamp
may be advanced so that the newstate
value is preferred over any olderstate
value.epoch – optional epoch timestamp which represents the time at which sharding was enabled for a container.
reported – optional indicator that this shard and its stats have been reported to the root container.
tombstones – the number of tombstones in the shard range; defaults to -1 to indicate that the value is unknown.
- copy(timestamp=None, **kwargs)¶
Creates a copy of the ShardRange.
- Parameters:
timestamp – (optional) If given, the returned ShardRange will have all of its timestamps set to this value. Otherwise the returned ShardRange will have the original timestamps.
- Returns:
an instance of
ShardRange
- find_ancestors(shard_ranges)¶
Find this shard range’s ancestor ranges in the given
shard_ranges
.This method makes a best-effort attempt to identify this shard range’s parent shard range, the parent’s parent, etc., up to and including the root shard range. It is only possible to directly identify the parent of a particular shard range, so the search is recursive; if any member of the ancestry is not found then the search ends and older ancestors that may be in the list are not identified. The root shard range, however, will always be identified if it is present in the list.
For example, given a list that contains parent, grandparent, great-great-grandparent and root shard ranges, but is missing the great-grandparent shard range, only the parent, grand-parent and root shard ranges will be identified.
- Parameters:
shard_ranges – a list of instances of
ShardRange
- Returns:
a list of instances of
ShardRange
containing items in the givenshard_ranges
that can be identified as ancestors of this shard range. The list may not be complete if there are gaps in the ancestry, but is guaranteed to contain at least the parent and root shard ranges if they are present.
- find_root(shard_ranges)¶
Find this shard range’s root shard range in the given
shard_ranges
.- Parameters:
shard_ranges – a list of instances of
ShardRange
- Returns:
this shard range’s root shard range if it is found in the list, otherwise None.
- classmethod from_dict(params)¶
Return an instance constructed using the given dict of params. This method is deliberately less flexible than the class __init__() method and requires all of the __init__() args to be given in the dict of params.
- Parameters:
params – a dict of parameters
- Returns:
an instance of this class
- increment_meta(object_count, bytes_used)¶
Increment the object stats metadata by the given values and update the meta_timestamp to the current time.
- Parameters:
object_count – should be an integer
bytes_used – should be an integer
- Raises:
ValueError – if
object_count
orbytes_used
cannot be cast to an int.
- is_child_of(parent)¶
Test if this shard range is a child of another shard range. The parent-child relationship is inferred from the names of the shard ranges. This method is limited to work only within the scope of the same user-facing account (with and without shard prefix).
- Parameters:
parent – an instance of
ShardRange
.- Returns:
True if
parent
is the parent of this shard range, False otherwise, assuming that they are within the same account.
- classmethod make_path(shards_account, root_container, parent_container, timestamp, index)¶
Returns a path for a shard container that is valid to use as a name when constructing a
ShardRange
.- Parameters:
shards_account – the hidden internal account to which the shard container belongs.
root_container – the name of the root container for the shard.
parent_container – the name of the parent container for the shard; for initial first generation shards this should be the same as
root_container
; for shards of shards this should be the name of the sharding shard container.timestamp – an instance of
Timestamp
index – a unique index that will distinguish the path from any other path generated using the same combination of
shards_account
,root_container
,parent_container
andtimestamp
.
- Returns:
a string of the form <account_name>/<container_name>
- classmethod resolve_state(state)¶
Given a value that may be either the name or the number of a state return a tuple of (state number, state name).
- Parameters:
state – Either a string state name or an integer state number.
- Returns:
A tuple (state number, state name)
- Raises:
ValueError – if
state
is neither a valid state name nor a valid state number.
- property row_count¶
Returns the total number of rows in the shard range i.e. the sum of objects and tombstones.
- Returns:
the row count
- set_deleted(timestamp=None)¶
Mark the shard range deleted and set timestamp to the current time.
- Parameters:
timestamp – optional timestamp to set; if not given the current time will be set.
- Returns:
True if the deleted attribute or timestamp was changed, False otherwise
- update_meta(object_count, bytes_used, meta_timestamp=None)¶
Set the object stats metadata to the given values and update the meta_timestamp to the current time.
- Parameters:
object_count – should be an integer
bytes_used – should be an integer
meta_timestamp – timestamp for metadata; if not given the current time will be set.
- Raises:
ValueError – if
object_count
orbytes_used
cannot be cast to an int, or if meta_timestamp is neither None nor can be cast to aTimestamp
.
- update_state(state, state_timestamp=None)¶
Set state to the given value and optionally update the state_timestamp to the given time.
- Parameters:
state – new state, should be an integer
state_timestamp – timestamp for state; if not given the state_timestamp will not be changed.
- Returns:
True if the state or state_timestamp was changed, False otherwise
- update_tombstones(tombstones, meta_timestamp=None)¶
Set the tombstones metadata to the given values and update the meta_timestamp to the current time.
- Parameters:
tombstones – should be an integer
meta_timestamp – timestamp for metadata; if not given the current time will be set.
- Raises:
ValueError – if
tombstones
cannot be cast to an int, or if meta_timestamp is neither None nor can be cast to aTimestamp
.
- class swift.common.utils.ShardRangeList(initlist=None)¶
Bases:
UserList
This class provides some convenience functions for working with lists of
ShardRange
.This class does not enforce ordering or continuity of the list items: callers should ensure that items are added in order as appropriate.
- property bytes_used¶
Returns the total number of bytes in all items in the list.
- Returns:
total bytes used
- filter(includes=None, marker=None, end_marker=None)¶
Filter the list for those shard ranges whose namespace includes the
includes
name or any part of the namespace betweenmarker
andend_marker
. If none ofincludes
,marker
orend_marker
are specified then all shard ranges will be returned.- Parameters:
includes – a string; if not empty then only the shard range, if any, whose namespace includes this string will be returned, and
marker
andend_marker
will be ignored.marker – if specified then only shard ranges whose upper bound is greater than this value will be returned.
end_marker – if specified then only shard ranges whose lower bound is less than this value will be returned.
- Returns:
A new instance of
ShardRangeList
containing the filtered shard ranges.
- find_lower(condition)¶
Finds the first shard range satisfies the given condition and returns its lower bound.
- Parameters:
condition – A function that must accept a single argument of type
ShardRange
and return True if the shard range satisfies the condition or False otherwise.- Returns:
The lower bound of the first shard range to satisfy the condition, or the
upper
value of this list if no such shard range is found.
- includes(other)¶
Check if another ShardRange namespace is enclosed between the list’s
lower
andupper
properties. Note: the list’slower
andupper
properties will only equal the outermost bounds of all items in the list if the list has previously been sorted.Note: the list does not need to contain an item matching
other
for this method to return True, although if the list has been sorted and does contain an item matchingother
then the method will return True.- Parameters:
other – an instance of
ShardRange
- Returns:
True if other’s namespace is enclosed, False otherwise.
- property lower¶
Returns the lower bound of the first item in the list. Note: this will only be equal to the lowest bound of all items in the list if the list contents has been sorted.
- Returns:
lower bound of first item in the list, or Namespace.MIN if the list is empty.
- property object_count¶
Returns the total number of objects of all items in the list.
- Returns:
total object count
- property row_count¶
Returns the total number of rows of all items in the list.
- Returns:
total row count
- property upper¶
Returns the upper bound of the last item in the list. Note: this will only be equal to the uppermost bound of all items in the list if the list has previously been sorted.
- Returns:
upper bound of last item in the list, or Namespace.MIN if the list is empty.
- class swift.common.utils.Spliterator(source_iterable)¶
Bases:
object
Takes an iterator yielding sliceable things (e.g. strings or lists) and yields subiterators, each yielding up to the requested number of items from the source.
>>> si = Spliterator(["abcde", "fg", "hijkl"]) >>> ''.join(si.take(4)) "abcd" >>> ''.join(si.take(3)) "efg" >>> ''.join(si.take(1)) "h" >>> ''.join(si.take(3)) "ijk" >>> ''.join(si.take(3)) "l" # shorter than requested; this can happen with the last iterator
- class swift.common.utils.StreamingPile(size)¶
Bases:
GreenAsyncPile
Runs jobs in a pool of green threads, spawning more jobs as results are retrieved and worker threads become available.
When used as a context manager, has the same worker-killing properties as
ContextPool
.- asyncstarmap(func, args_iter)¶
This is the same as
itertools.starmap()
, except that func is executed in a separate green thread for each item, and results won’t necessarily have the same order as inputs.
- class swift.common.utils.StringAlong(iterable, other_iter, unexpected_items_func)¶
Bases:
ClosingIterator
This iterator wraps and iterates over a first iterator until it stops, and then iterates a second iterator, expecting it to stop immediately. This “stringing along” of the second iterator is useful when the exit of the second iterator must be delayed until the first iterator has stopped. For example, when the second iterator has already yielded its item(s) but has resources that mustn’t be garbage collected until the first iterator has stopped.
The second iterator is expected to have no more items and raise StopIteration when called. If this is not the case then
unexpected_items_func
is called.- Parameters:
iterable – a first iterator that is wrapped and iterated.
other_iter – a second iterator that is stopped once the first iterator has stopped.
unexpected_items_func – a no-arg function that will be called if the second iterator is found to have remaining items.
- class swift.common.utils.Watchdog¶
Bases:
object
Implements a watchdog to efficiently manage concurrent timeouts.
Compared to eventlet.timeouts.Timeout, it reduces the number of context switching in eventlet by avoiding to schedule actions (throw an Exception), then unschedule them if the timeouts are cancelled.
- at T+0, request timeout(10)
=> wathdog greenlet sleeps 10 seconds
- at T+1, request timeout(15)
- => the timeout will expire after the current, no need to wake up the
watchdog greenlet
- at T+2, request timeout(5)
- => the timeout will expire before the first timeout, wake up the
watchdog greenlet to calculate a new sleep period
- at T+7, the 3rd timeout expires
- => the exception is raised, then the greenlet watchdog sleep(3) to
wake up for the 1st timeout expiration
- kill()¶
Stop the watchdog greenthread.
- spawn()¶
Start the watchdog greenthread.
- start(timeout, exc, timeout_at=None)¶
Schedule a timeout action
- Parameters:
timeout – duration before the timeout expires
exc – exception to throw when the timeout expire, must inherit from eventlet.Timeout
timeout_at – allow to force the expiration timestamp
- Returns:
id of the scheduled timeout, needed to cancel it
- stop(key)¶
Cancel a scheduled timeout
- Parameters:
key – timeout id, as returned by start()
- class swift.common.utils.WatchdogTimeout(watchdog, timeout, exc, timeout_at=None)¶
Bases:
object
Context manager to schedule a timeout in a Watchdog instance
- swift.common.utils.audit_location_generator(devices, datadir, suffix='', mount_check=True, logger=None, devices_filter=None, partitions_filter=None, suffixes_filter=None, hashes_filter=None, hook_pre_device=None, hook_post_device=None, hook_pre_partition=None, hook_post_partition=None, hook_pre_suffix=None, hook_post_suffix=None, hook_pre_hash=None, hook_post_hash=None, error_counter=None, yield_hash_dirs=False)¶
Given a devices path and a data directory, yield (path, device, partition) for all files in that directory
(devices|partitions|suffixes|hashes)_filter are meant to modify the list of elements that will be iterated. eg: they can be used to exclude some elements based on a custom condition defined by the caller.
hook_pre_(device|partition|suffix|hash) are called before yielding the element, hook_pos_(device|partition|suffix|hash) are called after the element was yielded. They are meant to do some pre/post processing. eg: saving a progress status.
- Parameters:
devices – parent directory of the devices to be audited
datadir – a directory located under self.devices. This should be one of the DATADIR constants defined in the account, container, and object servers.
suffix – path name suffix required for all names returned (ignored if yield_hash_dirs is True)
mount_check – Flag to check if a mount check should be performed on devices
logger – a logger object
devices_filter – a callable taking (devices, [list of devices]) as parameters and returning a [list of devices]
partitions_filter – a callable taking (datadir_path, [list of parts]) as parameters and returning a [list of parts]
suffixes_filter – a callable taking (part_path, [list of suffixes]) as parameters and returning a [list of suffixes]
hashes_filter – a callable taking (suff_path, [list of hashes]) as parameters and returning a [list of hashes]
hook_pre_device – a callable taking device_path as parameter
hook_post_device – a callable taking device_path as parameter
hook_pre_partition – a callable taking part_path as parameter
hook_post_partition – a callable taking part_path as parameter
hook_pre_suffix – a callable taking suff_path as parameter
hook_post_suffix – a callable taking suff_path as parameter
hook_pre_hash – a callable taking hash_path as parameter
hook_post_hash – a callable taking hash_path as parameter
error_counter – a dictionary used to accumulate error counts; may add keys ‘unmounted’ and ‘unlistable_partitions’
yield_hash_dirs – if True, yield hash dirs instead of individual files
- swift.common.utils.backward(f, blocksize=4096)¶
A generator returning lines from a file starting with the last line, then the second last line, etc. i.e., it reads lines backwards. Stops when the first line (if any) is read. This is useful when searching for recent activity in very large files.
- Parameters:
f – file object to read
blocksize – no of characters to go backwards at each block
- swift.common.utils.cache_from_env(env, allow_none=False)¶
Get memcache connection pool from the environment (which had been previously set by the memcache middleware
- Parameters:
env – wsgi environment dict
- Returns:
swift.common.memcached.MemcacheRing from environment
- swift.common.utils.closing_if_possible(maybe_closable)¶
Like contextlib.closing(), but doesn’t crash if the object lacks a close() method.
PEP 333 (WSGI) says: “If the iterable returned by the application has a close() method, the server or gateway must call that method upon completion of the current request[.]” This function makes that easier.
- swift.common.utils.compute_eta(start_time, current_value, final_value)¶
Compute an ETA. Now only if we could also have a progress bar…
- Parameters:
start_time – Unix timestamp when the operation began
current_value – Current value
final_value – Final value
- Returns:
ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)
- swift.common.utils.csv_append(csv_string, item)¶
Appends an item to a comma-separated string.
If the comma-separated string is empty/None, just returns item.
- swift.common.utils.distribute_evenly(items, num_buckets)¶
Distribute items as evenly as possible into N buckets.
- swift.common.utils.document_iters_to_http_response_body(ranges_iter, boundary, multipart, logger)¶
Takes an iterator of range iters and turns it into an appropriate HTTP response body, whether that’s multipart/byteranges or not.
This is almost, but not quite, the inverse of request_helpers.http_response_to_document_iters(). This function only yields chunks of the body, not any headers.
- Parameters:
ranges_iter –
an iterator of dictionaries, one per range. Each dictionary must contain at least the following key: “part_iter”: iterator yielding the bytes in the range
Additionally, if multipart is True, then the following other keys are required:
”start_byte”: index of the first byte in the range “end_byte”: index of the last byte in the range “content_type”: value for the range’s Content-Type header
- Finally, there is one optional key that is used in the
multipart/byteranges case:
- ”entity_length”: length of the requested entity (not necessarily
equal to the response length). If omitted, “*” will be used.
Each part_iter will be exhausted prior to calling next(ranges_iter).
boundary – MIME boundary to use, sans dashes (e.g. “boundary”, not “–boundary”).
multipart – True if the response should be multipart/byteranges, False otherwise. This should be True if and only if you have 2 or more ranges.
logger – a logger
- swift.common.utils.document_iters_to_multipart_byteranges(ranges_iter, boundary)¶
Takes an iterator of range iters and yields a multipart/byteranges MIME document suitable for sending as the body of a multi-range 206 response.
See document_iters_to_http_response_body for parameter descriptions.
- swift.common.utils.drain_and_close(response_or_app_iter, read_limit=None)¶
Drain and close a swob or WSGI response.
This ensures we don’t log a 499 in the proxy just because we realized we don’t care about the body of an error.
- swift.common.utils.drop_privileges(user)¶
Sets the userid/groupid of the current process, get session leader, etc.
- Parameters:
user – User name to change privileges to
- swift.common.utils.dump_recon_cache(cache_dict, cache_file, logger, lock_timeout=2, set_owner=None)¶
Update recon cache values
- Parameters:
cache_dict – Dictionary of cache key/value pairs to write out
cache_file – cache file to update
logger – the logger to use to log an encountered error
lock_timeout – timeout (in seconds)
set_owner – Set owner of recon cache file
- swift.common.utils.eventlet_monkey_patch()¶
Install the appropriate Eventlet monkey patches.
- swift.common.utils.extract_swift_bytes(content_type)¶
- Parse a content-type and return a tuple containing:
the content_type string minus any swift_bytes param,
the swift_bytes value or None if the param was not found
- Parameters:
content_type – a content-type string
- Returns:
a tuple of (content-type, swift_bytes or None)
- swift.common.utils.fallocate(fd, size, offset=0)¶
Pre-allocate disk space for a file.
This function can be disabled by calling disable_fallocate(). If no suitable C function is available in libc, this function is a no-op.
- Parameters:
fd – file descriptor
size – size to allocate (in bytes)
- swift.common.utils.fdatasync(fd)¶
Sync modified file data to disk.
- Parameters:
fd – file descriptor
- swift.common.utils.filter_namespaces(namespaces, includes, marker, end_marker)¶
Filter the given Namespaces/ShardRanges to those whose namespace includes the
includes
name or any part of the namespace betweenmarker
andend_marker
. If none ofincludes
,marker
orend_marker
are specified then all Namespaces will be returned.- Parameters:
namespaces – A list of
Namespace
orShardRange
.includes – a string; if not empty then only the Namespace, if any, whose namespace includes this string will be returned,
marker
andend_marker
will be ignored.marker – if specified then only shard ranges whose upper bound is greater than this value will be returned.
end_marker – if specified then only shard ranges whose lower bound is less than this value will be returned.
- Returns:
A filtered list of
Namespace
.
- swift.common.utils.find_namespace(item, namespaces)¶
Find a Namespace/ShardRange in given list of
namespaces
whose namespace containsitem
.- Parameters:
item – The item for a which a Namespace is to be found.
ranges – a sorted list of Namespaces.
- Returns:
the Namespace/ShardRange whose namespace contains
item
, or None if no suitable Namespace is found.
- swift.common.utils.friendly_close(resp)¶
Close a swob or WSGI response and maybe drain it.
It’s basically free to “read” a HEAD or HTTPException response - the bytes are probably already in our network buffers. For a larger response we could possibly burn a lot of CPU/network trying to drain an un-used response. This method will read up to DEFAULT_DRAIN_LIMIT bytes to avoid logging a 499 in the proxy when it would otherwise be easy to just throw away the small/empty body.
- swift.common.utils.fs_has_free_space(fs_path_or_fd, space_needed, is_percent)¶
Check to see whether or not a filesystem has the given amount of space free. Unlike fallocate(), this does not reserve any space.
- Parameters:
fs_path_or_fd – path to a file or directory on the filesystem, or an open file descriptor; if a directory, typically the path to the filesystem’s mount point
space_needed – minimum bytes or percentage of free space
is_percent – if True, then space_needed is treated as a percentage of the filesystem’s capacity; if False, space_needed is a number of free bytes.
- Returns:
True if the filesystem has at least that much free space, False otherwise
- Raises:
OSError – if fs_path does not exist
- swift.common.utils.fsync(fd)¶
Sync modified file data and metadata to disk.
- Parameters:
fd – file descriptor
- swift.common.utils.fsync_dir(dirpath)¶
Sync directory entries to disk.
- Parameters:
dirpath – Path to the directory to be synced.
- swift.common.utils.get_db_files(db_path)¶
Given the path to a db file, return a sorted list of all valid db files that actually exist in that path’s dir. A valid db filename has the form:
<hash>[_<epoch>].db
where <hash> matches the <hash> part of the given db_path as would be parsed by
parse_db_filename()
.- Parameters:
db_path – Path to a db file that does not necessarily exist.
- Returns:
List of valid db files that do exist in the dir of the
db_path
. This list may be empty.
- swift.common.utils.get_expirer_container(x_delete_at, expirer_divisor, acc, cont, obj)¶
Returns an expiring object container name for given X-Delete-At and (native string) a/c/o.
- swift.common.utils.get_hub()¶
Checks whether poll is available and falls back on select if it isn’t.
Note about epoll:
Review: https://review.opendev.org/#/c/18806/
There was a problem where once out of every 30 quadrillion connections, a coroutine wouldn’t wake up when the client closed its end. Epoll was not reporting the event or it was getting swallowed somewhere. Then when that file descriptor was re-used, eventlet would freak right out because it still thought it was waiting for activity from it in some other coro.
Another note about epoll: it’s hard to use when forking. epoll works like so:
create an epoll instance:
efd = epoll_create(...)
register file descriptors of interest with
epoll_ctl(efd, EPOLL_CTL_ADD, fd, ...)
wait for events with
epoll_wait(efd, ...)
If you fork, you and all your child processes end up using the same epoll instance, and everyone becomes confused. It is possible to use epoll and fork and still have a correct program as long as you do the right things, but eventlet doesn’t do those things. Really, it can’t even try to do those things since it doesn’t get notified of forks.
In contrast, both poll() and select() specify the set of interesting file descriptors with each call, so there’s no problem with forking.
As eventlet monkey patching is now done before call get_hub() in wsgi.py if we use ‘import select’ we get the eventlet version, but since version 0.20.0 eventlet removed select.poll() function in patched select (see: http://eventlet.net/doc/changelog.html and https://github.com/eventlet/eventlet/commit/614a20462).
We use eventlet.patcher.original function to get python select module to test if poll() is available on platform.
- swift.common.utils.get_partition_for_hash(hex_hash, part_power)¶
Return partition number for given hex hash and partition power. :param hex_hash: A hash string :param part_power: partition power :returns: partition number
- swift.common.utils.get_partition_from_path(devices, path)¶
- Parameters:
devices – directory where devices are mounted (e.g. /srv/node)
path – full path to a object file or hashdir
- Returns:
the (integer) partition from the path
- swift.common.utils.get_redirect_data(response)¶
Extract a redirect location from a response’s headers.
- Parameters:
response – a response
- Returns:
a tuple of (path, Timestamp) if a Location header is found, otherwise None
- Raises:
ValueError – if the Location header is found but a X-Backend-Redirect-Timestamp is not found, or if there is a problem with the format of etiher header
- swift.common.utils.get_time_units(time_amount)¶
Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)
- Parameters:
time_amount – length of time in seconds
- Returns:
A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)
- swift.common.utils.get_zero_indexed_base_string(base, index)¶
This allows the caller to make a list of things with indexes, where the first item (zero indexed) is just the bare base string, and subsequent indexes are appended ‘-1’, ‘-2’, etc.
e.g.:
'lock', None => 'lock' 'lock', 0 => 'lock' 'lock', 1 => 'lock-1' 'object', 2 => 'object-2'
- Parameters:
base – a string, the base string; when
index
is 0 (or None) this is the identity function.index – a digit, typically an integer (or None); for values other than 0 or None this digit is appended to the base string separated by a hyphen.
- swift.common.utils.hash_path(account, container=None, object=None, raw_digest=False)¶
Get the canonical hash for an account/container/object
- Parameters:
account – Account
container – Container
object – Object
raw_digest – If True, return the raw version rather than a hex digest
- Returns:
hash string
- swift.common.utils.human_readable(value)¶
Returns the number in a human readable format; for example 1048576 = “1Mi”.
- swift.common.utils.is_file_older(path, age)¶
Test if a file mtime is older than the given age, suppressing any OSErrors.
- Parameters:
path – first and only argument passed to os.stat
age – age in seconds
- Returns:
True if age is less than or equal to zero or if the file mtime is more than
age
in the past; False if age is greater than zero and the file mtime is less than or equal toage
in the past or if there is an OSError while stat’ing the file.
- swift.common.utils.ismount(path)¶
Test whether a path is a mount point. This will catch any exceptions and translate them into a False return value Use ismount_raw to have the exceptions raised instead.
- swift.common.utils.ismount_raw(path)¶
Test whether a path is a mount point. Whereas ismount will catch any exceptions and just return False, this raw version will not catch exceptions.
This is code hijacked from C Python 2.6.8, adapted to remove the extra lstat() system call.
- swift.common.utils.item_from_env(env, item_name, allow_none=False)¶
Get a value from the wsgi environment
- Parameters:
env – wsgi environment dict
item_name – name of item to get
- Returns:
the value from the environment
- swift.common.utils.iter_multipart_mime_documents(wsgi_input, boundary, read_chunk_size=4096)¶
Given a multi-part-mime-encoded input file object and boundary, yield file-like objects for each part. Note that this does not split each part into headers and body; the caller is responsible for doing that if necessary.
- Parameters:
wsgi_input – The file-like object to read from.
boundary – The mime boundary to separate new file-like objects on.
- Returns:
A generator of file-like objects for each part.
- Raises:
MimeInvalid – if the document is malformed
- swift.common.utils.link_fd_to_path(fd, target_path, dirs_created=0, retries=2, fsync=True)¶
Creates a link to file descriptor at target_path specified. This method does not close the fd for you. Unlike rename, as linkat() cannot overwrite target_path if it exists, we unlink and try again.
Attempts to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.
- Parameters:
fd – File descriptor to be linked
target_path – Path in filesystem where fd is to be linked
dirs_created – Number of newly created directories that needs to be fsync’d.
retries – number of retries to make
fsync – fsync on containing directory of target_path and also all the newly created directories.
- swift.common.utils.list_from_csv(comma_separated_str)¶
Splits the str given and returns a properly stripped list of the comma separated values.
- swift.common.utils.load_recon_cache(cache_file)¶
Load a recon cache file. Treats missing file as empty.
- swift.common.utils.lock_file(filename, timeout=None, append=False, unlink=True)¶
Context manager that acquires a lock on a file. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
- Parameters:
filename – file to be locked
timeout – timeout (in seconds). If None, defaults to DEFAULT_LOCK_TIMEOUT
append – True if file should be opened in append mode
unlink – True if the file should be unlinked at the end
- swift.common.utils.lock_parent_directory(filename, timeout=None)¶
Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
- Parameters:
filename – file path of the parent directory to be locked
timeout – timeout (in seconds). If None, defaults to DEFAULT_LOCK_TIMEOUT
- swift.common.utils.lock_path(directory, timeout=None, timeout_class=None, limit=1, name=None)¶
Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).
For locking exclusively, file or directory has to be opened in Write mode. Python doesn’t allow directories to be opened in Write Mode. So we workaround by locking a hidden file in the directory.
- Parameters:
directory – directory to be locked
timeout – timeout (in seconds). If None, defaults to DEFAULT_LOCK_TIMEOUT
timeout_class – The class of the exception to raise if the lock cannot be granted within the timeout. Will be constructed as timeout_class(timeout, lockpath). Default: LockTimeout
limit – The maximum number of locks that may be held concurrently on the same directory at the time this method is called. Note that this limit is only applied during the current call to this method and does not prevent subsequent calls giving a larger limit. Defaults to 1.
name – A string to distinguishes different type of locks in a directory
- Raises:
TypeError – if limit is not an int.
ValueError – if limit is less than 1.
- swift.common.utils.make_db_file_path(db_path, epoch)¶
Given a path to a db file, return a modified path whose filename part has the given epoch.
A db filename takes the form
<hash>[_<epoch>].db
; this method replaces the<epoch>
part of the givendb_path
with the givenepoch
value, or drops the epoch part if the givenepoch
isNone
.- Parameters:
db_path – Path to a db file that does not necessarily exist.
epoch – A string (or
None
) that will be used as the epoch in the new path’s filename; non-None
values will be normalized to the normal string representation of aTimestamp
.
- Returns:
A modified path to a db file.
- Raises:
ValueError – if the
epoch
is not valid for constructing aTimestamp
.
- swift.common.utils.makedirs_count(path, count=0)¶
Same as os.makedirs() except that this method returns the number of new directories that had to be created.
Also, this does not raise an error if target directory already exists. This behaviour is similar to Python 3.x’s os.makedirs() called with exist_ok=True. Also similar to swift.common.utils.mkdirs()
- swift.common.utils.maybe_multipart_byteranges_to_document_iters(app_iter, content_type)¶
Takes an iterator that may or may not contain a multipart MIME document as well as content type and returns an iterator of body iterators.
- Parameters:
app_iter – iterator that may contain a multipart MIME document
content_type – content type of the app_iter, used to determine whether it conains a multipart document and, if so, what the boundary is between documents
- swift.common.utils.md5_hash_for_file(fname)¶
Get the MD5 checksum of a file.
- Parameters:
fname – path to file
- Returns:
MD5 checksum, hex encoded
- swift.common.utils.memcached_timing_stats(**dec_kwargs)¶
Returns a decorator that logs timing events or errors for public methods in MemcacheRing class, such as memcached set, get and etc.
- swift.common.utils.mime_to_document_iters(input_file, boundary, read_chunk_size=4096)¶
Takes a file-like object containing a multipart MIME document and returns an iterator of (headers, body-file) tuples.
- Parameters:
input_file – file-like object with the MIME doc in it
boundary – MIME boundary, sans dashes (e.g. “divider”, not “–divider”)
read_chunk_size – size of strings read via input_file.read()
- swift.common.utils.mkdirs(path)¶
Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.
- Parameters:
path – path to create
- swift.common.utils.monkey_patch()¶
Apply all swift monkey patching consistently in one place.
- swift.common.utils.multipart_byteranges_to_document_iters(input_file, boundary, read_chunk_size=4096)¶
Takes a file-like object containing a multipart/byteranges MIME document (see RFC 7233, Appendix A) and returns an iterator of (first-byte, last-byte, length, document-headers, body-file) 5-tuples.
- Parameters:
input_file – file-like object with the MIME doc in it
boundary – MIME boundary, sans dashes (e.g. “divider”, not “–divider”)
read_chunk_size – size of strings read via input_file.read()
- swift.common.utils.node_to_string(node_dict, replication=False)¶
Get a string representation of a node’s location.
- Parameters:
node_dict – a dict describing a node
replication – if True then the replication ip address and port are used, otherwise the normal ip address and port are used.
- Returns:
a string of the form <ip address>:<port>/<device>
- swift.common.utils.override_bytes_from_content_type(listing_dict, logger=None)¶
Takes a dict from a container listing and overrides the content_type, bytes fields if swift_bytes is set.
- swift.common.utils.pairs(item_list)¶
Returns an iterator of all pairs of elements from item_list.
- Parameters:
item_list – items (no duplicates allowed)
- swift.common.utils.parse_content_disposition(header)¶
Given the value of a header like: Content-Disposition: form-data; name=”somefile”; filename=”test.html”
Return data like (“form-data”, {“name”: “somefile”, “filename”: “test.html”})
- Parameters:
header – Value of a header (the part after the ‘: ‘).
- Returns:
(value name, dict) of the attribute data parsed (see above).
- swift.common.utils.parse_content_range(content_range)¶
Parse a content-range header into (first_byte, last_byte, total_size).
See RFC 7233 section 4.2 for details on the header format, but it’s basically “Content-Range: bytes ${start}-${end}/${total}”.
- Parameters:
content_range – Content-Range header value to parse, e.g. “bytes 100-1249/49004”
- Returns:
3-tuple (start, end, total)
- Raises:
ValueError – if malformed
- swift.common.utils.parse_content_type(content_type)¶
Parse a content-type and its parameters into values. RFC 2616 sec 14.17 and 3.7 are pertinent.
Examples:
'text/plain; charset=UTF-8' -> ('text/plain', [('charset, 'UTF-8')]) 'text/plain; charset=UTF-8; level=1' -> ('text/plain', [('charset, 'UTF-8'), ('level', '1')])
- Parameters:
content_type – content_type to parse
- Returns:
a tuple containing (content type, list of k, v parameter tuples)
- swift.common.utils.parse_db_filename(filename)¶
Splits a db filename into three parts: the hash, the epoch, and the extension.
>>> parse_db_filename("ab2134.db") ('ab2134', None, '.db') >>> parse_db_filename("ab2134_1234567890.12345.db") ('ab2134', '1234567890.12345', '.db')
- Parameters:
filename – A db file basename or path to a db file.
- Returns:
A tuple of (hash , epoch, extension).
epoch
may be None.- Raises:
ValueError – if
filename
is not a path to a file.
- swift.common.utils.parse_mime_headers(doc_file)¶
Takes a file-like object containing a MIME document and returns a HeaderKeyDict containing the headers. The body of the message is not consumed: the position in doc_file is left at the beginning of the body.
This function was inspired by the Python standard library’s http.client.parse_headers.
- Parameters:
doc_file – binary file-like object containing a MIME document
- Returns:
a swift.common.swob.HeaderKeyDict containing the headers
- swift.common.utils.parse_options(parser=None, once=False, test_config=False, test_args=None)¶
Parse standard swift server/daemon options with optparse.OptionParser.
- Parameters:
parser – OptionParser to use. If not sent one will be created.
once – Boolean indicating the “once” option is available
test_config – Boolean indicating the “test-config” option is available
test_args – Override sys.argv; used in testing
- Returns:
Tuple of (config, options); config is an absolute path to the config file, options is the parser options as a dictionary.
- Raises:
SystemExit – First arg (CONFIG) is required, file must exist
- swift.common.utils.parse_override_options(**kwargs)¶
Figure out which policies, devices, and partitions we should operate on, based on kwargs.
If ‘override_policies’ is already present in kwargs, then return that value. This happens when using multiple worker processes; the parent process supplies override_policies=X to each child process.
Otherwise, in run-once mode, look at the ‘policies’ keyword argument. This is the value of the “–policies” command-line option. In run-forever mode or if no –policies option was provided, an empty list will be returned.
The procedures for devices and partitions are similar.
- Returns:
a named tuple with fields “devices”, “partitions”, and “policies”.
- swift.common.utils.private(func)¶
Decorator to declare which methods are privately accessible as HTTP requests with an
X-Backend-Allow-Private-Methods: True
override- Parameters:
func – function to make private
- swift.common.utils.public(func)¶
Decorator to declare which methods are publicly accessible as HTTP requests
- Parameters:
func – function to make public
- swift.common.utils.punch_hole(fd, offset, length)¶
De-allocate disk space in the middle of a file.
- Parameters:
fd – file descriptor
offset – index of first byte to de-allocate
length – number of bytes to de-allocate
- swift.common.utils.put_recon_cache_entry(cache_entry, key, item)¶
Update a recon cache entry item.
If
item
is an empty dict then any existingkey
incache_entry
will be deleted. Similarly ifitem
is a dict and any of its values are empty dicts then the corresponding key will be deleted from the nested dict incache_entry
.We use nested recon cache entries when the object auditor runs in parallel or else in ‘once’ mode with a specified subset of devices.
- Parameters:
cache_entry – a dict of existing cache entries
key – key for item to update
item – value for item to update
- swift.common.utils.quorum_size(n)¶
quorum size as it applies to services that use ‘replication’ for data integrity (Account/Container services). Object quorum_size is defined on a storage policy basis.
Number of successful backend requests needed for the proxy to consider the client request successful.
- swift.common.utils.ratelimit_sleep(running_time, max_rate, incr_by=1, rate_buffer=5)¶
Will eventlet.sleep() for the appropriate time so that the max_rate is never exceeded. If max_rate is 0, will not ratelimit. The maximum recommended rate should not exceed (1000 * incr_by) a second as eventlet.sleep() does involve some overhead. Returns running_time that should be used for subsequent calls.
- Parameters:
running_time – the running time in milliseconds of the next allowable request. Best to start at zero.
max_rate – The maximum rate per second allowed for the process.
incr_by – How much to increment the counter. Useful if you want to ratelimit 1024 bytes/sec and have differing sizes of requests. Must be > 0 to engage rate-limiting behavior.
rate_buffer – Number of seconds the rate counter can drop and be allowed to catch up (at a faster than listed rate). A larger number will result in larger spikes in rate but better average accuracy. Must be > 0 to engage rate-limiting behavior.
- Returns:
The absolute time for the next interval in milliseconds; note that time could have passed well beyond that point, but the next call will catch that and skip the sleep.
- swift.common.utils.reiterate(iterable)¶
Consume the first truthy item from an iterator, then re-chain it to the rest of the iterator. This is useful when you want to make sure the prologue to downstream generators have been executed before continuing. :param iterable: an iterable object
- swift.common.utils.remove_directory(path)¶
Wrapper for os.rmdir, ENOENT and ENOTEMPTY are ignored
- Parameters:
path – first and only argument passed to os.rmdir
- swift.common.utils.remove_file(path)¶
Quiet wrapper for os.unlink, OSErrors are suppressed
- Parameters:
path – first and only argument passed to os.unlink
- swift.common.utils.renamer(old, new, fsync=True)¶
Attempt to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.
The containing directory of ‘new’ and of all newly created directories are fsync’d by default. This _will_ come at a performance penalty. In cases where these additional fsyncs are not necessary, it is expected that the caller of renamer() turn it off explicitly.
- Parameters:
old – old path to be renamed
new – new path to be renamed to
fsync – fsync on containing directory of new and also all the newly created directories.
- swift.common.utils.replace_partition_in_path(devices, path, part_power)¶
Takes a path and a partition power and returns the same path, but with the correct partition number. Most useful when increasing the partition power.
- Parameters:
devices – directory where devices are mounted (e.g. /srv/node)
path – full path to a object file or hashdir
part_power – partition power to compute correct partition number
- Returns:
Path with re-computed partition power
- swift.common.utils.replication(func)¶
Decorator to declare which methods are accessible for different type of servers:
If option replication_server is None then this decorator doesn’t matter.
If option replication_server is True then ONLY decorated with this decorator methods will be started.
If option replication_server is False then decorated with this decorator methods will NOT be started.
- Parameters:
func – function to mark accessible for replication
- swift.common.utils.round_robin_iter(its)¶
Takes a list of iterators, yield an element from each in a round-robin fashion until all of them are exhausted. :param its: list of iterators
- swift.common.utils.rsync_ip(ip)¶
Transform ip string to an rsync-compatible form
Will return ipv4 addresses unchanged, but will nest ipv6 addresses inside square brackets.
- Parameters:
ip – an ip string (ipv4 or ipv6)
- Returns:
a string ip address
- swift.common.utils.rsync_module_interpolation(template, device)¶
Interpolate devices variables inside a rsync module template
- Parameters:
template – rsync module template as a string
device – a device from a ring
- Returns:
a string with all variables replaced by device attributes
- swift.common.utils.search_tree(root, glob_match, ext='', exts=None, dir_ext=None)¶
Look in root, for any files/dirs matching glob, recursively traversing any found directories looking for files ending with ext
- Parameters:
root – start of search path
glob_match – glob to match in root, matching dirs are traversed with os.walk
ext – only files that end in ext will be returned
exts – a list of file extensions; only files that end in one of these extensions will be returned; if set this list overrides any extension specified using the ‘ext’ param.
dir_ext – if present directories that end with dir_ext will not be traversed and instead will be returned as a matched path
- Returns:
list of full paths to matching files, sorted
- swift.common.utils.select_ip_port(node_dict, use_replication=False)¶
Get the ip address and port that should be used for the given
node_dict
.If
use_replication
is True then the replication ip address and port are returned.If
use_replication
is False (the default) and thenode
dict has an item with keyuse_replication
then that item’s value will determine if the replication ip address and port are returned.If neither
use_replication
nornode_dict['use_replication']
indicate otherwise then the normal ip address and port are returned.- Parameters:
node_dict – a dict describing a node
use_replication – if True then the replication ip address and port are returned.
- Returns:
a tuple of (ip address, port)
- swift.common.utils.set_swift_dir(swift_dir)¶
Sets the directory from which swift config files will be read. If the given directory differs from that already set then the swift.conf file in the new directory will be validated and storage policies will be reloaded from the new swift.conf file.
- Parameters:
swift_dir – non-default directory to read swift.conf from
- swift.common.utils.storage_directory(datadir, partition, name_hash)¶
Get the storage directory
- Parameters:
datadir – Base data directory
partition – Partition
name_hash – Account, container or object name hash
- Returns:
Storage directory
- swift.common.utils.streq_const_time(s1, s2)¶
Constant-time string comparison.
- Params s1:
the first string
- Params s2:
the second string
- Returns:
True if the strings are equal.
This function takes two strings and compares them. It is intended to be used when doing a comparison for authentication purposes to help guard against timing attacks.
- swift.common.utils.strict_b64decode(value, allow_line_breaks=False)¶
Validate and decode Base64-encoded data.
The stdlib base64 module silently discards bad characters, but we often want to treat them as an error.
- Parameters:
value – some base64-encoded data
allow_line_breaks – if True, ignore carriage returns and newlines
- Returns:
the decoded data
- Raises:
ValueError – if
value
is not a string, contains invalid characters, or has insufficient padding
- swift.common.utils.systemd_notify(logger=None, msg=b'READY=1')¶
Send systemd-compatible notifications.
Notify the service manager that started this process, if it has set the NOTIFY_SOCKET environment variable. For example, systemd will set this when the unit has
Type=notify
. More information can be found in systemd documentation: https://www.freedesktop.org/software/systemd/man/sd_notify.htmlCommon messages include:
READY=1 RELOADING=1 STOPPING=1 STATUS=<some string>
- Parameters:
logger – a logger object
msg – the message to send
- swift.common.utils.timing_stats(**dec_kwargs)¶
Returns a decorator that logs timing events or errors for public methods in swift’s wsgi server controllers, based on response code.
- swift.common.utils.unlink_older_than(path, mtime)¶
Remove any file in a given path that was last modified before mtime.
- Parameters:
path – path to remove file from
mtime – timestamp of oldest file to keep
- swift.common.utils.unlink_paths_older_than(filepaths, mtime)¶
Remove any files from the given list that were last modified before mtime.
- Parameters:
filepaths – a list of strings, the full paths of files to check
mtime – timestamp of oldest file to keep
- swift.common.utils.validate_device_partition(device, partition)¶
Validate that a device and a partition are valid and won’t lead to directory traversal when used.
- Parameters:
device – device to validate
partition – partition to validate
- Raises:
ValueError – if given an invalid device or partition
- swift.common.utils.validate_sync_to(value, allowed_sync_hosts, realms_conf)¶
Validates an X-Container-Sync-To header value, returning the validated endpoint, realm, and realm_key, or an error string.
- Parameters:
value – The X-Container-Sync-To header value to validate.
allowed_sync_hosts – A list of allowed hosts in endpoints, if realms_conf does not apply.
realms_conf – An instance of swift.common.container_sync_realms.ContainerSyncRealms to validate against.
- Returns:
A tuple of (error_string, validated_endpoint, realm, realm_key). The error_string will None if the rest of the values have been validated. The validated_endpoint will be the validated endpoint to sync to. The realm and realm_key will be set if validation was done through realms_conf.
- swift.common.utils.write_file(path, contents)¶
Write contents to file at path
- Parameters:
path – any path, subdirs will be created as needed
contents – data to write to file, will be converted to string
- swift.common.utils.write_pickle(obj, dest, tmp=None, pickle_protocol=0)¶
Ensure that a pickle file gets written to disk. The file is first written to a tmp location, ensure it is synced to disk, then perform a move to its final location
- Parameters:
obj – python object to be pickled
dest – path of final destination file
tmp – path to tmp to use, defaults to None
pickle_protocol – protocol to pickle the obj with, defaults to 0
WSGI¶
WSGI tools for use with swift.
- class swift.common.wsgi.ConfigDirLoader(conf_dir)¶
Bases:
NamedConfigLoader
Read configuration from multiple files under the given path.
- exception swift.common.wsgi.ConfigFileError¶
Bases:
Exception
- exception swift.common.wsgi.ConfigFilePortError¶
Bases:
ConfigFileError
- class swift.common.wsgi.ConfigString(config_string)¶
Bases:
NamedConfigLoader
Wrap a raw config string up for paste.deploy.
If you give one of these to our loadcontext (e.g. give it to our appconfig) we’ll intercept it and get it routed to the right loader.
- class swift.common.wsgi.NamedConfigLoader(filename)¶
Bases:
ConfigLoader
Patch paste.deploy’s ConfigLoader so each context object will know what config section it came from.
- class swift.common.wsgi.PipelineWrapper(context)¶
Bases:
object
This class provides a number of utility methods for modifying the composition of a wsgi pipeline.
- create_filter(entry_point_name)¶
Creates a context for a filter that can subsequently be added to a pipeline context.
- Parameters:
entry_point_name – entry point of the middleware (Swift only)
- Returns:
a filter context
- index(entry_point_name)¶
Returns the first index of the given entry point name in the pipeline.
Raises ValueError if the given module is not in the pipeline.
- insert_filter(ctx, index=0)¶
Inserts a filter module into the pipeline context.
- Parameters:
ctx – the context to be inserted
index – (optional) index at which filter should be inserted in the list of pipeline filters. Default is 0, which means the start of the pipeline.
- startswith(entry_point_name)¶
Tests if the pipeline starts with the given entry point name.
- Parameters:
entry_point_name – entry point of middleware or app (Swift only)
- Returns:
True if entry_point_name is first in pipeline, False otherwise
- class swift.common.wsgi.RestrictedGreenPool(size=1024)¶
Bases:
GreenPool
Works the same as GreenPool, but if the size is specified as one, then the spawn_n() method will invoke waitall() before returning to prevent the caller from doing any other work (like calling accept()).
- class swift.common.wsgi.ServersPerPortStrategy(conf, logger, servers_per_port)¶
Bases:
StrategyBase
WSGI server management strategy object for an object-server with one listen port per unique local port in the storage policy rings. The servers_per_port integer config setting determines how many workers are run per port.
Tracking data is a map like
port -> [(pid, socket), ...]
.Used in
run_wsgi()
.- Parameters:
conf (dict) – Server configuration dictionary.
logger – The server’s
LogAdaptor
object.servers_per_port (int) – The number of workers to run per port.
- iter_sockets()¶
Yields all known listen sockets.
- log_sock_exit(sock, data)¶
Log a server’s exit.
- loop_timeout()¶
Return timeout before checking for reloaded rings.
- Returns:
The time to wait for a child to exit before checking for reloaded rings (new ports).
- new_worker_socks()¶
Yield a sequence of (socket, (port, server_idx)) tuples for each server which should be forked-off and started.
Any sockets for “orphaned” ports no longer in any ring will be closed (causing their associated workers to gracefully exit) after all new sockets have been yielded.
The server_idx item for each socket will passed into the
log_sock_exit()
andregister_worker_start()
methods.
- no_fork_sock()¶
This strategy does not support running in the foreground.
- register_worker_exit(pid)¶
Called when a worker has exited.
- Parameters:
pid (int) – The PID of the worker that exited.
- register_worker_start(sock, data, pid)¶
Called when a new worker is started.
- Parameters:
sock (socket) – The listen socket for the worker just started.
data (tuple) – The socket’s (port, server_idx) as yielded by
new_worker_socks()
.pid (int) – The new worker process’ PID
- class swift.common.wsgi.StrategyBase(conf, logger)¶
Bases:
object
Some operations common to all strategy classes.
- post_fork_hook()¶
Called in each forked-off child process, prior to starting the actual wsgi server, to perform any initialization such as drop privileges.
- set_close_on_exec_on_listen_sockets()¶
Set the close-on-exec flag on any listen sockets.
- shutdown_sockets()¶
Shutdown any listen sockets.
- signal_ready()¶
Signal that the server is up and accepting connections.
- class swift.common.wsgi.WSGIContext(wsgi_app)¶
Bases:
object
This class provides a means to provide context (scope) for a middleware filter to have access to the wsgi start_response results like the request status and headers.
- class swift.common.wsgi.WorkersStrategy(conf, logger)¶
Bases:
StrategyBase
WSGI server management strategy object for a single bind port and listen socket shared by a configured number of forked-off workers.
Tracking data is a map of
pid -> socket
.Used in
run_wsgi()
.- Parameters:
conf (dict) – Server configuration dictionary.
logger – The server’s
LogAdaptor
object.
- iter_sockets()¶
Yields all known listen sockets.
- log_sock_exit(sock, _unused)¶
Log a server’s exit.
- Parameters:
sock (socket) – The listen socket for the worker just started.
_unused – The socket’s opaque_data yielded by
new_worker_socks()
.
- loop_timeout()¶
We want to keep from busy-waiting, but we also need a non-None value so the main loop gets a chance to tell whether it should keep running or not (e.g. SIGHUP received).
So we return 0.5.
- new_worker_socks()¶
Yield a sequence of (socket, opqaue_data) tuples for each server which should be forked-off and started.
The opaque_data item for each socket will passed into the
log_sock_exit()
andregister_worker_start()
methods where it will be ignored.
- no_fork_sock()¶
Return a server listen socket if the server should run in the foreground (no fork).
- register_worker_exit(pid)¶
Called when a worker has exited.
NOTE: a re-exec’ed server can reap the dead worker PIDs from the old server process that is being replaced as part of a service reload (SIGUSR1). So we need to be robust to getting some unknown PID here.
- Parameters:
pid (int) – The PID of the worker that exited.
- register_worker_start(sock, _unused, pid)¶
Called when a new worker is started.
- Parameters:
sock (socket) – The listen socket for the worker just started.
_unused – The socket’s opaque_data yielded by new_worker_socks().
pid (int) – The new worker process’ PID
- swift.common.wsgi.get_socket(conf)¶
Bind socket to bind ip:port in conf
- Parameters:
conf – Configuration dict to read settings from
- Returns:
a socket object as returned from socket.listen or ssl.wrap_socket if conf specifies cert_file
- swift.common.wsgi.init_request_processor(conf_path, app_section, *args, **kwargs)¶
Loads common settings from conf Sets the logger Loads the request processor
- Parameters:
conf_path – Path to paste.deploy style configuration file/directory
app_section – App name from conf file to load config from
- Returns:
the loaded application entry point
- Raises:
ConfigFileError – Exception is raised for config file error
- swift.common.wsgi.load_app_config(conf_file)¶
Read the app config section from a config file.
- Parameters:
conf_file – path to a config file
- Returns:
a dict
- swift.common.wsgi.loadapp(conf_file, global_conf=None, allow_modify_pipeline=True)¶
Loads a context from a config file, and if the context is a pipeline then presents the app with the opportunity to modify the pipeline.
- Parameters:
conf_file – path to a config file
global_conf – a dict of options to update the loaded config. Options in
global_conf
will override those inconf_file
except where theconf_file
option is preceded byset
.allow_modify_pipeline – if True, and the context is a pipeline, and the loaded app has a
modify_wsgi_pipeline
property, then that property will be called before the pipeline is loaded.
- Returns:
the loaded app
- swift.common.wsgi.make_env(env, method=None, path=None, agent='Swift', query_string=None, swift_source=None)¶
Returns a new fresh WSGI environment.
- Parameters:
env – The WSGI environment to base the new environment on.
method – The new REQUEST_METHOD or None to use the original.
path – The new path_info or none to use the original. path should NOT be quoted. When building a url, a Webob Request (in accordance with wsgi spec) will quote env[‘PATH_INFO’]. url += quote(environ[‘PATH_INFO’])
query_string – The new query_string or none to use the original. When building a url, a Webob Request will append the query string directly to the url. url += ‘?’ + env[‘QUERY_STRING’]
agent – The HTTP user agent to use; default ‘Swift’. You can put %(orig)s in the agent to have it replaced with the original env’s HTTP_USER_AGENT, such as ‘%(orig)s StaticWeb’. You also set agent to None to use the original env’s HTTP_USER_AGENT or ‘’ to have no HTTP_USER_AGENT.
swift_source – Used to mark the request as originating out of middleware. Will be logged in proxy logs.
- Returns:
Fresh WSGI environment.
- swift.common.wsgi.make_pre_authed_env(env, method=None, path=None, agent='Swift', query_string=None, swift_source=None)¶
Same as
make_env()
but with preauthorization.
- swift.common.wsgi.make_pre_authed_request(env, method=None, path=None, body=None, headers=None, agent='Swift', swift_source=None)¶
Same as
make_subrequest()
but with preauthorization.
- swift.common.wsgi.make_subrequest(env, method=None, path=None, body=None, headers=None, agent='Swift', swift_source=None, make_env=<function make_env>)¶
Makes a new swob.Request based on the current env but with the parameters specified.
- Parameters:
env – The WSGI environment to base the new request on.
method – HTTP method of new request; default is from the original env.
path – HTTP path of new request; default is from the original env. path should be compatible with what you would send to Request.blank. path should be quoted and it can include a query string. for example: ‘/a%20space?unicode_str%E8%AA%9E=y%20es’
body – HTTP body of new request; empty by default.
headers – Extra HTTP headers of new request; None by default.
agent – The HTTP user agent to use; default ‘Swift’. You can put %(orig)s in the agent to have it replaced with the original env’s HTTP_USER_AGENT, such as ‘%(orig)s StaticWeb’. You also set agent to None to use the original env’s HTTP_USER_AGENT or ‘’ to have no HTTP_USER_AGENT.
swift_source – Used to mark the request as originating out of middleware. Will be logged in proxy logs.
make_env – make_subrequest calls this make_env to help build the swob.Request.
- Returns:
Fresh swob.Request object.
- swift.common.wsgi.run_wsgi(conf_path, app_section, *args, **kwargs)¶
Runs the server according to some strategy. The default strategy runs a specified number of workers in pre-fork model. The object-server (only) may use a servers-per-port strategy if its config has a servers_per_port setting with a value greater than zero.
- Parameters:
conf_path – Path to paste.deploy style configuration file/directory
app_section – App name from conf file to load config from
allow_modify_pipeline – Boolean for whether the server should have an opportunity to change its own pipeline. Defaults to True
test_config – if False (the default) then load and validate the config and if successful then continue to run the server; if True then load and validate the config but do not run the server.
- Returns:
0 if successful, nonzero otherwise
- swift.common.wsgi.wrap_conf_type(f)¶
Wrap a function whos first argument is a paste.deploy style config uri, such that you can pass it an un-adorned raw filesystem path (or config string) and the config directive (either config:, config_dir:, or config_str:) will be added automatically based on the type of entity (either a file or directory, or if no such entity on the file system - just a string) before passing it through to the paste.deploy function.
Storage Policy¶
- class swift.common.storage_policy.BaseStoragePolicy(idx, name='', is_default=False, is_deprecated=False, object_ring=None, aliases='', diskfile_module='egg:swift#replication.fs')¶
Bases:
object
Represents a storage policy. Not meant to be instantiated directly; implement a derived subclasses (e.g. StoragePolicy, ECStoragePolicy, etc) or use
reload_storage_policies()
to load POLICIES fromswift.conf
.The object_ring property is lazy loaded once the service’s
swift_dir
is known viaget_object_ring()
, but it may be over-ridden via object_ring kwarg at create time for testing or actively loaded withload_ring()
.- add_name(name)¶
Adds an alias name to the storage policy. Shouldn’t be called directly from the storage policy but instead through the storage policy collection class, so lookups by name resolve correctly.
- Parameters:
name – a new alias for the storage policy
- change_primary_name(name)¶
Changes the primary/default name of the policy to a specified name.
- Parameters:
name – a string name to replace the current primary name.
- get_diskfile_manager(*args, **kwargs)¶
Return an instance of the diskfile manager class configured for this storage policy.
- Parameters:
args – positional args to pass to the diskfile manager constructor.
kwargs – keyword args to pass to the diskfile manager constructor.
- Returns:
A disk file manager instance.
- get_info(config=False)¶
Return the info dict and conf file options for this policy.
- Parameters:
config – boolean, if True all config options are returned
- load_ring(swift_dir, reload_time=None)¶
Load the ring for this policy immediately.
- Parameters:
swift_dir – path to rings
reload_time – time interval in seconds to check for a ring change
- property quorum¶
Number of successful backend requests needed for the proxy to consider the client request successful.
- classmethod register(policy_type)¶
Decorator for Storage Policy implementations to register their StoragePolicy class. This will also set the policy_type attribute on the registered implementation.
- remove_name(name)¶
Removes an alias name from the storage policy. Shouldn’t be called directly from the storage policy but instead through the storage policy collection class, so lookups by name resolve correctly. If the name removed is the primary name then the next available alias will be adopted as the new primary name.
- Parameters:
name – a name assigned to the storage policy
- validate_ring_data(ring_data)¶
Validation hook used when loading the ring; currently only used for EC
- class swift.common.storage_policy.ECStoragePolicy(idx, name='', aliases='', is_default=False, is_deprecated=False, object_ring=None, diskfile_module='egg:swift#erasure_coding.fs', ec_segment_size=1048576, ec_type=None, ec_ndata=None, ec_nparity=None, ec_duplication_factor=1)¶
Bases:
BaseStoragePolicy
Represents a storage policy of type ‘erasure_coding’.
Not meant to be instantiated directly; use
reload_storage_policies()
to load POLICIES fromswift.conf
.- property ec_scheme_description¶
This short hand form of the important parts of the ec schema is stored in Object System Metadata on the EC Fragment Archives for debugging.
- property fragment_size¶
Maximum length of a fragment, including header.
NB: a fragment archive is a sequence of 0 or more max-length fragments followed by one possibly-shorter fragment.
- get_backend_index(node_index)¶
Backend index for PyECLib
- Parameters:
node_index – integer of node index
- Returns:
integer of actual fragment index. if param is not an integer, return None instead
- get_info(config=False)¶
Return the info dict and conf file options for this policy.
- Parameters:
config – boolean, if True all config options are returned
- property quorum¶
Number of successful backend requests needed for the proxy to consider the client PUT request successful.
The quorum size for EC policies defines the minimum number of data + parity elements required to be able to guarantee the desired fault tolerance, which is the number of data elements supplemented by the minimum number of parity elements required by the chosen erasure coding scheme.
For example, for Reed-Solomon, the minimum number parity elements required is 1, and thus the quorum_size requirement is ec_ndata + 1.
Given the number of parity elements required is not the same for every erasure coding scheme, consult PyECLib for min_parity_fragments_needed()
- validate_ring_data(ring_data)¶
EC specific validation
Replica count check - we need _at_least_ (#data + #parity) replicas configured. Also if the replica count is larger than exactly that number there’s a non-zero risk of error for code that is considering the number of nodes in the primary list from the ring.
- exception swift.common.storage_policy.PolicyError(msg, index=None)¶
Bases:
ValueError
- class swift.common.storage_policy.StoragePolicy(idx, name='', is_default=False, is_deprecated=False, object_ring=None, aliases='', diskfile_module='egg:swift#replication.fs')¶
Bases:
BaseStoragePolicy
Represents a storage policy of type ‘replication’. Default storage policy class unless otherwise overridden from swift.conf.
Not meant to be instantiated directly; use
reload_storage_policies()
to load POLICIES fromswift.conf
.- property quorum¶
- Quorum concept in the replication case:
floor(number of replica / 2) + 1
- class swift.common.storage_policy.StoragePolicyCollection(pols)¶
Bases:
object
This class represents the collection of valid storage policies for the cluster and is instantiated as
StoragePolicy
objects are added to the collection whenswift.conf
is parsed byparse_storage_policies()
.When a StoragePolicyCollection is created, the following validation is enforced:
If a policy with index 0 is not declared and no other policies defined, Swift will create one
The policy index must be a non-negative integer
If no policy is declared as the default and no other policies are defined, the policy with index 0 is set as the default
Policy indexes must be unique
Policy names are required
Policy names are case insensitive
Policy names must contain only letters, digits or a dash
Policy names must be unique
The policy name ‘Policy-0’ can only be used for the policy with index 0
If any policies are defined, exactly one policy must be declared default
Deprecated policies can not be declared the default
- add_policy_alias(policy_index, *aliases)¶
Adds a new name or names to a policy
- Parameters:
policy_index – index of a policy in this policy collection.
aliases – arbitrary number of string policy names to add.
- change_policy_primary_name(policy_index, new_name)¶
Changes the primary or default name of a policy. The new primary name can be an alias that already belongs to the policy or a completely new name.
- Parameters:
policy_index – index of a policy in this policy collection.
new_name – a string name to set as the new default name.
- get_by_index(index)¶
Find a storage policy by its index.
An index of None will be treated as 0.
- Parameters:
index – numeric index of the storage policy
- Returns:
storage policy, or None if no such policy
- get_by_name(name)¶
Find a storage policy by its name.
- Parameters:
name – name of the policy
- Returns:
storage policy, or None
- get_object_ring(policy_idx, swift_dir)¶
Get the ring object to use to handle a request based on its policy.
An index of None will be treated as 0.
- Parameters:
policy_idx – policy index as defined in swift.conf
swift_dir – swift_dir used by the caller
- Returns:
appropriate ring object
- get_policy_info()¶
Build info about policies for the /info endpoint
- Returns:
list of dicts containing relevant policy information
- remove_policy_alias(*aliases)¶
Removes a name or names from a policy. If the name removed is the primary name then the next available alias will be adopted as the new primary name.
- Parameters:
aliases – arbitrary number of existing policy names to remove.
- class swift.common.storage_policy.StoragePolicySingleton¶
Bases:
object
An instance of this class is the primary interface to storage policies exposed as a module level global named
POLICIES
. This global reference wraps_POLICIES
which is normally instantiated by parsingswift.conf
and will result in an instance ofStoragePolicyCollection
.You should never patch this instance directly, instead patch the module level
_POLICIES
instance so that swift code which importedPOLICIES
directly will reference the patchedStoragePolicyCollection
.
- swift.common.storage_policy.get_policy_string(base, policy_or_index)¶
Helper function to construct a string from a base and the policy. Used to encode the policy index into either a file name or a directory name by various modules.
- Parameters:
base – the base string
policy_or_index – StoragePolicy instance, or an index (string or int), if None the legacy storage Policy-0 is assumed.
- Returns:
base name with policy index added
- Raises:
PolicyError – if no policy exists with the given policy_index
- swift.common.storage_policy.parse_storage_policies(conf)¶
Parse storage policies in
swift.conf
- note that validation is done when theStoragePolicyCollection
is instantiated.- Parameters:
conf – ConfigParser parser object for swift.conf
- swift.common.storage_policy.reload_storage_policies()¶
Reload POLICIES from
swift.conf
.
- swift.common.storage_policy.split_policy_string(policy_string)¶
Helper function to convert a string representing a base and a policy. Used to decode the policy from either a file name or a directory name by various modules.
- Parameters:
policy_string – base name with policy index added
- Raises:
PolicyError – if given index does not map to a valid policy
- Returns:
a tuple, in the form (base, policy) where base is the base string and policy is the StoragePolicy instance for the index encoded in the policy_string.