Object

Object Server

Object Server for Swift

class swift.obj.server.DiskFile(path, device, partition, account, container, obj, logger, keep_data_fp=False, disk_chunk_size=65536, iter_hook=None)

Bases: object

Manage object files on disk.

Parameters:

• path – path to devices on the node
• device – device name
• partition – partition on the device the object lives in
• account – account name for the object
• container – container name for the object
• obj – object name for the object
• keep_data_fp – if True, don’t close the fp, otherwise close it
• disk_chunk_size – size of chunks on file reads
• iter_hook – called when __iter__ returns a chunk

Raises DiskFileCollision:  

on md5 collision

app_iter_range(start, stop)

Returns an iterator over the data file for range (start, stop)

app_iter_ranges(ranges, content_type, boundary, size)

Returns an iterator over the data file for a set of ranges

close(verify_file=True)

Close the file. Will handle quarantining file if necessary.

Parameters:verify_file – Defaults to True. If false, will not check file to see if it needs quarantining.

drop_cache(fd, offset, length)

Method for no-oping buffer cache drop method.

get_data_file_size()

Returns the os.path.getsize for the file. Raises an exception if this file does not match the Content-Length stored in the metadata. Or if self.data_file does not exist.

Returns:

file size as an int

Raises:

• DiskFileError – on file size mismatch.
• DiskFileNotExist – on file not existing (including deleted)

is_deleted()

Check if the file is deleted.

Returns:True if the file doesn’t exist or has been flagged as deleted.

is_expired()

Check if the file is expired.

Returns:True if the file has an X-Delete-At in the past

mkstemp(*args, **kwds)

Contextmanager to make a temporary file.

Parameters:size – optional initial size of file to allocate on disk Raises DiskFileNoSpace:  if a size is specified and fallocate fails

put(fd, fsize, metadata, extension='.data')

Finalize writing the file on disk, and renames it from the temp file to the real location. This should be called after the data has been written to the temp file.

Parameters:

• fd – file descriptor of the temp file
• fsize – final on-disk size of the created file
• metadata – dictionary of metadata to be written
• extension – extension to be used when making the file

put_metadata(metadata, tombstone=False)

Short hand for putting metadata to .meta and .ts files.

Parameters:

• metadata – dictionary of metadata to be written
• tombstone – whether or not we are writing a tombstone

quarantine()

In the case that a file is corrupted, move it to a quarantined area to allow replication to fix it.

Returns:if quarantine is successful, path to quarantined directory otherwise None

unlinkold(timestamp)

Remove any older versions of the object file. Any file that has an older timestamp than timestamp will be deleted.

Parameters:timestamp – timestamp to compare with each file

class swift.obj.server.ObjectController(conf)

Bases: object

Implements the WSGI application for the Swift Object Server.

DELETE(*a, **kw)

Handle HTTP DELETE requests for the Swift Object Server.

GET(*a, **kw)

Handle HTTP GET requests for the Swift Object Server.

HEAD(*a, **kw)

Handle HTTP HEAD requests for the Swift Object Server.

POST(*a, **kw)

Handle HTTP POST requests for the Swift Object Server.

PUT(*a, **kw)

Handle HTTP PUT requests for the Swift Object Server.

REPLICATE(*a, **kw)

Handle REPLICATE requests for the Swift Object Server. This is used by the object replicator to get hashes for directories.

async_update(op, account, container, obj, host, partition, contdevice, headers_out, objdevice)

Sends or saves an async update.

Parameters:

• op – operation performed (ex: ‘PUT’, or ‘DELETE’)
• account – account name for the object
• container – container name for the object
• obj – object name
• host – host that the container is on
• partition – partition that the container is on
• contdevice – device name that the container is on
• headers_out – dictionary of headers to send in the container request
• objdevice – device name that the object is in

container_update(op, account, container, obj, request, headers_out, objdevice)

Update the container when objects are updated.

Parameters:

• op – operation performed (ex: ‘PUT’, or ‘DELETE’)
• account – account name for the object
• container – container name for the object
• obj – object name
• request – the original request object driving the update
• headers_out – dictionary of headers to send in the container request(s)
• objdevice – device name that the object is in

delete_at_update(op, delete_at, account, container, obj, request, objdevice)

Update the expiring objects container when objects are updated.

Parameters:

• op – operation performed (ex: ‘PUT’, or ‘DELETE’)
• account – account name for the object
• container – container name for the object
• obj – object name
• request – the original request driving the update
• objdevice – device name that the object is in

swift.obj.server.app_factory(global_conf, **local_conf)

paste.deploy app factory for creating WSGI object server apps

swift.obj.server.read_metadata(fd)

Helper function to read the pickled metadata from an object file.

Parameters:fd – file descriptor to load the metadata from Returns:dictionary of metadata

swift.obj.server.write_metadata(fd, metadata)

Helper function to write pickled metadata for an object file.

Parameters:

• fd – file descriptor to write the metadata
• metadata – metadata to write

Object Replicator

class swift.obj.replicator.ObjectReplicator(conf)

Bases: swift.common.daemon.Daemon

Replicate objects.

Encapsulates most logic and data needed by the object replication process. Each call to .replicate() performs one replication pass. It’s up to the caller to do this in a loop.

check_ring()

Check to see if the ring has been updated

Returns:boolean indicating whether or not the ring has changed

collect_jobs()

Returns a sorted list of jobs (dictionaries) that specify the partitions, nodes, etc to be rsynced.

detect_lockups()

In testing, the pool.waitall() call very occasionally failed to return. This is an attempt to make sure the replicator finishes its replication pass in some eventuality.

heartbeat()

Loop that runs in the background during replication. It periodically logs progress.

kill_coros()

Utility function that kills all coroutines currently running.

replicate(override_devices=None, override_partitions=None)

Run a replication pass

rsync(node, job, suffixes)

Synchronize local suffix directories from a partition with a remote node.

Parameters:

• node – the “dev” entry for the remote node to sync with
• job – information about the partition being synced
• suffixes – a list of suffixes which need to be pushed

Returns:

boolean indicating success or failure

run_forever(*args, **kwargs)

run_once(*args, **kwargs)

stats_line()

Logs various stats for the currently running replication pass.

update(job)

High-level method that replicates a single partition.

Parameters:job – a dict containing info about the partition to be replicated

update_deleted(job)

High-level method that replicates a single partition that doesn’t belong on this node.

Parameters:job – a dict containing info about the partition to be replicated

Object Updater

class swift.obj.updater.ObjectUpdater(conf)

Bases: swift.common.daemon.Daemon

Update object information in container listings.

get_container_ring()

Get the container ring. Load it, if it hasn’t been yet.

object_sweep(device)

If there are async pendings on the device, walk each one and update.

Parameters:device – path to device

object_update(node, part, op, obj, headers)

Perform the object update to the container

Parameters:

• node – node dictionary from the container ring
• part – partition that holds the container
• op – operation performed (ex: ‘POST’ or ‘DELETE’)
• obj – object name being updated
• headers – headers to send with the update

process_object_update(update_path, device)

Process the object information to be updated and update.

Parameters:

• update_path – path to pickled object update file
• device – path to device

run_forever(*args, **kwargs)

Run the updater continuously.

run_once(*args, **kwargs)

Run the updater once

swift.obj.updater.random() → x in the interval [0, 1).

Object Auditor

class swift.obj.auditor.AuditorWorker(conf, logger, zero_byte_only_at_fps=0)

Bases: object

Walk through file system to audit object

audit_all_objects(mode='once')

object_audit(path, device, partition)

Audits the given object path.

Parameters:

• path – a path to an object
• device – the device the path is on
• partition – the partition the path is on

class swift.obj.auditor.ObjectAuditor(conf, **options)

Bases: swift.common.daemon.Daemon

Audit objects.

run_forever(*args, **kwargs)

Run the object audit until stopped.

run_once(*args, **kwargs)

Run the object audit once.