norfs package

Submodules

norfs.client module

class norfs.client.CopyClient(copier: norfs.copy.base.Copier)[source]

Bases: object

norfs.client.CopyClient provides a unified simple copy API for any norfs.filesystem.File or norfs.filesystem.Directory from any file system. It is usually accessed by using norfs.helpers.get_copy_client():

import norfs.helpers

local = norfs.helpers.local() cp_local_only = norfs.helpers.get_copy_client(local)

cp_local_only.copy(local.file('source_file.txt'), local.file('target_file.txt'))


memory = norfs.helpers.memory()

import boto3 s3 = norfs.helpers.s3(s3_client=boto3.client('s3'))

cp_for_all = norfs.helpers.get_copy_client(local, s3, memory)

cp_for_all.copy(s3.file('myBucket/source_file.txt'), local.file('target_file.txt'))

norfs.helpers.get_copy_client() returns a norfs.client.CopyClient instance configured with copy strategies for each of the file system clients passed.

A norfs.copy.base.Copier can have copy policies set for a pair of source and destination file systems to implement a better strategy of copying between them than read source and write destination. norfs.helpers.get_copy_client() helps you by setting these for you.

A norfs.client.CopyClient exposes the following interface:

property copier

The norfs.copy.base.Copier instance managed by the client.

copy(src: norfs.filesystem.BaseFileSystemObject, dst: norfs.filesystem.BaseFileSystemObject) → None[source]

Copies src to dst, no mater the file systems they are on. src and dst can by both norfs.filesystem.File or norfs.filesystem.Directory. The only operation not supported is copying from a norfs.filesystem.Directory into a norfs.filesystem.File as it does not make sense.

If source is a norfs.filesystem.Directory and destination is a norfs.filesystem.File it raises a TypeError.

On copy failure it raises a norfs.fs.base.FileSystemOperationError.

class norfs.client.FileSystemClient(fs: norfs.fs.base.BaseFileSystem)[source]

Bases: object

norfs.client.FileSystemClient provides a way to access the file system objects of a given file system. It is a handy class that provides easy access to :class:norfs.filesystem.File and norfs.filesystem.Directory instances. It is usually obtained using norfs.helpers:

import norfs.helpers

local_fs_client = norfs.helpers.local()

memory_fs_client = norfs.helpers.memory()

import boto3
s3_fs_client = norfs.helpers.s3(s3_client=boto3.client('s3'))

A norfs.client.FileSystemClient exposes the following interface:

dir(path: str)norfs.filesystem.Directory[source]

Returns a norfs.filesystem.Directory instance for the given path in the managed file system.

file(path: str)norfs.filesystem.File[source]

Returns a norfs.filesystem.File instance for the given path in the managed file system.

property fs

The norfs.filesystem.BaseFileSystemObject the client is managing.

norfs.filesystem module

norfs.filesystem.BaseFileSystemObject represents any object in the filesystem. It is the most abstract representation.

A norfs.filesystem.BaseFileSystemObject exposes the following interface:

class norfs.filesystem.BaseFileSystemObject(filesystem: norfs.fs.base.BaseFileSystem, path_str: Optional[str], *, _path: Optional[norfs.fs.base.Path] = None)[source]

Bases: object

as_dir()norfs.filesystem.Directory[source]

Returns itself as a Directory instance or raises a NotADirectoryError.

as_file()norfs.filesystem.File[source]

Returns itself as a norfs.filesystem.File instance or raises a norfs.fs.base.NotAFileError.

copy_object()norfs.copy.base.CopyFileSystemObject[source]
copy_to(dst: norfs.filesystem.BaseFileSystemObject, strategy: norfs.copy.base.CopyStrategy) → None[source]
exists() → bool[source]

Returns whether self exists in the file system.

is_dir() → bool[source]

Returns whether self is a Directory.

is_file() → bool[source]

Returns whether self is a File.

property name

The name of self.

parent()norfs.filesystem.Directory[source]

Return parent norfs.filesystem.Directory of self.

property path

The full, absolute, path of self in the file system.

remove() → None[source]

Tries to remove self from the file system. On failure it raises a norfs.fs.base.FileSystemOperationError

property uri

The URI that points to self in the file system.

class norfs.filesystem.Directory(filesystem: norfs.fs.base.BaseFileSystem, path_str: Optional[str], *, _path: Optional[norfs.fs.base.Path] = None)[source]

Bases: norfs.filesystem.BaseFileSystemObject

as_dir()norfs.filesystem.Directory[source]

Returns itself as a norfs.filesystem.Directory instance or raises a NotADirectoryError.

copy_object()norfs.copy.base.CopyFileSystemObject[source]
file(path: str)norfs.filesystem.File[source]

Returns a norfs.filesystem.File with its path as being the given path relative to the current directory.

is_dir() → bool[source]

Returns whether self is a norfs.filesystem.Directory.

list() → Iterable[norfs.filesystem.BaseFileSystemObject][source]

Returns the contents of the norfs.filesystem.Directory in the file system as a list of norfs.filesystem.BaseFileSystemObject s.

If the norfs.filesystem.Directory does not exist the list will be empty.

remove() → None[source]

Tries to remove self from the file system.

On failure it raises a norfs.fs.base.FileSystemOperationError

subdir(path: str)norfs.filesystem.Directory[source]

Returns a norfs.filesystem.Directory with its path as being the given path relative to the current directory.

class norfs.filesystem.File(filesystem: norfs.fs.base.BaseFileSystem, path_str: Optional[str], *, _path: Optional[norfs.fs.base.Path] = None)[source]

Bases: norfs.filesystem.BaseFileSystemObject

as_file()norfs.filesystem.File[source]

Returns itself as a norfs.filesystem.File instance or raises a norfs.fs.base.NotAFileError.

copy_object()norfs.copy.base.CopyFileSystemObject[source]
is_file() → bool[source]

Returns whether self is a norfs.filesystem.File.

read() → bytes[source]

Returns the contents of the file.

If it fails to read the file a norfs.fs.base.FileSystemOperationError will be raised.

remove() → None[source]

Tries to remove self from the file system.

On failure it raises a norfs.fs.base.FileSystemOperationError

set_perms(policies: List[norfs.permissions.Policy]) → None[source]

Set the access policies for the file.

The actual meaning of the scopes and permissions depends on the underlying norfs.fs.base.BaseFileSystem implementation.

set_props(*, content_type: Optional[str] = None, tags: Optional[Dict[str, str]] = None, metadata: Optional[Dict[str, str]] = None) → None[source]

Set the properties for the file.

The actual setting of the metadata depends on the underlying norfs.fs.base.BaseFileSystem implementation.

write(content: bytes) → None[source]

Sets the contents of the file. If the parent directory does not exist it is created.

If it fails to write the file a norfs.fs.base.FileSystemOperationError will be raised.

norfs.helpers module

The norfs.helpers offers functions that serve as shortcuts for common operations with the library.

The norfs.helpers contains the following functions:

norfs.helpers.get_copy_client(*args: norfs.client.FileSystemClient)norfs.client.CopyClient[source]

Helper function to get a norfs.copy.base.CopyClient instance configured with copy strategies for the given file systems. This function only sets the built-in norfs.copy.base.CopyStrategy s for the built-in file systems, all other will be ignored. For example:

# Given
s3_boto_1 = boto3.client('s3')
s3_boto_2 = boto3.client('s3', endpoint_url='http://my.s3.endpoint')
local = norfs.helpers.local()
s3_1 = norfs.helpers.s3(s3_boto_1)
s3_2 = norfs.helpers.s3(s3_boto_2)
memory = norfs.helpers.memory()

# Doing
cp = get_copy_client(local, s3_1, s3_2, memory)

# Is equivalent to
copier = norfs.copy.base.Copier(norfs.copy.base.GenericCopyStrategy())
copier.set_copy_policy(local.fs, local.fs, norfs.copy.local.LocalToLocalCopyStrategy())
copier.set_copy_policy(local.fs, s3_1.fs, norfs.copy.local.LocalToS3CopyStrategy(s3_boto_1))
copier.set_copy_policy(local.fs, s3_2.fs, norfs.copy.local.LocalToS3CopyStrategy(s3_boto_2))
copier.set_copy_policy(s3_1.fs, local.fs, norfs.copy.s3.S3ToLocalCopyStrategy())
copier.set_copy_policy(s3_1.fs, s3_1.fs, norfs.copy.s3.S3ToS3CopyStrategy(s3_boto_1))
copier.set_copy_policy(s3_2.fs, local.fs, norfs.copy.s3.S3ToLocalCopyStrategy())
copier.set_copy_policy(s3_2.fs, s3_2.fs, norfs.copy.s3.S3ToS3CopyStrategy(s3_boto_2))
cp = norfs.client.CopyClient(copier)
norfs.helpers.local()norfs.client.FileSystemClient[source]

Returns a new norfs.client.FileSystemClient managing a new instance of norfs.fs.local.LocalFileSystem.

norfs.helpers.memory(**kwargs: Any)norfs.client.FileSystemClient[source]

Returns a new norfs.client.FileSystemClient managing a new instance of norfs.fs.memory.MemoryFileSystem.

kwargs is passed directly to the file system constructor.

norfs.helpers.s3(*args: Any, **kwargs: Any)norfs.client.FileSystemClient[source]

Returns a new norfs.client.FileSystemClient managing a new instance of norfs.fs.s3.S3FileSystem.

args and kwargs are passed directly to the file system constructor.

Module contents

class norfs.Version[source]

Bases: object

MAJOR: str = '3'
MINOR: str = '0'
PATCH: str = '0'
RELEASE: str = '3.0.0'
VERSION: str = '3.0'