norfs

Normalized filesystem. This library offers a common interface to interact with multiple filesystems, local or remote:

import norfs.helpers

local = norfs.helpers.local()
demo_file = local.file('demo.txt')
demo_file.write(b'Hello World')
print(demo_file.read())

Installation

Install with pip: pip install norfs.

You can also download the source code from the git repository.

Supported Filesystems

• Local filesystem

• S3

• In-memory

The library is easily extensible! If you want a new filesystem supported implement it yourself or create an issue.

Local filesystem

Implements the norfs contract to work with files and directories in the local filesystem:

import norfs.helpers

local = norfs.helpers.local()

my_directory = local.dir("/my/local/directory")
my_file = local.file("/my/local/file.txt")

S3

Implements the norfs contract to work with files and directories in an S3 compatible store:

import norfs.helpers
import boto3

s3 = norfs.helpers.s3(s3_client=boto3.client("s3"))

my_directory = s3.dir("myBucket/my/s3/directory")
my_file = s3.file("myBucket/my/s3/file.txt")

In-memory

Implements the norfs contract to work with files and directories in a very simple in-memory filesystem:

import norfs.helpers

memory = norfs.helpers.memory()

my_directory = memory.dir("/my/memory/directory")
my_file = memory.file("/my/memory/file.txt")

The norfs interface

The most public interface norfs exposes is composed of the Clients, the Filesystem objects and the helpers module.

Clients

class norfs.client.CopyClient(copier: norfs.copy.base.Copier)

Bases: object

norfs.client.CopyClient provides a unified simple copy API for any norfs.filesystem.File or norfs.filesystem.Directory from any file system. It is usually accessed by using norfs.helpers.get_copy_client():

import norfs.helpers

local = norfs.helpers.local() cp_local_only = norfs.helpers.get_copy_client(local)

cp_local_only.copy(local.file('source_file.txt'), local.file('target_file.txt'))


memory = norfs.helpers.memory()

import boto3 s3 = norfs.helpers.s3(s3_client=boto3.client('s3'))

cp_for_all = norfs.helpers.get_copy_client(local, s3, memory)

cp_for_all.copy(s3.file('myBucket/source_file.txt'), local.file('target_file.txt'))

norfs.helpers.get_copy_client() returns a norfs.client.CopyClient instance configured with copy strategies for each of the file system clients passed.

A norfs.copy.base.Copier can have copy policies set for a pair of source and destination file systems to implement a better strategy of copying between them than read source and write destination. norfs.helpers.get_copy_client() helps you by setting these for you.

A norfs.client.CopyClient exposes the following interface:

property copier

The norfs.copy.base.Copier instance managed by the client.

copy(src: norfs.filesystem.BaseFileSystemObject, dst: norfs.filesystem.BaseFileSystemObject) → None

Copies src to dst, no mater the file systems they are on. src and dst can by both norfs.filesystem.File or norfs.filesystem.Directory. The only operation not supported is copying from a norfs.filesystem.Directory into a norfs.filesystem.File as it does not make sense.

If source is a norfs.filesystem.Directory and destination is a norfs.filesystem.File it raises a TypeError.

On copy failure it raises a norfs.fs.base.FileSystemOperationError.

class norfs.client.FileSystemClient(fs: norfs.fs.base.BaseFileSystem)

Bases: object

norfs.client.FileSystemClient provides a way to access the file system objects of a given file system. It is a handy class that provides easy access to :class:norfs.filesystem.File and norfs.filesystem.Directory instances. It is usually obtained using norfs.helpers:

import norfs.helpers

local_fs_client = norfs.helpers.local()

memory_fs_client = norfs.helpers.memory()

import boto3
s3_fs_client = norfs.helpers.s3(s3_client=boto3.client('s3'))

A norfs.client.FileSystemClient exposes the following interface:

dir(path: str) → norfs.filesystem.Directory

Returns a norfs.filesystem.Directory instance for the given path in the managed file system.

file(path: str) → norfs.filesystem.File

Returns a norfs.filesystem.File instance for the given path in the managed file system.

property fs

The norfs.filesystem.BaseFileSystemObject the client is managing.

Filesystem objects

norfs.filesystem.BaseFileSystemObject represents any object in the filesystem. It is the most abstract representation.

A norfs.filesystem.BaseFileSystemObject exposes the following interface:

class norfs.filesystem.BaseFileSystemObject(filesystem: norfs.fs.base.BaseFileSystem, path_str: Optional[str], *, _path: Optional[norfs.fs.base.Path] = None)

Bases: object

as_dir() → norfs.filesystem.Directory

Returns itself as a Directory instance or raises a NotADirectoryError.

as_file() → norfs.filesystem.File

Returns itself as a norfs.filesystem.File instance or raises a norfs.fs.base.NotAFileError.

copy_object() → norfs.copy.base.CopyFileSystemObject

copy_to(dst: norfs.filesystem.BaseFileSystemObject, strategy: norfs.copy.base.CopyStrategy) → None

exists() → bool

Returns whether self exists in the file system.

is_dir() → bool

Returns whether self is a Directory.

is_file() → bool

Returns whether self is a File.

property name

The name of self.

parent() → norfs.filesystem.Directory

Return parent norfs.filesystem.Directory of self.

property path

The full, absolute, path of self in the file system.

remove() → None

Tries to remove self from the file system. On failure it raises a norfs.fs.base.FileSystemOperationError

property uri

The URI that points to self in the file system.

class norfs.filesystem.Directory(filesystem: norfs.fs.base.BaseFileSystem, path_str: Optional[str], *, _path: Optional[norfs.fs.base.Path] = None)

Bases: norfs.filesystem.BaseFileSystemObject

as_dir() → norfs.filesystem.Directory

Returns itself as a norfs.filesystem.Directory instance or raises a NotADirectoryError.

copy_object() → norfs.copy.base.CopyFileSystemObject

file(path: str) → norfs.filesystem.File

Returns a norfs.filesystem.File with its path as being the given path relative to the current directory.

is_dir() → bool

Returns whether self is a norfs.filesystem.Directory.

list() → Iterable[norfs.filesystem.BaseFileSystemObject]

Returns the contents of the norfs.filesystem.Directory in the file system as a list of norfs.filesystem.BaseFileSystemObject s.

If the norfs.filesystem.Directory does not exist the list will be empty.

remove() → None

Tries to remove self from the file system.

On failure it raises a norfs.fs.base.FileSystemOperationError

subdir(path: str) → norfs.filesystem.Directory

Returns a norfs.filesystem.Directory with its path as being the given path relative to the current directory.

class norfs.filesystem.File(filesystem: norfs.fs.base.BaseFileSystem, path_str: Optional[str], *, _path: Optional[norfs.fs.base.Path] = None)

Bases: norfs.filesystem.BaseFileSystemObject

as_file() → norfs.filesystem.File

Returns itself as a norfs.filesystem.File instance or raises a norfs.fs.base.NotAFileError.

copy_object() → norfs.copy.base.CopyFileSystemObject

is_file() → bool

Returns whether self is a norfs.filesystem.File.

read() → bytes

Returns the contents of the file.

If it fails to read the file a norfs.fs.base.FileSystemOperationError will be raised.

remove() → None

Tries to remove self from the file system.

On failure it raises a norfs.fs.base.FileSystemOperationError

set_perms(policies: List[norfs.permissions.Policy]) → None

Set the access policies for the file.

The actual meaning of the scopes and permissions depends on the underlying norfs.fs.base.BaseFileSystem implementation.

set_props(*, content_type: Optional[str] = None, tags: Optional[Dict[str, str]] = None, metadata: Optional[Dict[str, str]] = None) → None

Set the properties for the file.

The actual setting of the metadata depends on the underlying norfs.fs.base.BaseFileSystem implementation.

write(content: bytes) → None

Sets the contents of the file. If the parent directory does not exist it is created.

If it fails to write the file a norfs.fs.base.FileSystemOperationError will be raised.

Helpers module

The norfs.helpers offers functions that serve as shortcuts for common operations with the library.

The norfs.helpers contains the following functions:

norfs.helpers.get_copy_client(*args: norfs.client.FileSystemClient) → norfs.client.CopyClient

Helper function to get a norfs.copy.base.CopyClient instance configured with copy strategies for the given file systems. This function only sets the built-in norfs.copy.base.CopyStrategy s for the built-in file systems, all other will be ignored. For example:

# Given
s3_boto_1 = boto3.client('s3')
s3_boto_2 = boto3.client('s3', endpoint_url='http://my.s3.endpoint')
local = norfs.helpers.local()
s3_1 = norfs.helpers.s3(s3_boto_1)
s3_2 = norfs.helpers.s3(s3_boto_2)
memory = norfs.helpers.memory()

# Doing
cp = get_copy_client(local, s3_1, s3_2, memory)

# Is equivalent to
copier = norfs.copy.base.Copier(norfs.copy.base.GenericCopyStrategy())
copier.set_copy_policy(local.fs, local.fs, norfs.copy.local.LocalToLocalCopyStrategy())
copier.set_copy_policy(local.fs, s3_1.fs, norfs.copy.local.LocalToS3CopyStrategy(s3_boto_1))
copier.set_copy_policy(local.fs, s3_2.fs, norfs.copy.local.LocalToS3CopyStrategy(s3_boto_2))
copier.set_copy_policy(s3_1.fs, local.fs, norfs.copy.s3.S3ToLocalCopyStrategy())
copier.set_copy_policy(s3_1.fs, s3_1.fs, norfs.copy.s3.S3ToS3CopyStrategy(s3_boto_1))
copier.set_copy_policy(s3_2.fs, local.fs, norfs.copy.s3.S3ToLocalCopyStrategy())
copier.set_copy_policy(s3_2.fs, s3_2.fs, norfs.copy.s3.S3ToS3CopyStrategy(s3_boto_2))
cp = norfs.client.CopyClient(copier)

norfs.helpers.local() → norfs.client.FileSystemClient

Returns a new norfs.client.FileSystemClient managing a new instance of norfs.fs.local.LocalFileSystem.

norfs.helpers.memory(**kwargs: Any) → norfs.client.FileSystemClient

Returns a new norfs.client.FileSystemClient managing a new instance of norfs.fs.memory.MemoryFileSystem.

kwargs is passed directly to the file system constructor.

norfs.helpers.s3(*args: Any, **kwargs: Any) → norfs.client.FileSystemClient

Returns a new norfs.client.FileSystemClient managing a new instance of norfs.fs.s3.S3FileSystem.

args and kwargs are passed directly to the file system constructor.

Indices and tables

• Index

• Module Index

• Search Page