re
/
afero
forked from mirror/afero
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Update readme to reflect migration of filters to filesystems

This commit is contained in:
Steve Francia 2016-01-11 20:27:36 -05:00
parent 98c426b27b
commit 4f6cfb713a
2 changed files with 133 additions and 90 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

220
README.md
View File

@ -30,10 +30,8 @@ filesystem for full interoperability.
* Interoperation between a variety of file system types * Interoperation between a variety of file system types
* A set of interfaces to encourage and enforce interoperability between backends * A set of interfaces to encourage and enforce interoperability between backends
* An atomic cross platform memory backed file system * An atomic cross platform memory backed file system
* Support for compositional file systems by joining various different file systems (see httpFs) * Support for compositional (union) file systems by combining multiple file systems acting as one
* Filtering of calls to intercept opening / modifying files, several filters * Specialized backends which modify existing filesystems (Read Only, Regexp filtered)
may be stacked.
* Unions of filesystems to overlay two filesystems. These may be stacked.
* A set of utility functions ported from io, ioutil & hugo to be afero aware * A set of utility functions ported from io, ioutil & hugo to be afero aware
@ -224,7 +222,76 @@ func testExistence(name string, e bool, t *testing.T) {
} }
``` ```
## Using Afero with Http # Available Backends
## Operating System Native
### OsFs
The first is simply a wrapper around the native OS calls. This makes it
very easy to use as all of the calls are the same as the existing OS
calls. It also makes it trivial to have your code use the OS during
operation and a mock filesystem during testing or as needed.
## Memory Backed Storage
### MemMapFs
Afero also provides a fully atomic memory backed filesystem perfect for use in
mocking and to speed up unnecessary disk io when persistence isn’t
necessary. It is fully concurrent and will work within go routines
safely.
#### InMemoryFile
As part of MemMapFs, Afero also provides an atomic, fully concurrent memory
backed file implementation. This can be used in other memory backed file
systems with ease. Plans are to add a radix tree memory stored file
system using InMemoryFile.
## Network Interfaces
### SftpFs
Afero has experimental support for secure file transfer protocol (sftp). Which can
be used to perform file operations over a encrypted channel.
## Filtering Backends
### BasePathFs
The BasePathFs restricts all operations to a given path within an Fs.
The given file name to the operations on this Fs will be prepended with
the base path before calling the source Fs.
```go
bp := &BasePathFs{source: &OsFs{}, path: "/base/path"}
```
### ReadOnlyFs
A thin wrapper around the source Fs providing a read only view.
```go
fs := &ReadOnlyFs{source: &OsFs{}}
_, err := fs.Create("/file.txt")
// err = syscall.EPERM
```
# RegexpFs
A filtered view on file names, any file NOT matching
the passed regexp will be treated as non-existing.
Files not matching the regexp provided will not be created.
Directories are not filtered.
```go
fs := &RegexpFs{re: regexp.MustCompile(`\.txt$`), source: &MemMapFs{}}
_, err := fs.Create("/file.html")
// err = syscall.ENOENT
```
### HttpFs
Afero provides an http compatible backend which can wrap any of the existing Afero provides an http compatible backend which can wrap any of the existing
backends. backends.
@ -236,38 +303,76 @@ Afero provides an httpFs file system which satisfies this requirement.
Any Afero FileSystem can be used as an httpFs. Any Afero FileSystem can be used as an httpFs.
```go ```go
httpFs := &afero.HttpFs{SourceFs: <ExistingFS>} httpFs := &afero.HttpFs{source: <ExistingFS>}
fileserver := http.FileServer(httpFs.Dir(<PATH>))) fileserver := http.FileServer(httpFs.Dir(<PATH>)))
http.Handle("/", fileserver) http.Handle("/", fileserver)
``` ```
# Available Backends ## Composite Backends
## OsFs Afero provides the ability have two filesystems (or more) act as a single
file system.
The first is simply a wrapper around the native OS calls. This makes it ### CacheOnReadFs
very easy to use as all of the calls are the same as the existing OS
calls. It also makes it trivial to have your code use the OS during
operation and a mock filesystem during testing or as needed.
## MemMapFs The CacheOnReadFs will lazily make copies of any accessed files from the base
layer into the overlay. Subsequent reads will be pulled from the overlay
directly permitting the request is within the cache duration of when it was
created in the overlay.
Afero also provides a fully atomic memory backed filesystem perfect for use in If the base filesystem is writeable, any changes to files will be
mocking and to speed up unnecessary disk io when persistence isn’t done first to the base, then to the overlay layer. Write calls to open file
necessary. It is fully concurrent and will work within go routines handles like `Write()` or `Truncate()` to the overlay first.
safely.
### InMemoryFile To writing files to the overlay only, you can use the overlay Fs directly (not
via the union Fs).
As part of MemMapFs, Afero also provides an atomic, fully concurrent memory Cache files in the layer for the given time.Duration, a cache duration of 0
backed file implementation. This can be used in other memory backed file means "forever" meaning the file will not be re-requested from the base ever.
systems with ease. Plans are to add a radix tree memory stored file
system using InMemoryFile.
## SftpFs A read-only base will make the overlay also read-only but still copy files
from the base to the overlay when they're not present (or outdated) in the
caching layer.
```go
base := &OsFs{}
layer := &MemMapFs{}
ufs := &CacheOnReadFs{base: base, layer: layer, cacheTime: 100 * time.Second}
```
### CopyOnWriteFs()
The CopyOnWriteFs is a read only base file system with a potentially
writeable layer on top.
Read operations will first look in the overlay and if not found there, will
serve the file from the base.
Changes to the file system will only be made in the overlay.
Any attempt to modify a file found only in the base will copy the file to the
overlay layer before modification (including opening a file with a writable
handle).
Removing and Renaming files present only in the base layer is not currently
permitted. If a file is present in the base layer and the overlay, only the
overlay will be removed/renamed.
The writable overlay layer is currently limited to MemMapFs.
```go
base := &OsFs{}
roBase := &ReadOnlyFs{source: base}
ufs := &CopyOnWriteFs{base: roBase, layer: &MemMapFs{}}
fh, _ = ufs.Create("/home/test/file2.txt")
fh.WriteString("This is a test")
fh.Close()
```
In this example all write operations will only occur in memory (&MemMapFs{})
leaving the base filesystem (&OsFs) untouched.
Afero has experimental support for secure file transfer protocol (sftp). Which can
be used to perform file operations over a encrypted channel.
## Desired/possible backends ## Desired/possible backends
@ -278,71 +383,6 @@ implement:
* ZIP * ZIP
* TAR * TAR
* S3 * S3
* Mem buffering to disk/network
# Filters
You can add "filtering" to an Fs by adding a FilterFs to an existing Afero Fs
like
```go
ROFs := afero.NewFilter(AppFs)
ROFs.AddFilter(afero.NewReadonlyFilter())
```
The ROFs behaves like a normal afero.Fs now, with the only exception, that it
provides a readonly view of the underlying AppFs.
The FilterFs is run before the source Fs and may intercept the call to the
underlying source Fs and can modify the returned data. If it does not wish to
do so, it just returns the data from the source.
The `AddFilter` adds a new FilterFs before any existing filters.
## Available filters
# NewBasePathFs(Fs, path)
The BasePathFs restricts all operations to a given path within an Fs.
The given file name to the operations on this Fs will be prepended with
the base path before calling the base Fs.
# NewReadonlyFilter()
provide a read only view of the source Fs
# NewRegexpFilter(\*regexp.Regexp)
provide a filtered view on file names, any file (not directory) NOT matching
the passed regexp will be treated as non-existing
## Unions
Afero has the possibilty to overlay two filesystems as a union, these are
special types of filters. To create a new union Fs use the `NewUnionFs()`. The
example below creates an memory cache for the OsFs:
```go
ufs := NewUnionFs(&OsFs{}, &MemMapFs{}, NewCacheUnionFs(1 * time.Minute))
```
Available UnionFs are:
### NewCacheUnionFs(time.Duration)
Cache files in the layer for the given time.Duration, a cache duration of 0
means "forever".
If the base filesystem is writeable, any changes to files will be done first
to the base, then to the overlay layer. Write calls to open file handles
like `Write()` or `Truncate()` to the overlay first.
A read-only base will make the overlay also read-only but still copy files
from the base to the overlay when they're not present (or outdated) in the
caching layer.
### NewCoWUnionFs()
A CopyOnWrite union: any attempt to modify a file in the base will copy
the file to the overlay layer before modification. This overlay layer is
currently limited to MemMapFs.
# About the project # About the project

3
union.go
View File

@ -229,6 +229,7 @@ func copyToLayer(base Fs, layer Fs, name string) error {
} }
defer bfh.Close() defer bfh.Close()
// First make sure the directory exists
exists, err := Exists(layer, filepath.Dir(name)) exists, err := Exists(layer, filepath.Dir(name))
if err != nil { if err != nil {
return err return err
@ -240,12 +241,14 @@ func copyToLayer(base Fs, layer Fs, name string) error {
} }
} }
// Create the file on the overlay
lfh, err := layer.Create(name) lfh, err := layer.Create(name)
if err != nil { if err != nil {
return err return err
} }
n, err := io.Copy(lfh, bfh) n, err := io.Copy(lfh, bfh)
if err != nil { if err != nil {
// If anything fails, clean up the file
layer.Remove(name) layer.Remove(name)
lfh.Close() lfh.Close()
return err return err