B
|
¹_òJ ã @ s¢ d dl Z d dlZd dlZd dlZd dlmZ d dlmZmZm Z m
|
Z
|
m Z m Z m Z mZ ddgZd e e edddZG dd d ZG d
|
d d eZG d d dZdS )é N)Ú OrderedDict)ÚIOÚAnyÚCallableÚDictÚListÚMutableMappingÚOptionalÚUnionÚ PathManagerÚ get_cache_dir)Ú cache_dirÚreturnc C s | dkr t j t dd¡¡} | S )aQ
|
Returns a default directory to cache static files
|
(usually downloaded from Internet), if None is provided.
|
Args:
|
cache_dir (None or str): if not None, will be returned as is.
|
If None, returns the default cache directory as:
|
1) $FVCORE_CACHE, if set
|
2) otherwise ~/.torch/fvcore_cache
|
NZ FVCORE_CACHEz~/.torch/fvcore_cache)ÚosÚpathÚ
|
expanduserÚgetenv)r © r ú./utils/file_io.pyr s
|
c @ s e Zd ZdZdZeeef ddddZe e ddd Z
|
eeed
|
d d Z d"eee ee ee ee f dddZd#eeeeedddZeeed
|
ddZeeed
|
ddZeeed
|
ddZeee e d
|
d d Zeedd
|
d d Zeedd
|
d d!ZdS )$Ú PathHandlerzÒ
|
PathHandler is a base class that defines common I/O functionality for a URI
|
protocol. It routes I/O for a generic URI which may look like "protocol://*"
|
or a canonical filepath "/foo/bar/baz".
|
TN)Úkwargsr c C sX | j r"t|dkrTtd |¡n2t t¡}x&| ¡ D ]\}}| d ||¡¡ q6W dS )a
|
Checks if the given arguments are empty. Throws a ValueError if strict
|
kwargs checking is enabled and args are non-empty. If strict kwargs
|
checking is disabled, only a warning is logged.
|
Args:
|
kwargs (Dict[str, Any])
|
r zUnused arguments: {}z$[PathManager] {}={} argument ignoredN) Ú_strict_kwargs_checkÚlenÚ
|
ValueErrorÚformatÚloggingÚ getLoggerÚ__name__ÚitemsÚwarning)Úselfr ÚloggerÚkÚvr r r Ú _check_kwargs1 s
|
zPathHandler._check_kwargs)r c C s
|
t dS )zg
|
Returns:
|
List[str]: the list of URI prefixes this PathHandler can support
|
N)ÚNotImplementedError)r r r r Ú_get_supported_prefixesC s z#PathHandler._get_supported_prefixes)r r r c K s
|
t dS )aã
|
Get a filepath which is compatible with native Python I/O such as `open`
|
and `os.path`.
|
If URI points to a remote resource, this function may download and cache
|
the resource to local disk. In this case, this function is meant to be
|
used with read-only resources.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
local_path (str): a file path which exists on the local file system
|
N)r% )r r r r r r Ú_get_local_pathJ s zPathHandler._get_local_pathÚréÿÿÿÿ)r ÚmodeÚ bufferingr r c K s
|
t dS )a¨
|
Open a stream to a URI, similar to the built-in `open`.
|
Args:
|
path (str): A URI supported by this PathHandler
|
mode (str): Specifies the mode in which the file is opened. It defaults
|
to 'r'.
|
buffering (int): An optional integer used to set the buffering policy.
|
Pass 0 to switch buffering off and an integer >= 1 to indicate the
|
size in bytes of a fixed-size chunk buffer. When no buffering
|
argument is given, the default buffering policy depends on the
|
underlying I/O implementation.
|
Returns:
|
file: a file-like object.
|
N)r% )r r r* r+ r r r r Ú_openX s zPathHandler._openF)Úsrc_pathÚdst_pathÚ overwriter r c K s
|
t dS )aV
|
Copies a source path to a destination path.
|
Args:
|
src_path (str): A URI supported by this PathHandler
|
dst_path (str): A URI supported by this PathHandler
|
overwrite (bool): Bool flag for forcing overwrite of existing file
|
Returns:
|
status (bool): True on success
|
N)r% )r r- r. r/ r r r r Ú_copyk s zPathHandler._copyc K s
|
t dS )zÆ
|
Checks if there is a resource at the given URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path exists
|
N)r% )r r r r r r Ú_exists} s zPathHandler._existsc K s
|
t dS )zÌ
|
Checks if the resource at the given URI is a file.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path is a file
|
N)r% )r r r r r r Ú_isfile s zPathHandler._isfilec K s
|
t dS )zÖ
|
Checks if the resource at the given URI is a directory.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path is a directory
|
N)r% )r r r r r r Ú_isdir s zPathHandler._isdirc K s
|
t dS )zÚ
|
List the contents of the directory at the provided URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
List[str]: list of contents in given path
|
N)r% )r r r r r r Ú_ls s zPathHandler._lsc K s
|
t dS )a
|
Recursive directory creation function. Like mkdir(), but makes all
|
intermediate-level directories needed to contain the leaf directory.
|
Similar to the native `os.makedirs`.
|
Args:
|
path (str): A URI supported by this PathHandler
|
N)r% )r r r r r r Ú_mkdirs¥ s zPathHandler._mkdirsc K s
|
t dS )z
|
Remove the file (not directory) at the provided URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
N)r% )r r r r r r Ú_rm¯ s zPathHandler._rm)r( r) )F)r Ú
|
__module__Ú __qualname__Ú__doc__r r Ústrr r$ r r& r' Úintr
|
r Úbytesr, Úboolr0 r1 r2 r3 r4 r5 r6 r r r r r ( s
|
|
|
|
|
|
r c @ sì e Zd ZdZeeedddZd eeeee ee ee e ee
|
ee e e e e f d
|
d
|
d Zd eee ee d ddZeee dddZeee dddZeee dddZeeee dddZeeddddZeeddddZdS ) ÚNativePathHandlerz
|
Handles paths that can be accessed using Python native system calls. This
|
handler uses `open()` and `os.*` calls on the given path.
|
)r r r c K s | |¡ |S )N)r$ )r r r r r r r' ¾ s
|
z!NativePathHandler._get_local_pathr( r) NT)
|
r r* r+ ÚencodingÚerrorsÚnewlineÚclosefdÚopenerr r c
|
|
K s" | | ¡ t||||||||dS )uö
|
Open a path.
|
Args:
|
path (str): A URI supported by this PathHandler
|
mode (str): Specifies the mode in which the file is opened. It defaults
|
to 'r'.
|
buffering (int): An optional integer used to set the buffering policy.
|
Pass 0 to switch buffering off and an integer >= 1 to indicate the
|
size in bytes of a fixed-size chunk buffer. When no buffering
|
argument is given, the default buffering policy works as follows:
|
* Binary files are buffered in fixed-size chunks; the size of
|
the buffer is chosen using a heuristic trying to determine the
|
underlying deviceâs âblock sizeâ and falling back on
|
io.DEFAULT_BUFFER_SIZE. On many systems, the buffer will
|
typically be 4096 or 8192 bytes long.
|
encoding (Optional[str]): the name of the encoding used to decode or
|
encode the file. This should only be used in text mode.
|
errors (Optional[str]): an optional string that specifies how encoding
|
and decoding errors are to be handled. This cannot be used in binary
|
mode.
|
newline (Optional[str]): controls how universal newlines mode works
|
(it only applies to text mode). It can be None, '', '
|
', ' ',
|
and '
|
'.
|
closefd (bool): If closefd is False and a file descriptor rather than
|
a filename was given, the underlying file descriptor will be kept
|
open when the file is closed. If a filename is given closefd must
|
be True (the default) otherwise an error will be raised.
|
opener (Optional[Callable]): A custom opener can be used by passing
|
a callable as opener. The underlying file descriptor for the file
|
object is then obtained by calling opener with (file, flags).
|
opener must return an open file descriptor (passing os.open as opener
|
results in functionality similar to passing None).
|
See https://docs.python.org/3/library/functions.html#open for details.
|
Returns:
|
file: a file-like object.
|
)r+ r? r@ rA rB rC )r$ Úopen)
|
r r r* r+ r? r@ rA rB rC r r r r r, Â s 0
|
zNativePathHandler._openF)r- r. r/ r r c
|
K s | |¡ tj |¡r8|s8t t¡}| d |¡¡ dS yt
|
||¡ dS t k
|
r } z"t t¡}| d t |¡¡ dS d}~X Y nX dS )aV
|
Copies a source path to a destination path.
|
Args:
|
src_path (str): A URI supported by this PathHandler
|
dst_path (str): A URI supported by this PathHandler
|
overwrite (bool): Bool flag for forcing overwrite of existing file
|
Returns:
|
status (bool): True on success
|
z#Destination file {} already exists.FTzError in file copy - {}N) r$ r r Úexistsr r r Úerrorr ÚshutilÚcopyfileÚ Exceptionr: )r r- r. r/ r r! Úer r r r0 þ s
|
|
|
zNativePathHandler._copyc K s | |¡ tj |¡S )N)r$ r r rE )r r r r r r r1 s
|
zNativePathHandler._existsc K s | |¡ tj |¡S )N)r$ r r Úisfile)r r r r r r r2 ! s
|
zNativePathHandler._isfilec K s | |¡ tj |¡S )N)r$ r r Úisdir)r r r r r r r3 % s
|
zNativePathHandler._isdirc K s | |¡ t |¡S )N)r$ r Úlistdir)r r r r r r r4 ) s
|
zNativePathHandler._lsc
|
K sR | |¡ ytj|dd W n0 tk
|
rL } z|jtjkr< W d d }~X Y nX d S )NT)Úexist_ok)r$ r ÚmakedirsÚOSErrorÚerrnoÚEEXIST)r r r rJ r r r r5 - s
|
zNativePathHandler._mkdirsc K s | |¡ t |¡ d S )N)r$ r Úremove)r r r r r r r6 6 s
|
zNativePathHandler._rm)r( r) NNNTN)F)r r7 r8 r9 r: r r' r; r r= r r
|
r r< r, r0 r1 r2 r3 r r4 r5 r6 r r r r r> ¸ s( " 5 r> c
|
@ sP e Zd ZU dZe Zeeef e d< e
|
Z e eedddZ e d%eeeeeee ee f dd d
|
Ze d&eeeeed d dZe eeedddZe eeedddZe eeedddZe eeedddZe eeee dddZe eedddd Z e eeddd d Z e edd d d!Z e edd"d#d$Z dS )'r z[
|
A class for users to open generic paths or translate generic paths to file names.
|
Ú_PATH_HANDLERS)r r c C s. x&t j ¡ D ]}| |¡r t j| S q W t jS )zü
|
Finds a PathHandler that supports the given path. Falls back to the native
|
PathHandler if no other handler is found.
|
Args:
|
path (str): URI path to resource
|
Returns:
|
handler (PathHandler)
|
)r rT ÚkeysÚ
|
startswithÚ_NATIVE_PATH_HANDLER)r Úpr r r Z__get_path_handlerC s
|
|
z PathManager.__get_path_handlerr( r) )r r* r+ r r c K s t | ¡j| |fd|i|S )a¨
|
Open a stream to a URI, similar to the built-in `open`.
|
Args:
|
path (str): A URI supported by this PathHandler
|
mode (str): Specifies the mode in which the file is opened. It defaults
|
to 'r'.
|
buffering (int): An optional integer used to set the buffering policy.
|
Pass 0 to switch buffering off and an integer >= 1 to indicate the
|
size in bytes of a fixed-size chunk buffer. When no buffering
|
argument is given, the default buffering policy depends on the
|
underlying I/O implementation.
|
Returns:
|
file: a file-like object.
|
r+ )r Ú _PathManager__get_path_handlerr, )r r* r+ r r r r rD R s
|
zPathManager.openF)r- r. r/ r r c K s0 t | ¡t |¡kstt | ¡j| ||f|S )aV
|
Copies a source path to a destination path.
|
Args:
|
src_path (str): A URI supported by this PathHandler
|
dst_path (str): A URI supported by this PathHandler
|
overwrite (bool): Bool flag for forcing overwrite of existing file
|
Returns:
|
status (bool): True on success
|
)r rY ÚAssertionErrorr0 )r- r. r/ r r r r Úcopyh s
|
|
zPathManager.copy)r r r c K s t | ¡j| f|S )a
|
Get a filepath which is compatible with native Python I/O such as `open`
|
and `os.path`.
|
If URI points to a remote resource, this function may download and cache
|
the resource to local disk.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
local_path (str): a file path which exists on the local file system
|
)r rY r' )r r r r r Úget_local_path~ s zPathManager.get_local_pathc K s t | ¡j| f|S )zÆ
|
Checks if there is a resource at the given URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path exists
|
)r rY r1 )r r r r r rE s
|
zPathManager.existsc K s t | ¡j| f|S )zÒ
|
Checks if there the resource at the given URI is a file.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path is a file
|
)r rY r2 )r r r r r rK s
|
zPathManager.isfilec K s t | ¡j| f|S )zÖ
|
Checks if the resource at the given URI is a directory.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path is a directory
|
)r rY r3 )r r r r r rL ¨ s
|
zPathManager.isdirc K s t | ¡j| f|S )zÚ
|
List the contents of the directory at the provided URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
List[str]: list of contents in given path
|
)r rY r4 )r r r r r Úlsµ s
|
zPathManager.lsNc K s t | ¡j| f|S )a
|
Recursive directory creation function. Like mkdir(), but makes all
|
intermediate-level directories needed to contain the leaf directory.
|
Similar to the native `os.makedirs`.
|
Args:
|
path (str): A URI supported by this PathHandler
|
)r rY r5 )r r r r r Úmkdirs s
|
zPathManager.mkdirsc K s t | ¡j| f|S )z
|
Remove the file (not directory) at the provided URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
)r rY r6 )r r r r r ÚrmÏ s
|
zPathManager.rm)Úhandlerr c C s^ t | tst| x(| ¡ D ] }|tjks.t| tj|< q W tttj ¡ dd ddt_dS )z¡
|
Register a path handler associated with `handler._get_supported_prefixes`
|
URI prefixes.
|
Args:
|
handler (PathHandler)
|
c S s | d S )Nr r )Útr r r Ú<lambda>ì ó z.PathManager.register_handler.<locals>.<lambda>T)ÚkeyÚreverseN) Ú
|
isinstancer rZ r& r rT r Úsortedr )r` Úprefixr r r Úregister_handlerÚ s z PathManager.register_handler)Úenabler c C s& | t j_xt j ¡ D ]
|
}| |_qW dS )a
|
Toggles strict kwargs checking. If enabled, a ValueError is thrown if any
|
unused parameters are passed to a PathHandler function. If disabled, only
|
a warning is given.
|
With a centralized file API, there's a tradeoff of convenience and
|
correctness delegating arguments to the proper I/O layers. An underlying
|
`PathHandler` may support custom arguments which should not be statically
|
exposed on the `PathManager` function. For example, a custom `HTTPURLHandler`
|
may want to expose a `cache_timeout` argument for `open()` which specifies
|
how old a locally cached resource can be before it's refetched from the
|
remote server. This argument would not make sense for a `NativePathHandler`.
|
If strict kwargs checking is disabled, `cache_timeout` can be passed to
|
`PathManager.open` which will forward the arguments to the underlying
|
handler. By default, checking is enabled since it is innately unsafe:
|
multiple `PathHandler`s could reuse arguments with different semantic
|
meanings or types.
|
Args:
|
enable (bool)
|
N)r rW r rT Úvalues)rj r` r r r Úset_strict_kwargs_checkingñ s z&PathManager.set_strict_kwargs_checking)r( r) )F) r r7 r8 r9 r rT r r: r Ú__annotations__r> rW Ú staticmethodrY r; r r
|
r r< rD r= r[ r\ rE rK rL r r] r^ r_ ri rl r r r r r ; s:
|
|
|
|
)N)rQ r r rG Ú collectionsr Útypingr r r r r r r r
|
Ú__all__r: r r r> r r r r r Ú<module> s (
|
