U
|
¹_òJ ã @ s¢ d dl Z d dlZd dlZd dlZd dlmZ d dlmZmZm Z m
|
Z
|
m Z m Z m Z mZ ddgZd e e edddZG dd d ZG d
|
d d eZG d d dZdS )é N)Ú OrderedDict)ÚIOÚAnyÚCallableÚDictÚListÚMutableMappingÚOptionalÚUnionÚ PathManagerÚ get_cache_dir)Ú cache_dirÚreturnc C s | dkr t j t dd¡¡} | S )aQ
|
Returns a default directory to cache static files
|
(usually downloaded from Internet), if None is provided.
|
Args:
|
cache_dir (None or str): if not None, will be returned as is.
|
If None, returns the default cache directory as:
|
1) $FVCORE_CACHE, if set
|
2) otherwise ~/.torch/fvcore_cache
|
NZ FVCORE_CACHEz~/.torch/fvcore_cache)ÚosÚpathÚ
|
expanduserÚgetenv)r © r ú./utils/file_io.pyr s
|
|
|
ÿc @ s e Zd ZdZdZeeef ddddZe e ddd Z
|
eeed
|
d d Z d"eee ee ee ee f dddZd#eeeeedddZeeed
|
ddZeeed
|
ddZeeed
|
ddZeee e d
|
d d Zeedd
|
d d Zeedd
|
d d!ZdS )$Ú PathHandlerzÒ
|
PathHandler is a base class that defines common I/O functionality for a URI
|
protocol. It routes I/O for a generic URI which may look like "protocol://*"
|
or a canonical filepath "/foo/bar/baz".
|
TN)Úkwargsr c C sT | j r"t|dkrPtd |¡n.t t¡}| ¡ D ]\}}| d ||¡¡ q4dS )a
|
Checks if the given arguments are empty. Throws a ValueError if strict
|
kwargs checking is enabled and args are non-empty. If strict kwargs
|
checking is disabled, only a warning is logged.
|
Args:
|
kwargs (Dict[str, Any])
|
r zUnused arguments: {}z$[PathManager] {}={} argument ignoredN) Ú_strict_kwargs_checkÚlenÚ
|
ValueErrorÚformatÚloggingÚ getLoggerÚ__name__ÚitemsÚwarning)Úselfr ÚloggerÚkÚvr r r Ú _check_kwargs1 s
|
|
ÿzPathHandler._check_kwargs)r c C s
|
t dS )zg
|
Returns:
|
List[str]: the list of URI prefixes this PathHandler can support
|
N©ÚNotImplementedError)r r r r Ú_get_supported_prefixesC s z#PathHandler._get_supported_prefixes©r r r c K s
|
t dS )aã
|
Get a filepath which is compatible with native Python I/O such as `open`
|
and `os.path`.
|
If URI points to a remote resource, this function may download and cache
|
the resource to local disk. In this case, this function is meant to be
|
used with read-only resources.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
local_path (str): a file path which exists on the local file system
|
Nr% ©r r r r r r Ú_get_local_pathJ s zPathHandler._get_local_pathÚréÿÿÿÿ©r ÚmodeÚ bufferingr r c K s
|
t dS )á¨
|
Open a stream to a URI, similar to the built-in `open`.
|
Args:
|
path (str): A URI supported by this PathHandler
|
mode (str): Specifies the mode in which the file is opened. It defaults
|
to 'r'.
|
buffering (int): An optional integer used to set the buffering policy.
|
Pass 0 to switch buffering off and an integer >= 1 to indicate the
|
size in bytes of a fixed-size chunk buffer. When no buffering
|
argument is given, the default buffering policy depends on the
|
underlying I/O implementation.
|
Returns:
|
file: a file-like object.
|
Nr% )r r r. r/ r r r r Ú_openX s zPathHandler._openF©Úsrc_pathÚdst_pathÚ overwriter r c K s
|
t dS )áV
|
Copies a source path to a destination path.
|
Args:
|
src_path (str): A URI supported by this PathHandler
|
dst_path (str): A URI supported by this PathHandler
|
overwrite (bool): Bool flag for forcing overwrite of existing file
|
Returns:
|
status (bool): True on success
|
Nr% )r r3 r4 r5 r r r r Ú_copyk s zPathHandler._copyc K s
|
t dS )úÆ
|
Checks if there is a resource at the given URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path exists
|
Nr% r) r r r Ú_exists} s zPathHandler._existsc K s
|
t dS )zÌ
|
Checks if the resource at the given URI is a file.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path is a file
|
Nr% r) r r r Ú_isfile s zPathHandler._isfilec K s
|
t dS )úÖ
|
Checks if the resource at the given URI is a directory.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path is a directory
|
Nr% r) r r r Ú_isdir s zPathHandler._isdirc K s
|
t dS )úÚ
|
List the contents of the directory at the provided URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
List[str]: list of contents in given path
|
Nr% r) r r r Ú_ls s zPathHandler._lsc K s
|
t dS )á
|
Recursive directory creation function. Like mkdir(), but makes all
|
intermediate-level directories needed to contain the leaf directory.
|
Similar to the native `os.makedirs`.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Nr% r) r r r Ú_mkdirs¥ s zPathHandler._mkdirsc K s
|
t dS )ú
|
Remove the file (not directory) at the provided URI.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Nr% r) r r r Ú_rm¯ s zPathHandler._rm)r+ r, )F)r Ú
|
__module__Ú __qualname__Ú__doc__r r Ústrr r$ r r' r* Úintr
|
r Úbytesr1 Úboolr7 r9 r: r< r> r@ rB r r r r r ( s8 ÿ þ üú
|
|
|
|
|
r c @ sì e Zd ZdZeeedddZd eeeee ee ee e ee
|
ee e e e e f d
|
d
|
d Zd eee ee d ddZeee dddZeee dddZeee dddZeeee dddZeeddddZeeddddZdS ) ÚNativePathHandlerz
|
Handles paths that can be accessed using Python native system calls. This
|
handler uses `open()` and `os.*` calls on the given path.
|
r( c K s | |¡ |S ©N)r$ r) r r r r* ¾ s
|
z!NativePathHandler._get_local_pathr+ r, NT)
|
r r. r/ ÚencodingÚerrorsÚnewlineÚclosefdÚopenerr r c
|
|
K s" | | ¡ t||||||||dS )uö
|
Open a path.
|
Args:
|
path (str): A URI supported by this PathHandler
|
mode (str): Specifies the mode in which the file is opened. It defaults
|
to 'r'.
|
buffering (int): An optional integer used to set the buffering policy.
|
Pass 0 to switch buffering off and an integer >= 1 to indicate the
|
size in bytes of a fixed-size chunk buffer. When no buffering
|
argument is given, the default buffering policy works as follows:
|
* Binary files are buffered in fixed-size chunks; the size of
|
the buffer is chosen using a heuristic trying to determine the
|
underlying deviceâs âblock sizeâ and falling back on
|
io.DEFAULT_BUFFER_SIZE. On many systems, the buffer will
|
typically be 4096 or 8192 bytes long.
|
encoding (Optional[str]): the name of the encoding used to decode or
|
encode the file. This should only be used in text mode.
|
errors (Optional[str]): an optional string that specifies how encoding
|
and decoding errors are to be handled. This cannot be used in binary
|
mode.
|
newline (Optional[str]): controls how universal newlines mode works
|
(it only applies to text mode). It can be None, '', '
|
', ' ',
|
and '
|
'.
|
closefd (bool): If closefd is False and a file descriptor rather than
|
a filename was given, the underlying file descriptor will be kept
|
open when the file is closed. If a filename is given closefd must
|
be True (the default) otherwise an error will be raised.
|
opener (Optional[Callable]): A custom opener can be used by passing
|
a callable as opener. The underlying file descriptor for the file
|
object is then obtained by calling opener with (file, flags).
|
opener must return an open file descriptor (passing os.open as opener
|
results in functionality similar to passing None).
|
See https://docs.python.org/3/library/functions.html#open for details.
|
Returns:
|
file: a file-like object.
|
)r/ rL rM rN rO rP )r$ Úopen)
|
r r r. r/ rL rM rN rO rP r r r r r1 Â s 0
|
øzNativePathHandler._openFr2 c
|
K s | |¡ tj |¡r8|s8t t¡}| d |¡¡ dS zt
|
||¡ W dS t k
|
r } z(t t¡}| d t |¡¡ W Y ¢dS d}~X Y nX dS )r6 z#Destination file {} already exists.FTzError in file copy - {}N) r$ r r Úexistsr r r Úerrorr ÚshutilÚcopyfileÚ ExceptionrF )r r3 r4 r5 r r! Úer r r r7 þ s
|
|
|
zNativePathHandler._copyc K s | |¡ tj |¡S rK )r$ r r rR r) r r r r9 s
|
zNativePathHandler._existsc K s | |¡ tj |¡S rK )r$ r r Úisfiler) r r r r: ! s
|
zNativePathHandler._isfilec K s | |¡ tj |¡S rK )r$ r r Úisdirr) r r r r< % s
|
zNativePathHandler._isdirc K s | |¡ t |¡S rK )r$ r Úlistdirr) r r r r> ) s
|
zNativePathHandler._lsc
|
K sR | |¡ ztj|dd W n0 tk
|
rL } z|jtjkr< W 5 d }~X Y nX d S )NT)Úexist_ok)r$ r ÚmakedirsÚOSErrorÚerrnoÚEEXIST)r r r rW r r r r@ - s
|
zNativePathHandler._mkdirsc K s | |¡ t |¡ d S rK )r$ r Úremover) r r r rB 6 s
|
zNativePathHandler._rm)r+ r, NNNTN)F)r rC rD rE rF r r* rG r rI r r
|
r rH r1 r7 r9 r: r< r r> r@ rB r r r r rJ ¸ sF ÷õ @ üú rJ c
|
@ sP e Zd ZU dZe Zeeef e d< e
|
Z e eedddZ e d%eeeeeee ee f dd d
|
Ze d&eeeeed d dZe eeedddZe eeedddZe eeedddZe eeedddZe eeee dddZe eedddd Z e eeddd d Z e edd d d!Z e edd"d#d$Z dS )'r z[
|
A class for users to open generic paths or translate generic paths to file names.
|
Ú_PATH_HANDLERS)r r c C s. t j ¡ D ] }| |¡r
|
t j| S q
|
t jS )zü
|
Finds a PathHandler that supports the given path. Falls back to the native
|
PathHandler if no other handler is found.
|
Args:
|
path (str): URI path to resource
|
Returns:
|
handler (PathHandler)
|
)r ra ÚkeysÚ
|
startswithÚ_NATIVE_PATH_HANDLER)r Úpr r r Z__get_path_handlerC s
|
|
z PathManager.__get_path_handlerr+ r, r- c K s t | ¡j| |fd|i|S )r0 r/ )r Ú _PathManager__get_path_handlerr1 )r r. r/ r r r r rQ R s
|
ÿÿÿzPathManager.openFr2 c K s0 t | ¡t |¡kstt | ¡j| ||f|S )r6 )r rf ÚAssertionErrorr7 )r3 r4 r5 r r r r Úcopyh s ÿþ
|
ÿÿzPathManager.copyr( c K s t | ¡j| f|S )a
|
Get a filepath which is compatible with native Python I/O such as `open`
|
and `os.path`.
|
If URI points to a remote resource, this function may download and cache
|
the resource to local disk.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
local_path (str): a file path which exists on the local file system
|
)r rf r* ©r r r r r Úget_local_path~ s ÿþþzPathManager.get_local_pathc K s t | ¡j| f|S )r8 )r rf r9 ri r r r rR s
|
|
ÿÿzPathManager.existsc K s t | ¡j| f|S )zÒ
|
Checks if there the resource at the given URI is a file.
|
Args:
|
path (str): A URI supported by this PathHandler
|
Returns:
|
bool: true if the path is a file
|
)r rf r: ri r r r rX s
|
|
ÿÿzPathManager.isfilec K s t | ¡j| f|S )r; )r rf r< ri r r r rY ¨ s
|
|
ÿÿzPathManager.isdirc K s t | ¡j| f|S )r= )r rf r> ri r r r Úlsµ s
|
|
ÿÿzPathManager.lsNc K s t | ¡j| f|S )r? )r rf r@ ri r r r Úmkdirs s
|
|
ÿÿzPathManager.mkdirsc K s t | ¡j| f|S )rA )r rf rB ri r r r ÚrmÏ s
|
|
ÿÿzPathManager.rm)Úhandlerr c C sZ t | tst| | ¡ D ] }|tjks,t| tj|< qtttj ¡ dd ddt_dS )z¡
|
Register a path handler associated with `handler._get_supported_prefixes`
|
URI prefixes.
|
Args:
|
handler (PathHandler)
|
c S s | d S )Nr r )Útr r r Ú<lambda>ì ó z.PathManager.register_handler.<locals>.<lambda>T)ÚkeyÚreverseN) Ú
|
isinstancer rg r' r ra r Úsortedr )rn Úprefixr r r Úregister_handlerÚ s ýÿz PathManager.register_handler)Úenabler c C s" | t j_t j ¡ D ]
|
}| |_qdS )a
|
Toggles strict kwargs checking. If enabled, a ValueError is thrown if any
|
unused parameters are passed to a PathHandler function. If disabled, only
|
a warning is given.
|
With a centralized file API, there's a tradeoff of convenience and
|
correctness delegating arguments to the proper I/O layers. An underlying
|
`PathHandler` may support custom arguments which should not be statically
|
exposed on the `PathManager` function. For example, a custom `HTTPURLHandler`
|
may want to expose a `cache_timeout` argument for `open()` which specifies
|
how old a locally cached resource can be before it's refetched from the
|
remote server. This argument would not make sense for a `NativePathHandler`.
|
If strict kwargs checking is disabled, `cache_timeout` can be passed to
|
`PathManager.open` which will forward the arguments to the underlying
|
handler. By default, checking is enabled since it is innately unsafe:
|
multiple `PathHandler`s could reuse arguments with different semantic
|
meanings or types.
|
Args:
|
enable (bool)
|
N)r rd r ra Úvalues)rx rn r r r Úset_strict_kwargs_checkingñ s z&PathManager.set_strict_kwargs_checking)r+ r, )F) r rC rD rE r ra r rF r Ú__annotations__rJ rd Ú staticmethodrf rG r r
|
r rH rQ rI rh rj rR rX rY r rk rl rm rw rz r r r r r ; sT
|
ÿ þ ÿ þ
|
)N)r^ r r rT Ú collectionsr Útypingr r r r r r r r
|
Ú__all__rF r r rJ r r r r r Ú<module> s (
|
