B
|
ÿ`i7 ã @ sè d dl Z d dlZd dlZd dlZd dl mZ d dlmZ d dlZd dl Z d dl
|
m Z d dl m Z d dlmZmZ ddlmZ G dd d eZG d
|
d d Zed d dZed ddZe jedddZed ddZ edddZ dS )é N)Ú defaultdict)ÚAny)Úcolored)Ú DataParallelÚDistributedDataParallelé )Ú PathManagerc @ s² e Zd ZdZd%ddejeeedddZ ee
|
dd d
|
Z ed d d Z dd Z dd Zdd ZddeedddZedddZedd d Zed d d Ze
|
d!d"d#Zd$S )&Ú Checkpointerz^
|
A checkpointer that can save/load model as well as extra checkpointable
|
objects.
|
Ú T)Ú save_to_disk)ÚmodelÚsave_dirr Úcheckpointablesc K sB t |ttfr|j}|| _t |¡| _t t ¡| _
|
|| _ || _ dS )a
|
Args:
|
model (nn.Module): model.
|
save_dir (str): a directory to save and find checkpoints.
|
save_to_disk (bool): if True, save checkpoint to disk, otherwise
|
disable saving for this checkpointer.
|
checkpointables (object): any checkpointable objects, i.e., objects
|
that have the `state_dict()` and `load_state_dict()` method. For
|
example, it can be used like
|
`Checkpointer(model, "dir", optimizer=optimizer)`.
|
N) Ú
|
isinstancer r Úmoduler Úcopyr ÚloggingÚ getLoggerÚ__name__Úloggerr r )Úselfr r r r © r ú./utils/checkpoint.pyÚ__init__ s zCheckpointer.__init__)ÚnameÚkwargsc K sÆ | j r | jsdS i }| j ¡ |d< x"| j ¡ D ]\}}| ¡ ||< q.W | |¡ d |¡}tj
|
| j |¡}tj |¡|kst || j d |¡¡ t |d¡}t ||¡ W dQ R X | |¡ dS )z¬
|
Dump model and checkpointables to a file.
|
Args:
|
name (str): name of the file.
|
kwargs (dict): extra arbitrary data to save.
|
Nr z{}.pthzSaving checkpoint to {}Úwb)r r r Ú
|
state_dictr ÚitemsÚupdateÚformatÚosÚpathÚjoinÚbasenameÚAssertionErrorr Úinfor ÚopenÚtorchÚsaveÚtag_last_checkpoint) r r r ÚdataÚkeyÚobjr$ Ú save_fileÚfr r r r) 5 s
|
|
zCheckpointer.save)r" c C s² |s| j d¡ i S | j d |¡¡ tj |¡sVt |¡}tj |¡sVtd |¡| |¡}|
|
|¡ xB| j ¡ D ]4\}}||krv| j d ||¡¡ | | |¡¡ qvW |S )aÐ
|
Load from the given checkpoint. When path points to network file, this
|
function has to be called on all ranks.
|
Args:
|
path (str): path or url to the checkpoint. If empty, will not load
|
anything.
|
Returns:
|
dict:
|
extra data loaded from the checkpoint that has not been
|
processed. For example, those saved with
|
:meth:`.save(**extra_data)`.
|
z0No checkpoint found. Training model from scratchzLoading checkpoint from {}zCheckpoint {} not found!zLoading {} from {})r r& r r! r" Úisfiler Úget_local_pathr% Ú
|
_load_fileÚ _load_modelr r Úload_state_dictÚpop)r r" Ú
|
checkpointr, r- r r r ÚloadM s
|
|
|
zCheckpointer.loadc C s t j | jd¡}t |¡S )za
|
Returns:
|
bool: whether a checkpoint exists in the target directory.
|
Úlast_checkpoint)r! r" r# r r Úexists)r r. r r r Úhas_checkpointo s zCheckpointer.has_checkpointc C s^ t j | jd¡}y(t |d¡}| ¡ ¡ }W dQ R X W n tk
|
rL dS X t j | j|¡S )z[
|
Returns:
|
str: The latest checkpoint file in target directory.
|
r8 ÚrNr
|
) r! r" r# r r r' ÚreadÚstripÚIOError)r r. r/ Z
|
last_savedr r r Úget_checkpoint_filew s z Checkpointer.get_checkpoint_filec s fddt j¡D }|S )z}
|
Returns:
|
list: All available checkpoint files (.pth files) in target
|
directory.
|
c s< g | ]4}t tj j|¡¡r| d ¡rtj j|¡qS )z.pth)r r0 r! r" r# r Úendswith)Ú.0Úfile)r r r ú
|
<listcomp> s z9Checkpointer.get_all_checkpoint_files.<locals>.<listcomp>)r Úlsr )r Zall_model_checkpointsr )r r Úget_all_checkpoint_files s
|
z%Checkpointer.get_all_checkpoint_files)Úresume)r" rF c C s |r| ¡ r| ¡ }| |¡S )a©
|
If `resume` is True, this method attempts to resume from the last
|
checkpoint, if exists. Otherwise, load checkpoint from the given path.
|
This is useful when restarting an interrupted training job.
|
Args:
|
path (str): path to the checkpoint.
|
resume (bool): if True, resume from the last checkpoint if it exists.
|
Returns:
|
same as :meth:`load`.
|
)r: r? r7 )r r" rF r r r Úresume_or_load s zCheckpointer.resume_or_load)Úlast_filename_basenamec C s6 t j | jd¡}t |d¡}| |¡ W dQ R X dS )z
|
Tag the last checkpoint.
|
Args:
|
last_filename_basename (str): the basename of the last filename.
|
r8 ÚwN)r! r" r# r r r' Úwrite)r rH r. r/ r r r r* £ s z Checkpointer.tag_last_checkpoint)r/ c C s t j|t d¡dS )a
|
Load a checkpoint file. Can be overwritten by subclasses to support
|
different formats.
|
Args:
|
f (str): a locally mounted file path.
|
Returns:
|
dict: with keys "model" and optionally others that are saved by
|
the checkpointer dict["model"] must be a dict which maps strings
|
to torch.Tensor or numpy arrays.
|
Úcpu)Ú map_location)r( r7 Údevice)r r/ r r r r2 s zCheckpointer._load_file)r6 c C sÎ | d¡}| |¡ t|d | j ¡ }x`t| ¡ D ]P}||kr6t|| j}t|| j}||kr6| j
|
d |||¡¡ | |¡ q6W | jj |dd}|j r²| j t|j ¡ |jrÊ| j t|j¡ dS )z~
|
Load weights from a checkpoint.
|
Args:
|
checkpoint (Any): checkpoint contains the weights.
|
r zmodule.zA'{}' has shape {} in the checkpoint but {} in the model! Skipped.F)ÚstrictN)r5 Ú_convert_ndarray_to_tensorÚ_strip_prefix_if_presentr r ÚlistÚkeysÚtupleÚshaper Úwarningr r4 Ú missing_keysr& Ú get_missing_parameters_messageÚunexpected_keysÚ!get_unexpected_parameters_message)r r6 Zcheckpoint_state_dictZmodel_state_dictÚkZ shape_modelZshape_checkpointÚ incompatibler r r r3 º s*
|
|
|
|
|
zCheckpointer._load_model)r c C sh xbt | ¡ D ]R}|| }t|tjsFt|tjsFtd |t |¡t|tjst
|
|¡||< qW dS )z«
|
In-place convert all numpy arrays in the state_dict to torch tensor.
|
Args:
|
state_dict (dict): a state-dict to be loaded to the model.
|
z,Unsupported type found in checkpoint! {}: {}N) rQ rR r ÚnpÚndarrayr( ÚTensorÚ
|
ValueErrorr ÚtypeÚ
|
from_numpy)r r rZ Úvr r r rO ã s
|
z'Checkpointer._convert_ndarray_to_tensorN)r
|
)r Ú
|
__module__Ú __qualname__Ú__doc__ÚnnÚModuleÚstrÚboolÚobjectr Údictr) r7 r: r? rE rG r* r2 r r3 rO r r r r r s "
|
)r c @ sD e Zd ZdZd eeedddZeedddZeed d
|
d Z dS ) ÚPeriodicCheckpointerzÔ
|
Save checkpoints periodically. When `.step(iteration)` is called, it will
|
execute `checkpointer.save` on the given checkpointer, if iteration is a
|
multiple of period or if `max_iter` is reached.
|
N)Ú checkpointerÚperiodÚmax_iterc C s || _ t|| _|| _dS )a<
|
Args:
|
checkpointer (Any): the checkpointer object used to save
|
checkpoints.
|
period (int): the period to save checkpoint.
|
max_iter (int): maximum number of iterations. When it is reached,
|
a checkpoint named "model_final" will be saved.
|
N)rm Úintrn ro )r rm rn ro r r r r s
|
z PeriodicCheckpointer.__init__)Ú iterationr c K sb t |}d|i}| |¡ |d | j dkrB| jjd |¡f| || jd kr^| jjd| dS )a
|
Perform the appropriate action at the given iteration.
|
Args:
|
iteration (int): the current iteration, ranged in [0, max_iter-1].
|
kwargs (Any): extra data to save, same as in
|
:meth:`Checkpointer.save`.
|
rq r r z model_{:07d}Ú model_finalN)rr )rp r rn rm r) r ro )r rq r Zadditional_stater r r Ústep s
|
zPeriodicCheckpointer.step)r r c K s | j j|f| dS )a
|
Same argument as :meth:`Checkpointer.save`.
|
Use this method to manually save checkpoints outside the schedule.
|
Args:
|
name (str): file name.
|
kwargs (Any): extra data to save, same as in
|
:meth:`Checkpointer.save`.
|
N)rm r) )r r r r r r r) s zPeriodicCheckpointer.save)N)
|
r rc rd re r rp r rs rh r) r r r r rl ú s rl )rR c C s, t | }d}|d dd | ¡ D ¡7 }|S )zý
|
Get a logging-friendly message to report parameter names (keys) that are in
|
the model but not found in a checkpoint.
|
Args:
|
keys (list[str]): List of keys that were not found in the checkpoint.
|
Returns:
|
str: message.
|
z1Some model parameters are not in the checkpoint:
|
Ú
|
c s s( | ] \}}d t |t| d V qdS )z ÚblueN)r Ú _group_to_str)rA rZ rb r r r ú <genexpr>8 s z1get_missing_parameters_message.<locals>.<genexpr>)Ú_group_checkpoint_keysr# r )rR ÚgroupsÚmsgr r r rW , s
|
rW c C s, t | }d}|d dd | ¡ D ¡7 }|S )zú
|
Get a logging-friendly message to report parameter names (keys) that are in
|
the checkpoint but not found in the model.
|
Args:
|
keys (list[str]): List of keys that were not found in the model.
|
Returns:
|
str: message.
|
z:The checkpoint contains parameters not used by the model:
|
rt c s s( | ] \}}d t |t| d V qdS )z ZmagentaN)r rv )rA rZ rb r r r rw I s z4get_unexpected_parameters_message.<locals>.<genexpr>)rx r# r )rR ry rz r r r rY = s rY )r Úprefixc s¸ t | ¡ }t fdd|D s&dS x*|D ]"}|t d
}| |¡| |< q,W y
|
| j}W n tk
|
rp Y nDX x@t| ¡ D ]0}t|dkrq|t d
}| |¡||< qW dS )z¡
|
Strip the prefix in metadata, if any.
|
Args:
|
state_dict (OrderedDict): a state-dict to be loaded to the model.
|
prefix (str): prefix.
|
c 3 s$ | ] }t |d kp| ¡V qdS )r N)ÚlenÚ
|
startswith)rA r, )r{ r r rw W s z+_strip_prefix_if_present.<locals>.<genexpr>Nr )ÚsortedrR Úallr| r5 Ú _metadataÚAttributeErrorrQ )r r{ rR r, ZnewkeyÚmetadatar )r{ r rP O s
|
|
rP c C sf t t}xX| D ]P}| d¡}|dkrF|d|
||d d
g }}n
|
|g }}|| |¡ qW |S )a.
|
Group keys based on common prefixes. A prefix is the string up to the final
|
"." in each key.
|
Args:
|
keys (list[str]): list of parameter names, i.e. keys in the model
|
checkpoint dict.
|
Returns:
|
dict[list]: keys with common prefixes are grouped into lists.
|
Ú.r Nr )r rQ ÚrfindÚextend)rR ry r, ÚposÚheadÚtailr r r rx p s
|
|
|
"
|
rx )Úgroupc C s: t | dkrdS t | dkr(d| d S dd | ¡ d S )z¼
|
Format a group of parameter name suffixes into a loggable string.
|
Args:
|
group (list[str]): list of parameter name suffixes.
|
Returns:
|
str: formated string.
|
r r
|
r r z.{z, Ú})r| r# )r r r r rv
s
|
rv ) Ú collectionsr r r! r Útypingr Únumpyr\ r( Útorch.nnrf Z termcolorr Ztorch.nn.parallelr r Úfile_ior rj r rl rQ rW rY Ú OrderedDictrh rP rx rv r r r r Ú<module> s& g2!
|
