Skip to content

Fixed bug in ModelWeightAveraging class that led to corrupted model when metric to watch was NaN/Inf#1598

Merged
BloodAxe merged 1 commit into
masterfrom
feature/SG-1158-fix-model-averaging
Nov 1, 2023
Merged

Fixed bug in ModelWeightAveraging class that led to corrupted model when metric to watch was NaN/Inf#1598
BloodAxe merged 1 commit into
masterfrom
feature/SG-1158-fix-model-averaging

Conversation

@BloodAxe

@BloodAxe BloodAxe commented Nov 1, 2023

Copy link
Copy Markdown
Contributor

A class was not checking whether observed metric is a finite number and included a state dict of the model that had inf/nan metric value into averaging.

The fix was three-liner:

val = float(validation_results_dict[self.metric_to_watch])

if not np.isfinite(val):
  return False, None

@Louis-Dupont Louis-Dupont left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just make sure the greater_is_better without default is fine for classification (as raised by ofri)

@BloodAxe
BloodAxe merged commit c2821bb into master Nov 1, 2023
@BloodAxe
BloodAxe deleted the feature/SG-1158-fix-model-averaging branch November 1, 2023 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants