The 384p model keeps failing with covariance_matrix

[anooravk]

ValueError: Expected parameter covariance_matrix (Tensor of shape (4, 4)) of distribution MultivariateNormal(loc: torch.Size([4]), covariance_matrix: torch.Size([4, 4])) to satisfy the constraint PositiveDefinite(), but found invalid values:
2024-11-18T19:34:51.292478319Z tensor([[ 1.0000, -0.3333, -0.3333, -0.3333],
2024-11-18T19:34:51.292521520Z [-0.3333, 1.0000, -0.3333, -0.3333],
2024-11-18T19:34:51.292536989Z [-0.3333, -0.3333, 1.0000, -0.3333],
2024-11-18T19:34:51.292601801Z [-0.3333, -0.3333, -0.3333, 1.0000]])
2024-11-18T19:34:51.293700836Z {"prediction_id": null, "error": "Expected parameter covariance_matrix (Tensor of shape (4, 4)) of distribution MultivariateNormal(loc: torch.Size([4]), covariance_matrix: torch.Size([4, 4])) to satisfy the constraint PositiveDefinite(), but found invalid values:\ntensor([[ 1.0000, -0.3333, -0.3333, -0.3333],\n [-0.3333, 1.0000, -0.3333, -0.3333],\n [-0.3333, -0.3333, 1.0000, -0.3333],\n [-0.3333, -0.3333, -0.3333, 1.0000]])", "logger": "cog.server.runner", "timestamp": "2024-11-18T19:34:51.293277Z", "severity": "INFO", "message": "prediction failed"}

[feifeiobama]

There are several existing discussions of this issue (#121, #53). Please refer to them for fixes.