Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cluster panic : should check field when new point #4796

Closed
wants to merge 7 commits into from

Conversation

CrazyJvm
Copy link
Contributor

we saw influxdb panic when we use continuous queries in cluster mode, the panic code is :

func (w *WriteShardRequest) unmarshalPoints() []models.Point {
    points := make([]models.Point, len(w.pb.GetPoints()))
    for i, p := range w.pb.GetPoints() {
        pt, err := models.ParsePoints(p)
        if err != nil {
            // A error here means that one node parsed the point correctly but sent an
            // unparseable version to another node.  We could log and drop the point and allow
            // anti-entropy to resolve the discrepancy but this shouldn't ever happen.
            panic(fmt.Sprintf("failed to parse point: `%v`: %v", string(p), err))
        }
        points[i] = pt[0]
    }
    return points
}

after investigation , we found the root cause is lack of field check when new point, the current code will not throw any errors but will make above code panic. The reason why there may exist point without fields is that when cq calculate all points in an interval which there's no point in this interval at all.

func NewPoint(name string, tags Tags, fields Fields, time time.Time) (Point, error) {
    for key, value := range fields {
        if fv, ok := value.(float64); ok {
            // Ensure the caller validates and handles invalid field values
            if math.IsNaN(fv) {
                return nil, fmt.Errorf("NaN is an unsupported value for field %s", key)
            }
        }
    }

    return &point{
        key:    MakeKey([]byte(name), tags),
        time:   time,
        fields: fields.MarshalBinary(),
    }, nil
}

this PR add field check.

one more thing , really should panic here?

panic(fmt.Sprintf("failed to parse point: `%v`: %v", string(p), err))

It's better to log error than panic directly I think.

  • CHANGELOG.md updated
  • Rebased/mergable
  • Tests pass
  • Sign CLA (if not already signed)

@CrazyJvm CrazyJvm changed the title should check field when new point cluster panic : should check field when new point Nov 14, 2015
@CrazyJvm CrazyJvm closed this Nov 14, 2015
@CrazyJvm CrazyJvm reopened this Nov 14, 2015
@otoolep
Copy link
Contributor

otoolep commented Nov 14, 2015

@CrazyJvm -- your change is failing due to formatting issues.

https://circleci.com/gh/influxdb/influxdb/7538#artifacts

run 'go tool vet --composites=false ./' to see the errors it flags and correct your source code. bash circle-test.sh returned exit code 1

@CrazyJvm
Copy link
Contributor Author

retest this please

local : all tests passed and no format problem.

@CrazyJvm
Copy link
Contributor Author

@otoolep fixed format & test.

@otoolep
Copy link
Contributor

otoolep commented Nov 17, 2015

Thanks @CrazyJvm -- can you squash the commits? The messages are not very helpful, so one commit, with one message would be great.

@@ -962,6 +962,11 @@ func unescapeStringField(in string) string {
// NewPoint returns a new point with the given measurement name, tags, fields and timestamp. If
// an unsupported field value (NaN) is passed, this function returns an error.
func NewPoint(name string, tags Tags, fields Fields, time time.Time) (Point, error) {
//should check field exists
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: can you remove this comment? The code is quite clear without it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

@otoolep
Copy link
Contributor

otoolep commented Nov 17, 2015

This makes sense to me, I agree that a point without fields is nonsensical -- thanks @CrazyJvm

@jwilder -- can I get a +1?

+1 from me pending squash and minor requested changes.

pires and others added 6 commits November 17, 2015 12:54
should check field when new point

add test

fix format

fix test

should check field when new point

remove unnecessary comment

should check field when new point

fixed export lint issues in services/admin

updated CHANGELOG.md

corrected URL

Backporting changes from build.py to package.sh. Also fixes a few miscellaneous package permissions issues that Debian's lintian complained about.

Add interface for heap to support Reverse for `order by desc`

linted service/admin
@CrazyJvm
Copy link
Contributor Author

@otoolep removed comment and squashed commits already, but seems introducing another commit. I'm not sure it's ok for you?

@otoolep
Copy link
Contributor

otoolep commented Nov 17, 2015

You merged, you didn't rebase. Try this while on your branch:

git pull --rebase origin master

This assumes your origin is in sync with ours. You will also need to force push to your branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants