healthcheck problem #47
Comments
|
You most likely has rp_filter sysctl enabled and because of this packets with vip as a src dropped from eth0 (as they don’t pass rp_filter check with vip configured on lo). In general if you see packets in tcpdump - then it has passed xdp (as xdp works before tcpdump) |
|
Finally found beside rp_filter, accept_local should be open too, thanks for you advice |
|
Hi @liayusuke and @tehnerd,
Why katran can't implement healthcheck as haproxy or keepalived for Loadbalancing. Could you help me to debug this case. Thank you! |
|
this is a library. not a fully made service. it is implementing the way to forward healtchecks. but it does not actually implement any way to run them (it is up to user of the library how to implement them). so it is up to you how to do healtchecks and send them. katran library would help you to do ipip encapsulation if your healthcheck has proper somark. as for why it does not implement: grpc server is an example of how this library could be used. not a final product. also please create a new issue instead of posting in old one. FB folks, who owns this library, probably do not monitor closed issues, nor get notified that someone asked anything there |
facebook-github-bot
pushed a commit
that referenced
this issue
Mar 20, 2024
Summary: After diving in all the build system I found that the first error mentioned in #219 and #220 ``` 1597 | static_assert(formattable_char, "Mixing character types is disallowed."); ``` Was basically happening while compiling folly, after compiling it by itself I noticed this didn't happened, so I found that there was an issue with the fmt dependencies, removing the one that was downloaded by katran The issue mentioned in: #221 Was because katran was configured to use C++14 by default, and some of the libraries of folly require C++17, updated our requirements. Test Plan: TEST Output: ``` Test project /home/ubuntu/ivanmorett/katran/_build/build/katran/lib/tests Start 1: IpHelpersTests.testV4ParsingBe 1/56 Test #1: IpHelpersTests.testV4ParsingBe ....................... Passed 0.01 sec Start 2: IpHelpersTests.testV4ParsingInt 2/56 Test #2: IpHelpersTests.testV4ParsingInt ...................... Passed 0.01 sec Start 3: IpHelpersTests.testV6ParsingBe 3/56 Test #3: IpHelpersTests.testV6ParsingBe ....................... Passed 0.01 sec Start 4: IpHelpersTests.testV6ParsingInt 4/56 Test #4: IpHelpersTests.testV6ParsingInt ...................... Passed 0.01 sec Start 5: IpHelpersTests.testIncorrectAddr 5/56 Test #5: IpHelpersTests.testIncorrectAddr ..................... Passed 0.01 sec Start 6: CHHelpersTest.testMaglevCHSameWeight 6/56 Test #6: CHHelpersTest.testMaglevCHSameWeight ................. Passed 0.01 sec Start 7: CHHelpersTest.testMaglevV2CHSameWeight 7/56 Test #7: CHHelpersTest.testMaglevV2CHSameWeight ............... Passed 0.01 sec Start 8: CHHelpersTest.testMaglevCHDiffWeight 8/56 Test #8: CHHelpersTest.testMaglevCHDiffWeight ................. Passed 0.01 sec Start 9: CHHelpersTest.testMaglevV2CHDiffWeight 9/56 Test #9: CHHelpersTest.testMaglevV2CHDiffWeight ............... Passed 0.01 sec Start 10: CHHelpersTest.testMaglevWeightsSumLargerThanRing 10/56 Test #10: CHHelpersTest.testMaglevWeightsSumLargerThanRing ..... Passed 0.01 sec Start 11: CHHelpersTest.testMaglevWeightsSumBelowRingSize 11/56 Test #11: CHHelpersTest.testMaglevWeightsSumBelowRingSize ...... Passed 0.01 sec Start 12: KatranLbTest.testChangeMac 12/56 Test #12: KatranLbTest.testChangeMac ........................... Passed 0.01 sec Start 13: KatranLbTest.testIfIndex 13/56 Test #13: KatranLbTest.testIfIndex ............................. Passed 0.01 sec Start 14: KatranLbTest.testVipHelpers 14/56 Test #14: KatranLbTest.testVipHelpers .......................... Passed 0.14 sec Start 15: KatranLbTest.testAddingInvalidVip 15/56 Test #15: KatranLbTest.testAddingInvalidVip .................... Passed 0.01 sec Start 16: KatranLbTest.testRealHelpers 16/56 Test #16: KatranLbTest.testRealHelpers ......................... Passed 0.01 sec Start 17: KatranLbTest.testRealFlags 17/56 Test #17: KatranLbTest.testRealFlags ........................... Passed 0.01 sec Start 18: KatranLbTest.testVipStatsHelper 18/56 Test #18: KatranLbTest.testVipStatsHelper ...................... Passed 0.01 sec Start 19: KatranLbTest.testLruStatsHelper 19/56 Test #19: KatranLbTest.testLruStatsHelper ...................... Passed 0.01 sec Start 20: KatranLbTest.testLruMissStatsHelper 20/56 Test #20: KatranLbTest.testLruMissStatsHelper .................. Passed 0.01 sec Start 21: KatranLbTest.testHcHelpers 21/56 Test #21: KatranLbTest.testHcHelpers ........................... Passed 0.01 sec Start 22: KatranLbTest.getVipFlags 22/56 Test #22: KatranLbTest.getVipFlags ............................. Passed 0.01 sec Start 23: KatranLbTest.getAllVips 23/56 Test #23: KatranLbTest.getAllVips .............................. Passed 0.01 sec Start 24: KatranLbTest.testUpdateRealsHelper 24/56 Test #24: KatranLbTest.testUpdateRealsHelper ................... Passed 0.07 sec Start 25: KatranLbTest.testUpdateQuicRealsHelper 25/56 Test #25: KatranLbTest.testUpdateQuicRealsHelper ............... Passed 0.06 sec Start 26: KatranLbTest.testUpdateQuicReal 26/56 Test #26: KatranLbTest.testUpdateQuicReal ...................... Passed 0.01 sec Start 27: KatranLbTest.getRealsForVip 27/56 Test #27: KatranLbTest.getRealsForVip .......................... Passed 0.01 sec Start 28: KatranLbTest.getHealthcheckersDst 28/56 Test #28: KatranLbTest.getHealthcheckersDst .................... Passed 0.01 sec Start 29: KatranLbTest.invalidAddressHandling 29/56 Test #29: KatranLbTest.invalidAddressHandling .................. Passed 0.01 sec Start 30: KatranLbTest.addInvalidSrcRoutingRule 30/56 Test #30: KatranLbTest.addInvalidSrcRoutingRule ................ Passed 0.01 sec Start 31: KatranLbTest.addValidSrcRoutingRuleV4 31/56 Test #31: KatranLbTest.addValidSrcRoutingRuleV4 ................ Passed 0.01 sec Start 32: KatranLbTest.addValidSrcRoutingRuleV6 32/56 Test #32: KatranLbTest.addValidSrcRoutingRuleV6 ................ Passed 0.01 sec Start 33: KatranLbTest.addMaxSrcRules 33/56 Test #33: KatranLbTest.addMaxSrcRules .......................... Passed 0.01 sec Start 34: KatranLbTest.delSrcRules 34/56 Test #34: KatranLbTest.delSrcRules ............................. Passed 0.01 sec Start 35: KatranLbTest.clearSrcRules 35/56 Test #35: KatranLbTest.clearSrcRules ........................... Passed 0.01 sec Start 36: KatranLbTest.addFewInvalidNets 36/56 Test #36: KatranLbTest.addFewInvalidNets ....................... Passed 0.01 sec Start 37: KatranLbTest.addInvalidDecapDst 37/56 Test #37: KatranLbTest.addInvalidDecapDst ...................... Passed 0.01 sec Start 38: KatranLbTest.addInvalidDecapDstNet 38/56 Test #38: KatranLbTest.addInvalidDecapDstNet ................... Passed 0.01 sec Start 39: KatranLbTest.addValidDecapDst 39/56 Test #39: KatranLbTest.addValidDecapDst ........................ Passed 0.01 sec Start 40: KatranLbTest.delValidDecapDst 40/56 Test #40: KatranLbTest.delValidDecapDst ........................ Passed 0.01 sec Start 41: KatranLbTest.delInvalidDecapDst 41/56 Test #41: KatranLbTest.delInvalidDecapDst ...................... Passed 0.01 sec Start 42: KatranLbTest.addMaxDecapDst 42/56 Test #42: KatranLbTest.addMaxDecapDst .......................... Passed 0.01 sec Start 43: VipTestF.testBatchUpdateReals 43/56 Test #43: VipTestF.testBatchUpdateReals ........................ Passed 0.04 sec Start 44: VipTestF.testBatchUpdateRealsWeight 44/56 Test #44: VipTestF.testBatchUpdateRealsWeight .................. Passed 0.05 sec Start 45: VipTestF.testGetRealsAndWeight 45/56 Test #45: VipTestF.testGetRealsAndWeight ....................... Passed 0.01 sec Start 46: VipTestF.testGetReals 46/56 Test #46: VipTestF.testGetReals ................................ Passed 0.02 sec Start 47: VipTest.testAddRemoveReal 47/56 Test #47: VipTest.testAddRemoveReal ............................ Passed 0.01 sec Start 48: EventPipeCallbackTest.SimpleCallbackTest 48/56 Test #48: EventPipeCallbackTest.SimpleCallbackTest ............. Passed 0.01 sec Start 49: EventPipeCallbackTest.LargeWriteTest 49/56 Test #49: EventPipeCallbackTest.LargeWriteTest ................. Passed 0.15 sec Start 50: TestMonitoringServiceCore.SimpleAcceptSubscription 50/56 Test #50: TestMonitoringServiceCore.SimpleAcceptSubscription ... Passed 0.01 sec Start 51: TestMonitoringServiceCore.SimpleErrors 51/56 Test #51: TestMonitoringServiceCore.SimpleErrors ............... Passed 0.01 sec Start 52: TestMonitoringServiceCore.EventIntersection 52/56 Test #52: TestMonitoringServiceCore.EventIntersection .......... Passed 0.01 sec Start 53: TestMonitoringServiceCore.RacingClients 53/56 Test #53: TestMonitoringServiceCore.RacingClients .............. Passed 0.01 sec Start 54: TestMonitoringServiceCore.SubscribeAndCancel 54/56 Test #54: TestMonitoringServiceCore.SubscribeAndCancel ......... Passed 0.01 sec Start 55: PcapWriterTest.SingleWriter 55/56 Test #55: PcapWriterTest.SingleWriter .......................... Passed 0.02 sec Start 56: PcapWriterTest.MultiWriter 56/56 Test #56: PcapWriterTest.MultiWriter ........................... Passed 0.01 sec 100% tests passed, 0 tests failed out of 56 Total Test time (real) = 1.04 sec + cd ../testing/ + ctest -v ./CMakeFiles ./CTestTestfile.cmake ./Makefile ./base64helpers-tests './base64helpers-tests[1]_include.cmake' './base64helpers-tests[1]_tests.cmake' ./cmake_install.cmake ./katran_tester ./libbase64_helpers.a ./libbpftester.a ./libkatran_test_provision.a ./libkatran_test_util.a ./libpcap_parser.a ctest: /usr/local/lib/libcurl.so.4: no version information available (required by ctest) Test project /home/ubuntu/ivanmorett/katran/_build/build/katran/lib/testing Start 1: Base64Tests.testEncode 1/2 Test #1: Base64Tests.testEncode ........... Passed 0.01 sec Start 2: Base64Tests.testDecode 2/2 Test #2: Base64Tests.testDecode ........... Passed 0.01 sec 100% tests passed, 0 tests failed out of 2 Total Test time (real) = 0.01 sec + popd ~/ivanmorett/katran/_build ``` Differential Revision: D55108012 Pulled By: lima1756
lima1756
added a commit
that referenced
this issue
Mar 20, 2024
Summary: After diving in all the build system I found that the first error mentioned in #219 and #220 ``` 1597 | static_assert(formattable_char, "Mixing character types is disallowed."); ``` Was basically happening while compiling folly, after compiling it by itself I noticed this didn't happened, so I found that there was an issue with the fmt dependencies, removing the one that was downloaded by katran The issue mentioned in: #221 Was because katran was configured to use C++14 by default, and some of the libraries of folly require C++17, updated our requirements. Test Plan: TEST Output: ``` Test project /home/ubuntu/ivanmorett/katran/_build/build/katran/lib/tests Start 1: IpHelpersTests.testV4ParsingBe 1/56 Test #1: IpHelpersTests.testV4ParsingBe ....................... Passed 0.01 sec Start 2: IpHelpersTests.testV4ParsingInt 2/56 Test #2: IpHelpersTests.testV4ParsingInt ...................... Passed 0.01 sec Start 3: IpHelpersTests.testV6ParsingBe 3/56 Test #3: IpHelpersTests.testV6ParsingBe ....................... Passed 0.01 sec Start 4: IpHelpersTests.testV6ParsingInt 4/56 Test #4: IpHelpersTests.testV6ParsingInt ...................... Passed 0.01 sec Start 5: IpHelpersTests.testIncorrectAddr 5/56 Test #5: IpHelpersTests.testIncorrectAddr ..................... Passed 0.01 sec Start 6: CHHelpersTest.testMaglevCHSameWeight 6/56 Test #6: CHHelpersTest.testMaglevCHSameWeight ................. Passed 0.01 sec Start 7: CHHelpersTest.testMaglevV2CHSameWeight 7/56 Test #7: CHHelpersTest.testMaglevV2CHSameWeight ............... Passed 0.01 sec Start 8: CHHelpersTest.testMaglevCHDiffWeight 8/56 Test #8: CHHelpersTest.testMaglevCHDiffWeight ................. Passed 0.01 sec Start 9: CHHelpersTest.testMaglevV2CHDiffWeight 9/56 Test #9: CHHelpersTest.testMaglevV2CHDiffWeight ............... Passed 0.01 sec Start 10: CHHelpersTest.testMaglevWeightsSumLargerThanRing 10/56 Test #10: CHHelpersTest.testMaglevWeightsSumLargerThanRing ..... Passed 0.01 sec Start 11: CHHelpersTest.testMaglevWeightsSumBelowRingSize 11/56 Test #11: CHHelpersTest.testMaglevWeightsSumBelowRingSize ...... Passed 0.01 sec Start 12: KatranLbTest.testChangeMac 12/56 Test #12: KatranLbTest.testChangeMac ........................... Passed 0.01 sec Start 13: KatranLbTest.testIfIndex 13/56 Test #13: KatranLbTest.testIfIndex ............................. Passed 0.01 sec Start 14: KatranLbTest.testVipHelpers 14/56 Test #14: KatranLbTest.testVipHelpers .......................... Passed 0.14 sec Start 15: KatranLbTest.testAddingInvalidVip 15/56 Test #15: KatranLbTest.testAddingInvalidVip .................... Passed 0.01 sec Start 16: KatranLbTest.testRealHelpers 16/56 Test #16: KatranLbTest.testRealHelpers ......................... Passed 0.01 sec Start 17: KatranLbTest.testRealFlags 17/56 Test #17: KatranLbTest.testRealFlags ........................... Passed 0.01 sec Start 18: KatranLbTest.testVipStatsHelper 18/56 Test #18: KatranLbTest.testVipStatsHelper ...................... Passed 0.01 sec Start 19: KatranLbTest.testLruStatsHelper 19/56 Test #19: KatranLbTest.testLruStatsHelper ...................... Passed 0.01 sec Start 20: KatranLbTest.testLruMissStatsHelper 20/56 Test #20: KatranLbTest.testLruMissStatsHelper .................. Passed 0.01 sec Start 21: KatranLbTest.testHcHelpers 21/56 Test #21: KatranLbTest.testHcHelpers ........................... Passed 0.01 sec Start 22: KatranLbTest.getVipFlags 22/56 Test #22: KatranLbTest.getVipFlags ............................. Passed 0.01 sec Start 23: KatranLbTest.getAllVips 23/56 Test #23: KatranLbTest.getAllVips .............................. Passed 0.01 sec Start 24: KatranLbTest.testUpdateRealsHelper 24/56 Test #24: KatranLbTest.testUpdateRealsHelper ................... Passed 0.07 sec Start 25: KatranLbTest.testUpdateQuicRealsHelper 25/56 Test #25: KatranLbTest.testUpdateQuicRealsHelper ............... Passed 0.06 sec Start 26: KatranLbTest.testUpdateQuicReal 26/56 Test #26: KatranLbTest.testUpdateQuicReal ...................... Passed 0.01 sec Start 27: KatranLbTest.getRealsForVip 27/56 Test #27: KatranLbTest.getRealsForVip .......................... Passed 0.01 sec Start 28: KatranLbTest.getHealthcheckersDst 28/56 Test #28: KatranLbTest.getHealthcheckersDst .................... Passed 0.01 sec Start 29: KatranLbTest.invalidAddressHandling 29/56 Test #29: KatranLbTest.invalidAddressHandling .................. Passed 0.01 sec Start 30: KatranLbTest.addInvalidSrcRoutingRule 30/56 Test #30: KatranLbTest.addInvalidSrcRoutingRule ................ Passed 0.01 sec Start 31: KatranLbTest.addValidSrcRoutingRuleV4 31/56 Test #31: KatranLbTest.addValidSrcRoutingRuleV4 ................ Passed 0.01 sec Start 32: KatranLbTest.addValidSrcRoutingRuleV6 32/56 Test #32: KatranLbTest.addValidSrcRoutingRuleV6 ................ Passed 0.01 sec Start 33: KatranLbTest.addMaxSrcRules 33/56 Test #33: KatranLbTest.addMaxSrcRules .......................... Passed 0.01 sec Start 34: KatranLbTest.delSrcRules 34/56 Test #34: KatranLbTest.delSrcRules ............................. Passed 0.01 sec Start 35: KatranLbTest.clearSrcRules 35/56 Test #35: KatranLbTest.clearSrcRules ........................... Passed 0.01 sec Start 36: KatranLbTest.addFewInvalidNets 36/56 Test #36: KatranLbTest.addFewInvalidNets ....................... Passed 0.01 sec Start 37: KatranLbTest.addInvalidDecapDst 37/56 Test #37: KatranLbTest.addInvalidDecapDst ...................... Passed 0.01 sec Start 38: KatranLbTest.addInvalidDecapDstNet 38/56 Test #38: KatranLbTest.addInvalidDecapDstNet ................... Passed 0.01 sec Start 39: KatranLbTest.addValidDecapDst 39/56 Test #39: KatranLbTest.addValidDecapDst ........................ Passed 0.01 sec Start 40: KatranLbTest.delValidDecapDst 40/56 Test #40: KatranLbTest.delValidDecapDst ........................ Passed 0.01 sec Start 41: KatranLbTest.delInvalidDecapDst 41/56 Test #41: KatranLbTest.delInvalidDecapDst ...................... Passed 0.01 sec Start 42: KatranLbTest.addMaxDecapDst 42/56 Test #42: KatranLbTest.addMaxDecapDst .......................... Passed 0.01 sec Start 43: VipTestF.testBatchUpdateReals 43/56 Test #43: VipTestF.testBatchUpdateReals ........................ Passed 0.04 sec Start 44: VipTestF.testBatchUpdateRealsWeight 44/56 Test #44: VipTestF.testBatchUpdateRealsWeight .................. Passed 0.05 sec Start 45: VipTestF.testGetRealsAndWeight 45/56 Test #45: VipTestF.testGetRealsAndWeight ....................... Passed 0.01 sec Start 46: VipTestF.testGetReals 46/56 Test #46: VipTestF.testGetReals ................................ Passed 0.02 sec Start 47: VipTest.testAddRemoveReal 47/56 Test #47: VipTest.testAddRemoveReal ............................ Passed 0.01 sec Start 48: EventPipeCallbackTest.SimpleCallbackTest 48/56 Test #48: EventPipeCallbackTest.SimpleCallbackTest ............. Passed 0.01 sec Start 49: EventPipeCallbackTest.LargeWriteTest 49/56 Test #49: EventPipeCallbackTest.LargeWriteTest ................. Passed 0.15 sec Start 50: TestMonitoringServiceCore.SimpleAcceptSubscription 50/56 Test #50: TestMonitoringServiceCore.SimpleAcceptSubscription ... Passed 0.01 sec Start 51: TestMonitoringServiceCore.SimpleErrors 51/56 Test #51: TestMonitoringServiceCore.SimpleErrors ............... Passed 0.01 sec Start 52: TestMonitoringServiceCore.EventIntersection 52/56 Test #52: TestMonitoringServiceCore.EventIntersection .......... Passed 0.01 sec Start 53: TestMonitoringServiceCore.RacingClients 53/56 Test #53: TestMonitoringServiceCore.RacingClients .............. Passed 0.01 sec Start 54: TestMonitoringServiceCore.SubscribeAndCancel 54/56 Test #54: TestMonitoringServiceCore.SubscribeAndCancel ......... Passed 0.01 sec Start 55: PcapWriterTest.SingleWriter 55/56 Test #55: PcapWriterTest.SingleWriter .......................... Passed 0.02 sec Start 56: PcapWriterTest.MultiWriter 56/56 Test #56: PcapWriterTest.MultiWriter ........................... Passed 0.01 sec 100% tests passed, 0 tests failed out of 56 Total Test time (real) = 1.04 sec + cd ../testing/ + ctest -v ./CMakeFiles ./CTestTestfile.cmake ./Makefile ./base64helpers-tests './base64helpers-tests[1]_include.cmake' './base64helpers-tests[1]_tests.cmake' ./cmake_install.cmake ./katran_tester ./libbase64_helpers.a ./libbpftester.a ./libkatran_test_provision.a ./libkatran_test_util.a ./libpcap_parser.a ctest: /usr/local/lib/libcurl.so.4: no version information available (required by ctest) Test project /home/ubuntu/ivanmorett/katran/_build/build/katran/lib/testing Start 1: Base64Tests.testEncode 1/2 Test #1: Base64Tests.testEncode ........... Passed 0.01 sec Start 2: Base64Tests.testDecode 2/2 Test #2: Base64Tests.testDecode ........... Passed 0.01 sec 100% tests passed, 0 tests failed out of 2 Total Test time (real) = 0.01 sec + popd ~/ivanmorett/katran/_build ``` Differential Revision: D55108012 Pulled By: lima1756
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi， my config is similar to this one about healthcheck while for some reason in our cvm environment, we modified balance_kern.c, DSR is replaced by TUNNEL and return packet will go through LB.
now packet forwarding is well but health check is not. Actually i can capture hc return packet with tcpdump, but the hc process do not get the response.
configure in my environment
LB（katran）:
RS:
configured with tunnel back
-eth0 172.33.35.7 mark 10000
health check program:
bind socket with 172.33.4.100 and send SYNC to 172.33.4.78(vip) with mark 10000.
for test, i modified balancer_kern, when recving IPIP pacekt, directly return XDP_PASS to kernel.
but my hc program do not recv ACK while i can capture ACK packet in LB with tcpdump.
strangely when i ifconfig down the vip in LB, hc program can recv ACK.
The text was updated successfully, but these errors were encountered: