all APIs erroring on POSTS to ElasticSearch #445

animaxcg · 2020-06-17T23:28:05Z

Micro services

Spring boot Applications:

Webapp

2.0

Setup

public ALB with public subnets (subnet has route table attachment with igw attached

Summary

all APIs erroring on POSTS to ElasticSearch

Reproduce steps

Deploy with public ALB login. no assest found (when looking at assets/volnerabillits cloudwatch logs all have stack traces when trying to do a http post to elasticsearch

Expected Results

The api's to work

Actual Results

assets cloudwatch logs:
2020-06-17T17:59:52.242-05:00
com.zaxxer.hikari.HikariDataSource.getConnection - HikariPool-1 - Start completed.
17-06-2020 23:00:07.536 [http-nio-80-exec-1] ERROR
c.t.p.api.commons.utils.PacHttpUtils.doHttpPost - URL: http://somedomain.us-east-1.es.amazonaws.com:80/aws/_search/?size=0 Body:
{
"size": "0",
"query": {
"bool": {
"must": [
{
"term": {
"latest": "true"
}
},
{
"term": {
"_entity": "true"
}
}
]
}
},
"aggs": {
"name": {
"terms": {
"field": "_type",
"size": 1000
}
}
}
}
17-06-2020 23:00:07.538 [http-nio-80-exec-1] ERROR

<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 5.6-c142 79.160924, 2017/07/13-01:06:39 ">
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<rdf:Description rdf:about=""/>
</rdf:RDF>
</x:xmpmeta>

  .cls-1, .cls-2, .cls-3, .cls-4 {
    fill: #fff;
    font-family: "Titillium Web";
  }

  .cls-1, .cls-2, .cls-3 {
    font-weight: 700;
  }

  .cls-2 {
    font-size: 50px;
  }

  .cls-3 {
    font-size: 100px;
  }

  .cls-4 {
    font-size: 36px;
    font-weight: 300;
  }
</style>

Pacbot Titillium Aa ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwzyz&£$%[]!?*º c.t.p.api.commons.utils.PacHttpUtils.doHttpPost - URL: http://somedomain.us-east-1.es.amazonaws.com:80/aws/_search/?size=0 Body: { "size": "0", "query": { "bool": { "must": [ { "term": { "latest": "true" } }, { "term": { "_entity": "true" } }, { "terms": { "_type": [ "ec2", "s3", "appelb", "asg", "classicelb", "stack", "dynamodb", "efs", "emr", "lambda", "nat", "eni", "rdscluster", "rdsdb", "redshift", "sg", "snapshot", "subnet", "targetgroup", "volume", "vpc", "api", "iamuser", "iamrole", "rdssnapshot", "account", "checks", "kms", "phd", "cloudfront", "elasticip", "routetable", "internetgateway", "launchconfig", "networkacl", "vpngateway", "asgpolicy", "snstopic", "dhcpoption", "peeringconnection", "customergateway", "vpnconnection", "directconnect", "virtualinterface", "elasticsearch", "elasticache" ] } } ] } }, "aggs": { "name": { "terms": { "field": "tags.Application.keyword", "size": 10000 }, "aggs": { "providers": { "terms": { "field": "_cloudType.keyword", "size": 10000 } } } } } } 17-06-2020 23:00:07.543 [http-nio-80-exec-3] ERROR 17-06-2020 23:00:07.543 [http-nio-80-exec-3] ERROR 17-06-2020 23:00:07.545 [http-nio-80-exec-3] ERROR 17-06-2020 23:00:07.545 [http-nio-80-exec-3] ERROR

c.t.p.a.a.r.AssetRepositoryImpl.getDistributionDataFromES - error retrieving inventory from ES
java.lang.Exception: unable to execute post request to http://somedomain.us-east-1.es.amazonaws.com:80/aws/_search/?size=0 because Not Found
at com.tmobile.pacman.api.commons.utils.PacHttpUtils.doHttpPost(PacHttpUtils.java:93)
at com.tmobile.pacman.api.asset.repository.AssetRepositoryImpl.getDistributionDataFromES(AssetRepositoryImpl.java:354)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.repository.AssetRepositoryImpl.getTotalDistributionForIndexAndTypeWithProviders(AssetRepositoryImpl.java:281)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.repository.AssetRepositoryImpl.getApplicationAssetCountByAssetGroupWithProvider(AssetRepositoryImpl.java:249)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.repository.AssetRepositoryImpl.getApplicationAssetCountByAssetGroup(AssetRepositoryImpl.java:225)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.repository.AssetRepositoryImpl.getApplicationByAssetGroup(AssetRepositoryImpl.java:215)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.repository.AssetRepositoryImpl$$FastClassBySpringCGLIB$$26e07b49.invoke()

2020-06-17T18:00:07.546-05:00
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

2020-06-17T18:00:07.546-05:00
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.repository.AssetRepositoryImpl$$EnhancerBySpringCGLIB$$587823aa.getApplicationByAssetGroup()

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.service.AssetServiceImpl.getAssetGroupInfo(AssetServiceImpl.java:206)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.service.AssetServiceImpl$$FastClassBySpringCGLIB$$74f7e251.invoke()

2020-06-17T18:00:07.546-05:00
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.service.AssetServiceImpl$$EnhancerBySpringCGLIB$$7829db24.getAssetGroupInfo()

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.controller.AssetController.getAssetGroupInfo(AssetController.java:169)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.controller.AssetController$$FastClassBySpringCGLIB$$b72bbe2d.invoke()

2020-06-17T18:00:07.546-05:00
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

2020-06-17T18:00:07.546-05:00
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

2020-06-17T18:00:07.546-05:00
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

2020-06-17T18:00:07.546-05:00
at com.tmobile.pacman.api.asset.controller.AssetController$$EnhancerBySpringCGLIB$$d2034a6d.getAssetGroupInfo()

2020-06-17T18:00:07.546-05:00
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2020-06-17T18:00:07.546-05:00
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

2020-06-17T18:00:07.546-05:00
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2020-06-17T18:00:07.546-05:00
at java.lang.reflect.Method.invoke(Method.java:498)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)

2020-06-17T18:00:07.546-05:00
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)

on the front end it shows 500s and 417s when calling API's for assets/vulnerabilities etc

I have tried also with 1.2 and 1.5 and 1.6 and get the same issue. is there something on the routing that is missing from the documentation
https://github.com/tmobile/pacbot/blob/master/installer/FAQs.md

The text was updated successfully, but these errors were encountered:

Akashbhaskaran · 2020-06-22T08:38:32Z

@animaxcg It seems like its not able to connect to the ElasticSearch endpoint. Could you please verify the es endpoint from the console and also check if you are able to reach the url . Also check the secuirty groups to make sure connections are enabled to ES .

As long as you have public alb , public subnets , NAT gateways and public ip assignment enabled, it should work .

animaxcg · 2020-06-27T15:58:20Z

when I do a curl from my computer I get:
curl http://esurl:80
{
"name" : "Mv6XIGF",
"cluster_name" : "000000000000:someprefix-data",
"cluster_uuid" : "0000-00000_00000",
"version" : {
"number" : "5.5.2",
"build_hash" : "0000",
"build_date" : "2019-04-05T17:18:30.396Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"
}

but when I do: curl http://esurl:80/aws/_search (what cloudwatch logs is showing error on).
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"aws","index_uuid":"na","index":"aws"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"aws","index_uuid":"na","index":"aws"},"status":404}

So even if I am missing something in networking between resources in the VPC it gives me the same error when looking at the path: url/aws/_search but no error on url.

animaxcg · 2020-06-27T22:52:31Z

Looks like it give 417 untill batch job runs... maybe should batch job run once as part of deployment so you don't have wait to up to 6 hours before the the assets are found?

thanks

Akashbhaskaran · 2020-06-28T02:48:52Z

You can either wait or change the cron for aws- data collector jobs in clouwatch. There by you schedule the data collection job right away

kaykumar assigned Akashbhaskaran Jun 18, 2020

varadh10 self-assigned this Jan 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

all APIs erroring on POSTS to ElasticSearch #445

all APIs erroring on POSTS to ElasticSearch #445

animaxcg commented Jun 17, 2020 •

edited

Loading

Akashbhaskaran commented Jun 22, 2020

animaxcg commented Jun 27, 2020

animaxcg commented Jun 27, 2020

Akashbhaskaran commented Jun 28, 2020

all APIs erroring on POSTS to ElasticSearch #445

all APIs erroring on POSTS to ElasticSearch #445

Comments

animaxcg commented Jun 17, 2020 • edited Loading

Micro services

Webapp

Setup

Summary

Reproduce steps

Expected Results

Actual Results

Akashbhaskaran commented Jun 22, 2020

animaxcg commented Jun 27, 2020

animaxcg commented Jun 27, 2020

Akashbhaskaran commented Jun 28, 2020

animaxcg commented Jun 17, 2020 •

edited

Loading