String encoding issue

[iahmad-khan]

Here is my monitor configuration:

resource "vtm_monitor" "tc-lit-solr_int" { 
 	 name = "tc-lit-solr_int" 
	 scope = "pernode"
	 type = "http"
	 back_off = true
	 delay = 3
	 failures = 3
	 health_only = false
	 timeout = 3
	 use_ssl = false
	 verbose = false
	 http_body_regex = "\"status\":\"OK\""
	 http_path = "/solr/citations/admin/ping?wt=json&distrib=true"
	 http_status_regex = "^[23][0-9][0-9]$"
	 rtsp_path = "/"
	 rtsp_status_regex = "^[234][0-9][0-9]$"
	 script_arguments = []
	 sip_status_regex = "^[234][0-9][0-9]$"
	 sip_transport = "udp"
	 tcp_response_regex = ".+"
	 tcp_max_response_len = 2048
	 udp_accept_all = false
}

when I push this through terraform , the http_path is not correct in the VTM , the resultant configuration done by terraform is with http_path:

 http_path = "/solr/citations/admin/ping?wt=jsonu0026distrib=true"

& is replaced by u0026 , how can i fix this,

thanks

[ndavidson-pulse]

That's weird - looks like a bug to me. @iahmad-khan can you check if this is a terraform issue by attempting to post the same body directly to the REST API?

[iahmad-khan]

@ndavidson-pulse currently i have fixed it by sending u0026 instead which is the correct & in traffic manager. I wonder why is it doing the conversion.

[pdabel]

I'm seeing the same issue on the ldap_group_filter property of the vtm_user_authenticator resource.

I was reading about this method https://golang.org/pkg/encoding/json/#HTMLEscape which is described as:

"String values encode as JSON strings coerced to valid UTF-8, replacing invalid bytes with the Unicode replacement rune. The angle brackets "<" and ">" are escaped to "\u003c" and "\u003e" to keep some browsers from misinterpreting JSON output as HTML. Ampersand "&" is also escaped to "\u0026" for the same reason. This escaping can be disabled using an Encoder that had SetEscapeHTML(false) called on it."

Does this need to be set in the go-vtm (https://github.com/pulse-vadc/go-vtm/blob/master/7.0/config_user_authenticator.go) code where the json is marshalled in the "Apply" method?

[ndavidson-pulse]

We need to properly escape the JSON in order for certain unicode code-points to work at all - so far your description makes it sound like some double encoding issue, I believe you've said the following two things:

• If you put http_path = "/solr/citations/admin/ping?wt=json&distrib=true the REST back-end gets http_path = "/solr/citations/admin/ping?wt=jsonu0026distrib=true and breaks.
• if you put http_path = "/solr/citations/admin/ping?wt=jsonu0026distrib=true the REST back-end works, but you've not described what it gets?
  Can you please try both requests directly to the REST API and report the behavior here.

[pdabel]

I tested the rest API directly using curl and json. The api seems to work as expected and is not encoding the "&".

Working:
{"properties":{"basic":{"description":"","enabled":true,"type":"ldap"},"ldap":{"base_dn":"cn=accounts,dc=example,dc=com","bind_dn":"","dn_method":"search","fallback_group":"admin","filter":"uid=%u","group_attribute":"member","group_field":"","group_filter":"(&(&(objectClass=posixgroup)(member=uid=%u,cn=users,cn=accounts,dc=example,dc=com))(cn=zxtm_access)","port":389,"search_dn":"","search_password":"","server":"localhost","timeout":30}}}

curl -vk -X PUT -H 'Content-Type: application/json' -d $(cat working) 'https://vtm.example.com:9070/api/tm/6.2/config/active/user_authenticators/working/' -u admin

Broken:
{"properties":{"basic":{"description":"","enabled":true,"type":"ldap"},"ldap":{"base_dn":"cn=accounts,dc=example,dc=com","bind_dn":"","dn_method":"search","fallback_group":"admin","filter":"uid=%u","group_attribute":"member","group_field":"","group_filter":"(u2600(u2600(objectClass=posixgroup)(member=uid=%u,cn=users,cn=accounts,dc=example,dc=com))(cn=zxtm_access)","port":389,"search_dn":"","search_password":"","server":"localhost","timeout":30}}}

curl -vk -X PUT -H 'Content-Type: application/json' -d $(cat broken) 'https://vtm.example.com:9070/api/tm/6.2/config/active/user_authenticators/broken/' -u admin

[ndavidson-pulse]

@pdabel does putting in the backslash work properly? e.g. does \u2600 work correctly?

[ravi-mrk]

We see this issue of string encoding with few more special characters that include the tags like: <, /, ", >.

[iahmad-khan]

Strange , have a look at this body regex:
http_body_regex = "<app-root></app-root>"

after terrafrom push itlooks this on traffic manager:

http_body_regex = "\u003capp-root\u003e\u003c/app-root\u003e"
and if I push this one from terrafrom to traffic manager:

http_body_regex = "\u003capp-root\u003e\u003c/app-root\u003e"

then it looks this on the problematic one ( corrected )
http_body_regex = "<app-root></app-root>"
but on the normal one , which is ok othervise , it is:
http_body_regex = "\u003capp-root\u003e\u003c/app-root\u003e"

so if we are going to fix one , the other is broken , both vtms are using same version 18.2

[ndavidson-pulse]

There are actually two related issues:

• The traffic manager REST API is not correctly translating the \uXXXX utf16 encoded character into the utf8 representation of the code-point just removing the back-slash escape. This has been assigned an internal ticket of VTM-42268
• The Terraform Provider is helpfully encoding HTML special characters as utf16 using the \u escaping syntax. This has been assigned an internal ticket of VTMTF-141.

These issues may be addressed in a future release. If you have a current support contract and have an urgent need for this issue, please contact us via https://support.pulsesecure.net to request prioritization.