Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proxify is missing some encodings #519

Open
ehsandeep opened this issue Mar 13, 2024 · 5 comments · Fixed by #523
Open

proxify is missing some encodings #519

ehsandeep opened this issue Mar 13, 2024 · 5 comments · Fixed by #523
Assignees
Labels
investigation investigation Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@ehsandeep
Copy link
Member

seems like proxify is missing some encodings ( + those are shown in jsonl but not yaml multidoc )
Screenshot 2024-03-12 at 9 58 34 PM

Steps to reproduce

  • proxify traffic ( yaml or/and jsonl )
  • visit proxify github repo
  • some body encoding are still encoded while others are not

Implementation Notes

Originally posted by @tarunKoyalwar in #438 (comment)

@ehsandeep ehsandeep added the Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Mar 13, 2024
@dogancanbakir dogancanbakir self-assigned this Mar 13, 2024
@dogancanbakir dogancanbakir added the investigation investigation label Mar 13, 2024
@dogancanbakir
Copy link
Member

I'm unable to reproduce the issue. Here are the steps I followed:

$ go run . -of jsonl -sr

$ python3 -c 'import requests; proxies = {"http": "http://127.0.0.1:8888", "https": "http://127.0.0.1:8888"}; response = requests.get("https://github.com/projectdiscovery/proxify", proxies=proxies, verify=False); print(response.text)'

I've also tried browser but couldn't.

@tarunKoyalwar Could you please provide me with the repro steps?

@tarunKoyalwar
Copy link
Member

tarunKoyalwar commented Mar 13, 2024

looks like there is another issue related to max response read size

$  proxify -v

                       _ ___    
   ___  _______ __ __ (_) _/_ __
  / _ \/ __/ _ \\ \ // / _/ // /
 / .__/_/  \___/_\_\/_/_/ \_, / 
/_/                      /___/

		projectdiscovery.io

[INF] Current proxify version v0.0.15 (latest)
[INF] HTTP Proxy Listening on 127.0.0.1:8888
[INF] Saving proxify logs to proxify_logs.jsonl
[WRN] responseChain: Error while dumping response: error reading response body: could not read response body: http: request body too large
$ curl -v -x http://127.0.0.1:8888 https://github.githubassets.com/asses/chunk-ui_packages_cookie-consent-link-element_cookie-consent-link-element_ts-80fd8ee00406.js
  • we can use io.MultiReader along with ResponseChain, so that processing limit of proxify is 4 MB but whether to read all bytes or not depends on client and not proxify

cc: @dogancanbakir

@tarunKoyalwar
Copy link
Member

also looks like some servers do send br encoded response body and this seems to be advertisied by firefox browser

      "Accept-Encoding": "gzip, deflate, br",

^ this seems to be the reason why this is reproducable when proxiying browser traffic and not when using curl

Screenshot 2024-03-13 at 5 35 38 PM

@tarunKoyalwar
Copy link
Member

  • can confirm issue is due to missing br encoding in proxify
"response": {
    "header": {
      "Accept-Ranges": "bytes",
      "Access-Control-Allow-Origin": "*",
      "Age": "1196786",
      "Cache-Control": "public, max-age=31536000, immutable",
      "Content-Encoding": "br",
      "Content-Type": "application/javascript",
      "Cross-Origin-Resource-Policy": "cross-origin",
      "Date": "Wed, 13 Mar 2024 12:08:32 GMT",
      "Etag": "0x8DC348B9980FBA0",
      "Last-Modified": "Fri, 23 Feb 2024 16:22:16 GMT",
      "Server": "Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0",
      "Strict-Transport-Security": "max-age=31536000",
      "Vary": "Accept-Encoding",
      "Via": "1.1 varnish, 1.1 varnish",
      "X-Cache": "HIT, HIT",
      "X-Cache-Hits": "88, 15269",
      "X-Fastly-Request-Id": "3c72b7382202b93fec00c5c1e1f06972ff1eb20c",
      "X-Served-By": "cache-iad-kiad7000175-IAD, cache-hyd1100027-HYD"
    },
    "body": " j����\r!�s1�]\u0015C&�B�9\u000f��cl�\u0010��a�D�ϩ�\u0018\u0002�D?(\u0010VK)3�;e�R�\u000b�:fZ\u0019Cz&�s?�Xs��[��bB��٘J�\u001d¦���|G��ݩ\u0012�ѩ\u001f�?�������\u0013fl/�\u001fE�ƣ,��ށ3\r�/f^�\u0015��V�\u0004�SϬ��\b�d��n\u000511H�\u0003\t2w\u000b�\u0016Iw=�Q�\\�D\u0010�*?r�쎲�\u0007���R\u001e\u0015\u0019\u001b�]�\u0005ƨm\u0019ļ��Ggq\u0017�q��6�\u0007wA|�_7��{��6fߒX�u/k\b3�|a��\u0012r�\u001b��9h#ю�\u0006���H}R��dIo��,\u001a��\\��\u001f�\u0003�\u0001��)�D17\u0000(�\u0017fɚ����Y\u0011*��/�[�\u007f�D�e�\u001a����[eY\u001a�\f��d��!�8�:���\u0005\u007f���>b\rHz�h�J����t�jcN\u0019p��xA�ns\u001c_pr\\�\"��\"oR\u0010O���)f��[�b\u001a%�M��^z��t\u0018�\u001e�\u001e��\u0018�IO\u0013%ԧZ�V�x��K��:L�.K �p\u0013��-�Km�۵Ph\u0013�/q\u001b�\u0006������1\u0007-4E\u0016�\u001b�lі��Q�b�<�\b�\u0000��Ҥ���z\u0006��$�EVa��9\u0011(ʭ\u001a\u0017�\r\u0018�\b���\u0002Ơ� \u0011\u0019\b\u0004�$d���9�\f�[���*7�OT|��F�ꈵW�v\u00048��Չ�⢾�3�\u0019<n\u0014cb�3+�B���\u0001���8��r�&(�*\u0018�?�\f\u001a���A�/\u0018\u001a���\u001a�\u0006ђ_c���IR\u0019]������N����\u001dc�V\u0007�\u0019������wo�)r'�\u0018�1[��b\bI�����\u0013h���\n�a���+�G���A�\b�l\u000e)%|\u000f>�\u000b�i��\"t�#�6\u0006����\u0002O��i\u001b(t�}�Z�\u0017D��\u001e��Wя�\nB\u0000\u0011s]|���v�Ui�Ԫ�\u001a�r�]�^\u0010�ҼiG����v\u001e� \u0010�w�{7y\u0012\u0011��5{(1�y�\n�m�X�Z\n����\u001e��ď�ӟz�\u0004Qmt\u0002�aa�y�1Fo���\u001ds�1\u0010Y॔\u0019�A�<K\u0006C\u0018�:\u0003\u00057�u\u0016���������+1N\u000fY�C0�?\u007f�����\u000ejR�|<\u0002�\u001au�j\u000e�+,`�[��e`I���JS T4\u00027�5}��m+ �mkLg�5ᚕI<\u0006~�7����ӓ���<\u0016׿�Hh����\u0015��M���H���R��{��\r6\u0012���KU���`{�\u000f�vncы\b\u001e\u001b'<(�\u0003��\u0018\u0003�\u0019 �\u001ct�������x\u0006�I&\u0005�n��IQ?� \u0002O��cb\u001e�D�/\u0017s\u0002Jb�\u007f\u0000�Ě{��K�yx\u0018\u000f\u000f���1jI���\u0000�y\u0000f���d\u0018\u0014W\u0013E\u0010��i���v�hQa�7p.�B`�[Z?�Y�\u0010\u001dJ�<��I��\u001b����Z\u0012)c\u0010]T�ഖ�\u001d\u000bpE\u0004�w���L�y�\u0012\u001a��� \u0002�X\b��Y�ʷ����X8�T�-8;g!�7K.\n̈m�\r\u000b��2�\u0006�Q\rd�\u0005����\u0004�>�9��b\rY\u001b��=��}�:�\u0010����~�l �RF��cN}�����\u0013��р˄0�]�\u0005��o/��n���]\f�\fV�psq%R\u0006�kx���\u000f���������_�J�\u0018|��Q��Ao���\u0000\u0019\u0005.�1�ڄNZ���\b�$������\u001d2���E玝�\u00057tH��\u007fh%��\u00064c}��u\"�N%\u007fd�8܈\u001c�i���\u0018�7ֈ�$��X�F\u0007\u00181dZ\u0014�œ\u0019���(c]�\\q���@J{x�.K\u000e�=/�؛P7�r�!��\u0016�_dp\u0015L�Ę��x�W��0�Gz��\u0004\u0004\u0003�F�g����ZT��\n��zC\u001e<\u001b�L�PA�\u0016��\u001d�\u0005\f=����\u0001LA3�$͂\u0000(�BG��4�\u001f��u�u�g�U:Q�\u0013�\n�H�cN�A%?��jI?�\u0014��\u0012y�z�8?<�ߗ\u0002�<٘�f{=l��ݦq����c!s�\u0012�˘�ǜ\"Ϩ\u001f���ja\n�`\u000b9�0��?1�6��.���s�\u0001��u�S��7t04\tv�\u001e5�ن�<?�\u000f0�������\u0011�Ay��QdPd����xʷ��IY�H\u0007�30�>�\u0001K�!��f̍�\r?�\u000eo��c��j\u000f�b\u0000�\u000b\u001e\u0018��\"�\u001e���7�\rT�3[�����3����hI�ˏ<��?�y��)8{�h2�Zt\u000f�WZ��\u0003�j�_x\r����nLK�\u001bW@�&KU�P�j[V��+�!`\\\\���f��\u0007�1N�^�=�̱\u0010o���\u0011,9��LM�4�,����:`�&����\u0000HT��7���\\H*)������VHu�\u001f��;\u0005P��b���Op�'GC�s\\T�M�\t0'HY7E�V\u0006��}6\u001a?-|���K\t�ֲ�/0���ÅN��ˆ<��ҡj�\no\u001a7��|̩4\u0010�{ֽ)�<<�\b�����\u0016���q��^�����\u000b�B�67�MW?�\u0006���Ww�\\郯��ѿw1�Z�a�M�3:'pQ��\u001f�����]G�\u0004�u\u0011ܛ8\u000b�x\u0019��4k1�f+FY$9\u0013��\fmDO�ۄќ?B�� �)I�\f\u00181�B\u0005�\t!.ԣ\u000fr2\u001e�{����jvx�p%'���1Y<\u000b�\u0015a������@����׏�V\u0005MJ�ֲ�\u0003\b}-� ���S���\u0005�\u0012.j�v��e���e���V\u0014�xL�m\u0013Qx��\u001dt�\u0007��H�%]Vk�\"\u001e\u000b1'(���M\u0014#h�bvx�J)�\u0003�r�Th�V�\u001bJ��؄�ul��h\r��v��\"6Қ��mWo��\b��V��\b�*�ꍨH�u\u0016I�\u0017�D��$)��~��WK)��k�O3��3\u0011L��$|,J\u0007L���g?�g�\u0007�7S\u0010��$��.c1��\u0010���\u000b)e9��\u0010\u0010�a7����ܐ}.\u0018\u0014��[��,��\u0001:���\fL~n\t_Q�Uj��\u001c_pТ��Y+_j�<\t�\u001c�*{-�km�Y\f�\u0014�ۙ�A���D���c�o\u0014�\u0000�Fm&��� ���l��a6\u001eͷd\u0010�ƾB�����\u0012r\u001b�l\u000f��\u0012x%�M3���M8��������S\u0006[g<�����V�S���� �ߩ(�\u0011\u0010���\tL*�������F{+s݂?BgJ\u0013Qß��Ȝ��K�Ā�z�QcT��39�b�����j�'*����\u0013\u0011[{�?zf(��[���l�*��m�F-�ϭl;*_b����!*��+��姇���y2W�ᡧ��\\ch��$�'����V�O&&�h2�8\u0015h�Q��\u001d� ;�\u0011\u0004�b�<g�29�p\"��\u0006\u0007\u00039[��@z���2�����u\u0010�0\u007f<w�|v���\u0010]\u0012*�!C��r���Hjr�7�@\t�%y&�AJ�W\u0011�\u001aC/W3���U�4�Z\u0019z6�q�M���h�H�5��V���M��I�7 ~N��\u0002�9H�d\u001fn�5��\fp�� /I�Yc\u0002Y��Bع��\u0006Y)Oq.\u0005��,^���ȸ�:\u0016K\u0017��\u001c�yx��\u0007Z�\u001f%\u0010�,M]dž���f'H��sL�͙j�\u0007�_��V��w��:B���w�AQa�~O�\u0002�\u0010@��ú\u0006�ϔ���4m�Q� �\u0011���X,q\u001bM�����|�/凎�\u001c��I�\b�\u0018�)�L�@��\n��]���M�J��\u007f�ؒs,B�A16�\u001b�\u001b\u0015�\u0019�X�\u001fj0&��\u0015Tќ���\\��W�\u0016�U�\u0003(�\u0006\u007fb��#>\u0015\u000bz�\u0015O���i\u0011ι���\u0001�9\u0016I�5H4ر�\u00114!\u0018s4���(��ss���3g�w&�{�\u0018H���ó�\u0005���\u0018\u0014f�BȺ��.�8W�n�H]��\u0003\u0016\u0017%sC\u0011�&;Dz�O\u0019U�\u0007�8�\u0002��(�!������E\u000e�����y�D��\u0002\u007f<�.�q\u0012+2q��\u0013�bb��\u0013\u0002\u001b�a09\"�~��/���~����.��\u0006�j\u001dH\u001f\t\u001f�?C�ًz�T��\u0013�������$��g�I5������l�o2��h�����Eq��c�;�Z5"

@dogancanbakir
Copy link
Member

depends on projectdiscovery/utils#370

@dogancanbakir dogancanbakir linked a pull request Mar 13, 2024 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
investigation investigation Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants