From cfc2d327e7af261b61b46566aa9e8326d512374d Mon Sep 17 00:00:00 2001 From: Zhengxi Li Date: Fri, 28 Jul 2023 15:32:28 +0000 Subject: [PATCH 1/7] Deprecate cqtx log field --- configs/logging.yaml.default | 6 ++-- doc/admin-guide/files/logging.yaml.en.rst | 2 +- doc/admin-guide/logging/examples.en.rst | 34 +++++++++---------- doc/admin-guide/logging/formatting.en.rst | 17 +++++----- doc/appendices/faq.en.rst | 9 ----- .../monitoring/logging/log-formats.en.po | 29 ++++------------ .../ja/LC_MESSAGES/appendices/faq.en.po | 26 -------------- proxy/logging/Log.cc | 5 --- tests/gold_tests/connect/connect.test.py | 2 +- tests/gold_tests/h2/httpbin.test.py | 4 +-- 10 files changed, 38 insertions(+), 96 deletions(-) diff --git a/configs/logging.yaml.default b/configs/logging.yaml.default index 0e8aebb01b3..0612def3343 100644 --- a/configs/logging.yaml.default +++ b/configs/logging.yaml.default @@ -32,15 +32,15 @@ logging: # Common Log Format. - name: common - format: '% - % [%] "%" % %' + format: '% - % [%] "% % %" % %' # Extended Log Format. - name: 'extended' - format: '% - % [%] "%" % % % % % % % % % % %' + format: '% - % [%] "% % %" % % % % % % % % % % %' # Extended2 Log Formats - name: "extended2" - format: '% - % [%] "%" % % % % % % % % % % % % % % %' + format: '% - % [%] "% % %" % % % % % % % % % % % % % % %' logs: - filename: squid diff --git a/doc/admin-guide/files/logging.yaml.en.rst b/doc/admin-guide/files/logging.yaml.en.rst index 5c77515d09f..98bb7457d13 100644 --- a/doc/admin-guide/files/logging.yaml.en.rst +++ b/doc/admin-guide/files/logging.yaml.en.rst @@ -117,7 +117,7 @@ the logging output. Multiple logging fields may of course be used:: - '% % % %' + '% % % % % %' Each logging field is separately enclosed in its own percent-brace set. diff --git a/doc/admin-guide/logging/examples.en.rst b/doc/admin-guide/logging/examples.en.rst index 1671697a0c8..100f844622d 100644 --- a/doc/admin-guide/logging/examples.en.rst +++ b/doc/admin-guide/logging/examples.en.rst @@ -52,18 +52,18 @@ The following figure shows a sample log entry in a Netscape Common log file. The numbered sections correspond to the following log fields in |TS|: -=== ====== ==================================================================== -No. Field Description -=== ====== ==================================================================== -1 chi The IP address of the client's host machine. -2 -- This hyphen (``-``) is always present in Netscape log entries. -3 caun The authenticated client username. A hyphen (``-``) means no - authentication was required. -4 cqtn The date and time of the client request, enclosed in brackets. -5 cqtx The request line, enclosed in quotes. -6 pssc The proxy response status code (HTTP reply code). -7 pscl The length of the |TS| response to the client in bytes. -=== ====== ==================================================================== +=== ============= =============================================================== +No. Field Description +=== ============= =============================================================== +1 chi The IP address of the client's host machine. +2 -- This hyphen (``-``) is always present in Netscape log entries. +3 caun The authenticated client username. A hyphen (``-``) means no + authentication was required. +4 cqtn The date and time of the client request, enclosed in brackets. +5 cqhm pqu cqpv The request line, enclosed in quotes. +6 pssc The proxy response status code (HTTP reply code). +7 pscl The length of the |TS| response to the client in bytes. +=== ============= =============================================================== To recreate this as a log format in :file:`logging.yaml` you would define the following format object: @@ -72,7 +72,7 @@ following format object: formats: - name: common - format: '% - % [%] "%" % %' + format: '% - % [%] "% % %" % %' .. _admin-logging-examples-extended: @@ -119,7 +119,7 @@ following format object: formats: - name: extended - format: '% - % [%] "%" % % % % % % % % % % %' + format: '% - % [%] "% % %" % % % % % % % % % % %' .. _admin-logging-examples-extended2: @@ -158,7 +158,7 @@ following format object: formats: - name: extended2 - format: '% - % [%] "%" % % % % % % % % % % % % % % %' + format: '% - % [%] "% % %" % % % % % % % % % % % % % % %' .. _admin-logging-examples-squid: @@ -261,7 +261,7 @@ of the event data. .. code:: yaml ourformat = format { - Format = '% - % [%] "%" % %' + Format = '% - % [%] "% % %" % %' } log.binary { @@ -286,7 +286,7 @@ for them to a UNIX pipe that the alerting software can constantly read from. formats: - name: canaryformat - format: '% - % [%] "%" % %' + format: '% - % [%] "% % %" % %' filters: - name: canaryfilter diff --git a/doc/admin-guide/logging/formatting.en.rst b/doc/admin-guide/logging/formatting.en.rst index 76a4dc10674..6bfcd8c2051 100644 --- a/doc/admin-guide/logging/formatting.en.rst +++ b/doc/admin-guide/logging/formatting.en.rst @@ -584,19 +584,18 @@ sshv Origin Response Origin server's response HTTP version. Request Details ~~~~~~~~~~~~~~~ -.. _cqtx: - -The following logging fields are used to obtain the actual HTTP request -details. +The following logging fields can be used in combination to obtain the actual HTTP request details in a HTTP/1.1 request line format, e.g. ``GET http://www.company.com HTTP/1.0``. ===== ============== ========================================================== Field Source Description ===== ============== ========================================================== -cqtx Client Request The full HTTP client request text, minus headers, e.g. - ``GET http://www.company.com HTTP/1.0``. In reverse proxy - mode, |TS| logs rewritten/mapped URL (according to the - rules in :file:`remap.config`), not the pristine/unmapped - URL. +cqhm Client Request HTTP method used in the client request to the |TS| proxy + (e.g. ``GET``, ``POST``, etc.). +pqu Proxy Request URL of the proxy request from |TS| to the origin. If a + remap rule is used (:file:`remap.config`), the original + client URL is replaced with the URL in the target of the + remap rule. +cqpv Client Request Client request protocol and version. ===== ============== ========================================================== .. _admin-logging-fields-ssl: diff --git a/doc/appendices/faq.en.rst b/doc/appendices/faq.en.rst index f5e4cbbdded..2620fc9060a 100644 --- a/doc/appendices/faq.en.rst +++ b/doc/appendices/faq.en.rst @@ -61,15 +61,6 @@ In Squid- and Netscape-format log files, what do the cache result codes mean? This is described in detail in the :ref:`admin-logging-cache-results` documentation. -What is recorded by the ``cqtx`` field in a custom log file? ------------------------------------------------------------- - -- In *forward proxy mode*, the ``cqtx`` field records the complete client - request in the log file (for example, ``GET http://www.company.com HTTP/1.0``). -- In *reverse proxy mode*, the ``cqtx`` field records the hostname or IP - address of the origin server because Traffic Server first remaps the - request as per map rules in the :file:`remap.config` file. - Does Traffic Server refresh entries in its host database after a certain period of time if they have not been used? ------------------------------------------------------------------------------------------------------------------- diff --git a/doc/locale/ja/LC_MESSAGES/admin-guide/monitoring/logging/log-formats.en.po b/doc/locale/ja/LC_MESSAGES/admin-guide/monitoring/logging/log-formats.en.po index fa105ce87d0..cd72844ecbe 100644 --- a/doc/locale/ja/LC_MESSAGES/admin-guide/monitoring/logging/log-formats.en.po +++ b/doc/locale/ja/LC_MESSAGES/admin-guide/monitoring/logging/log-formats.en.po @@ -444,8 +444,8 @@ msgid "The date and time of the client request, enclosed in brackets." msgstr "クライアントリクエストの角括弧で囲まれた日時" #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:167 -msgid "cqtx" -msgstr "cqtx" +msgid "cqhm pqu cqpv" +msgstr "cqhm pqu cqpv" #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:167 msgid "The request line, enclosed in quotes." @@ -977,15 +977,14 @@ msgstr "``cqhl``" #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:398 #, fuzzy msgid "``cqhm``" -msgstr "``cqhl``" +msgstr "``cqhm``" #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:397 msgid "" -"The HTTP method in the client request to |TS|: ``GET``, ``POST``, and so on " -"(subset of ``cqtx``)." +"The HTTP method in the client request to |TS|: ``GET``, ``POST``, and so on. " msgstr "" "|TS| へのクライアントリクエストの HTTP メソッド。``GET`` や ``POST`` やその" -"他のものです。( ``cqtx`` のサブセット )" +"他のものです。" #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:403 msgid "The client request HTTP version." @@ -1126,11 +1125,6 @@ msgstr "``sssc``" msgid "The cipher used by |TS| to communicate with the client over SSL." msgstr "SSL 上でクライアントと通信するのに |TS| が使用した暗号。" -#: ../../../admin-guide/monitoring/logging/log-formats.en.rst:479 -#, fuzzy -msgid "``cqtx``" -msgstr "``\"cqtx\"``" - #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:473 msgid "The full HTTP client request text, minus headers; for example, ::" msgstr "ヘッダーを除く HTTP クライアントリクエストテキストの全て。例えば" @@ -1152,11 +1146,9 @@ msgstr "``cqhl``" #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:484 msgid "" -"The universal resource identifier (URI) of the request from client to |TS| " -"(subset of ``cqtx`` )." +"The universal resource identifier (URI) of the request from client to |TS| ." msgstr "" "|TS| へのクライアントからのリクエストの universal resource identifier (URI) " -"(``cqtx`` のサブセット)" #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:500 #, fuzzy @@ -1961,11 +1953,6 @@ msgstr "[`cqtn`_]" msgid "\"req\"" msgstr "\"req\"" -#: ../../../admin-guide/monitoring/logging/log-formats.en.rst:873 -#: ../../../admin-guide/monitoring/logging/log-formats.en.rst:899 -msgid "\"`cqtx`_\"" -msgstr "\"`cqtx`_\"" - #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:874 #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:900 msgid "s1" @@ -2099,10 +2086,6 @@ msgstr "``[cqtn]``" msgid "``\"req\"``" msgstr "``\"req\"``" -#: ../../../admin-guide/monitoring/logging/log-formats.en.rst:935 -msgid "``\"cqtx\"``" -msgstr "``\"cqtx\"``" - #: ../../../admin-guide/monitoring/logging/log-formats.en.rst:936 msgid "``s1``" msgstr "``s1``" diff --git a/doc/locale/ja/LC_MESSAGES/appendices/faq.en.po b/doc/locale/ja/LC_MESSAGES/appendices/faq.en.po index c7bcf4e0177..36b8da75cbd 100644 --- a/doc/locale/ja/LC_MESSAGES/appendices/faq.en.po +++ b/doc/locale/ja/LC_MESSAGES/appendices/faq.en.po @@ -132,32 +132,6 @@ msgid "" msgstr "" "これについては :ref:`log-formats-squid-format` に詳細に説明されています。" -#: ../../../appendices/faq.en.rst:66 -msgid "What is recorded by the ``cqtx`` field in a custom log file?" -msgstr "" -"カスタムログファイル内の ``cqtx`` フィールドには何が記録されていますか？" - -#: ../../../appendices/faq.en.rst:68 -msgid "" -"In *forward proxy mode*, the ``cqtx`` field records the complete client " -"request in the log file (for example, ``GET http://www.company.com " -"HTTP/1.0``)." -msgstr "" -"*フォワードプロキシーモード* では、 ``cqtx`` フィールドは完全なクライアント" -"のリクエストをログファイルに記録します (例: ``GET http://www.company.com " -"HTTP/1.0``)。" - -#: ../../../appendices/faq.en.rst:70 -msgid "" -"In *reverse proxy mode*, the ``cqtx`` field records the hostname or IP " -"address of the origin server because Traffic Server first remaps the " -"request as per map rules in the :file:`remap.config` file." -msgstr "" -"*リバースプロキシーモード* では ``cqtx`` フィールドはオリジンサーバーのホス" -"ト名または IP アドレスを記録します。なぜなら Traffic Server はリクエストを :" -"file:`remap.config` ファイル内のマッピングルールごとにリクエストをまずリマッ" -"プを行うからです。" - #: ../../../appendices/faq.en.rst:75 msgid "" "Does Traffic Server refresh entries in its host database after a certain " diff --git a/proxy/logging/Log.cc b/proxy/logging/Log.cc index 3f2608103ac..470bba28fc7 100644 --- a/proxy/logging/Log.cc +++ b/proxy/logging/Log.cc @@ -388,11 +388,6 @@ Log::init_fields() global_field_list.add(field, false); field_symbol_hash.emplace("cqtt", field); - field = - new LogField("client_req_text", "cqtx", LogField::STRING, &LogAccess::marshal_client_req_text, &LogAccess::unmarshal_http_text); - global_field_list.add(field, false); - field_symbol_hash.emplace("cqtx", field); - field = new LogField("client_req_http_method", "cqhm", LogField::STRING, &LogAccess::marshal_client_req_http_method, &LogAccess::unmarshal_str); global_field_list.add(field, false); diff --git a/tests/gold_tests/connect/connect.test.py b/tests/gold_tests/connect/connect.test.py index 90638ae3d44..db3c5c57527 100644 --- a/tests/gold_tests/connect/connect.test.py +++ b/tests/gold_tests/connect/connect.test.py @@ -60,7 +60,7 @@ def __setupTS(self): logging: formats: - name: common - format: '% - % [%] "%" % %' + format: '% - % [%] "% % %" % %' logs: - filename: access format: common diff --git a/tests/gold_tests/h2/httpbin.test.py b/tests/gold_tests/h2/httpbin.test.py index ee4d67df6cc..31e14569cb2 100644 --- a/tests/gold_tests/h2/httpbin.test.py +++ b/tests/gold_tests/h2/httpbin.test.py @@ -30,7 +30,7 @@ Condition.HasCurlFeature('http2'), Condition.HasProgram("shasum", "shasum need to be installed on system for this test to work"), ) -#Test.ContinueOnFail = True +# Test.ContinueOnFail = True # ---- # Setup httpbin Origin Server @@ -65,7 +65,7 @@ logging: formats: - name: access - format: '[%] % % % % % % %' + format: '[%] % % % % % % % % %' logs: - filename: access From c7a33efba50d5a050c3b9e82a0fb9ffce4a5c6ef Mon Sep 17 00:00:00 2001 From: Zhengxi Li Date: Fri, 28 Jul 2023 17:48:45 +0000 Subject: [PATCH 2/7] Updated the rst table --- doc/admin-guide/logging/examples.en.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/admin-guide/logging/examples.en.rst b/doc/admin-guide/logging/examples.en.rst index 100f844622d..7b49ac1a19c 100644 --- a/doc/admin-guide/logging/examples.en.rst +++ b/doc/admin-guide/logging/examples.en.rst @@ -53,14 +53,14 @@ The following figure shows a sample log entry in a Netscape Common log file. The numbered sections correspond to the following log fields in |TS|: === ============= =============================================================== -No. Field Description +No. Field Description === ============= =============================================================== 1 chi The IP address of the client's host machine. 2 -- This hyphen (``-``) is always present in Netscape log entries. 3 caun The authenticated client username. A hyphen (``-``) means no authentication was required. 4 cqtn The date and time of the client request, enclosed in brackets. -5 cqhm pqu cqpv The request line, enclosed in quotes. +5 cqhm,pqu,cqpv The request line, enclosed in quotes. 6 pssc The proxy response status code (HTTP reply code). 7 pscl The length of the |TS| response to the client in bytes. === ============= =============================================================== From 0b7c6a651e68ca29df0245552a9081e93b28b071 Mon Sep 17 00:00:00 2001 From: Zhengxi Li Date: Fri, 28 Jul 2023 20:08:59 +0000 Subject: [PATCH 3/7] Updated some test expectations --- tests/gold_tests/connect/gold/connect_access.gold | 2 +- tests/gold_tests/h2/gold/httpbin_access.gold | 6 +++--- tests/gold_tests/h2/httpbin.test.py | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/gold_tests/connect/gold/connect_access.gold b/tests/gold_tests/connect/gold/connect_access.gold index 09a22312edc..188fddd3f72 100644 --- a/tests/gold_tests/connect/gold/connect_access.gold +++ b/tests/gold_tests/connect/gold/connect_access.gold @@ -1 +1 @@ -`` - - [``] "CONNECT ``/ HTTP/1.1" 200 `` +`` - - [``] "CONNECT ``/ http/1.1" 200 `` diff --git a/tests/gold_tests/h2/gold/httpbin_access.gold b/tests/gold_tests/h2/gold/httpbin_access.gold index d409c47d13a..71f1ae8ea10 100644 --- a/tests/gold_tests/h2/gold/httpbin_access.gold +++ b/tests/gold_tests/h2/gold/httpbin_access.gold @@ -1,4 +1,4 @@ -[``] GET http://127.0.0.1:``/get HTTP/1.1 http/2 `` `` TCP_MISS 200 `` -[``] GET http://127.0.0.1:``/bytes/0 HTTP/1.1 http/2 `` `` TCP_MISS 200 0 -[``] GET http://127.0.0.1:``/stream-bytes/102400?seed=0 HTTP/1.1 http/2 `` `` TCP_MISS 200 102400 +[``] GET http://127.0.0.1:``/get http/2 `` `` TCP_MISS 200 `` +[``] GET http://127.0.0.1:``/bytes/0 http/2 `` `` TCP_MISS 200 0 +[``] GET http://127.0.0.1:``/stream-bytes/102400?seed=0 http/2 `` `` TCP_MISS 200 102400 `` diff --git a/tests/gold_tests/h2/httpbin.test.py b/tests/gold_tests/h2/httpbin.test.py index 31e14569cb2..29f61de9963 100644 --- a/tests/gold_tests/h2/httpbin.test.py +++ b/tests/gold_tests/h2/httpbin.test.py @@ -65,7 +65,7 @@ logging: formats: - name: access - format: '[%] % % % % % % % % %' + format: '[%] % % % % % % % %' logs: - filename: access From 3cedc39452b8f24410ea5b1282286a7a3ed3e1ee Mon Sep 17 00:00:00 2001 From: Zhengxi Li Date: Fri, 28 Jul 2023 21:11:02 +0000 Subject: [PATCH 4/7] Updated release note. --- doc/release-notes/upgrading.en.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/release-notes/upgrading.en.rst b/doc/release-notes/upgrading.en.rst index fb0d18f80e3..172c6de79b3 100644 --- a/doc/release-notes/upgrading.en.rst +++ b/doc/release-notes/upgrading.en.rst @@ -85,3 +85,8 @@ Metrics ------------------ The HTTP connection metric proxy.process.http.dead_server.no_requests has been renamed to proxy.process.http.down_server.no_requests. + +Logging +------------------ + +The `cqtx` log field has been deprecated. From 8117670a292ff152915cca0fda18cc88ca5abebc Mon Sep 17 00:00:00 2001 From: Zhengxi Li Date: Mon, 31 Jul 2023 15:01:09 +0000 Subject: [PATCH 5/7] Fix the docs. --- doc/admin-guide/files/logging.yaml.en.rst | 2 +- doc/admin-guide/logging/formatting.en.rst | 2 +- doc/release-notes/upgrading.en.rst | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/admin-guide/files/logging.yaml.en.rst b/doc/admin-guide/files/logging.yaml.en.rst index 98bb7457d13..4f46c506111 100644 --- a/doc/admin-guide/files/logging.yaml.en.rst +++ b/doc/admin-guide/files/logging.yaml.en.rst @@ -117,7 +117,7 @@ the logging output. Multiple logging fields may of course be used:: - '% % % % % %' + '% % % % %' Each logging field is separately enclosed in its own percent-brace set. diff --git a/doc/admin-guide/logging/formatting.en.rst b/doc/admin-guide/logging/formatting.en.rst index 6bfcd8c2051..cdb0b836288 100644 --- a/doc/admin-guide/logging/formatting.en.rst +++ b/doc/admin-guide/logging/formatting.en.rst @@ -584,7 +584,7 @@ sshv Origin Response Origin server's response HTTP version. Request Details ~~~~~~~~~~~~~~~ -The following logging fields can be used in combination to obtain the actual HTTP request details in a HTTP/1.1 request line format, e.g. ``GET http://www.company.com HTTP/1.0``. +The following logging fields show the details of the actual HTTP request and can be used in combination to reproduce a HTTP/1.1 request line (e.g. ``GET http://www.company.com HTTP/1.0``). ===== ============== ========================================================== Field Source Description diff --git a/doc/release-notes/upgrading.en.rst b/doc/release-notes/upgrading.en.rst index 172c6de79b3..ebfc1267b03 100644 --- a/doc/release-notes/upgrading.en.rst +++ b/doc/release-notes/upgrading.en.rst @@ -89,4 +89,4 @@ The HTTP connection metric proxy.process.http.dead_server.no_requests has been r Logging ------------------ -The `cqtx` log field has been deprecated. +The ``cqtx`` log field has been deprecated. From 3e61e08470ec5db2ece8923455cfe22a4c4e0771 Mon Sep 17 00:00:00 2001 From: Zhengxi Li Date: Thu, 3 Aug 2023 16:57:13 +0000 Subject: [PATCH 6/7] Addressed review comments. --- doc/admin-guide/logging/formatting.en.rst | 2 +- doc/release-notes/upgrading.en.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/admin-guide/logging/formatting.en.rst b/doc/admin-guide/logging/formatting.en.rst index cdb0b836288..3067523c32a 100644 --- a/doc/admin-guide/logging/formatting.en.rst +++ b/doc/admin-guide/logging/formatting.en.rst @@ -584,7 +584,7 @@ sshv Origin Response Origin server's response HTTP version. Request Details ~~~~~~~~~~~~~~~ -The following logging fields show the details of the actual HTTP request and can be used in combination to reproduce a HTTP/1.1 request line (e.g. ``GET http://www.company.com HTTP/1.0``). +The following logging fields show the details of the actual HTTP request and can be used in combination to reproduce a client request line (e.g. ``GET http://www.company.com HTTP/1.0``). ===== ============== ========================================================== Field Source Description diff --git a/doc/release-notes/upgrading.en.rst b/doc/release-notes/upgrading.en.rst index ebfc1267b03..0904bce2f48 100644 --- a/doc/release-notes/upgrading.en.rst +++ b/doc/release-notes/upgrading.en.rst @@ -89,4 +89,4 @@ The HTTP connection metric proxy.process.http.dead_server.no_requests has been r Logging ------------------ -The ``cqtx`` log field has been deprecated. +The ``cqtx`` log field has been deprecated, but can be replaced by ``cqhm pqu cqpv``. From 1845380b7d92061ffd2f99194e9f91e18537cc9c Mon Sep 17 00:00:00 2001 From: Zhengxi Li Date: Tue, 8 Aug 2023 16:29:45 +0000 Subject: [PATCH 7/7] deprecated -> removed --- doc/release-notes/upgrading.en.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/upgrading.en.rst b/doc/release-notes/upgrading.en.rst index 0904bce2f48..1a7110b1374 100644 --- a/doc/release-notes/upgrading.en.rst +++ b/doc/release-notes/upgrading.en.rst @@ -89,4 +89,4 @@ The HTTP connection metric proxy.process.http.dead_server.no_requests has been r Logging ------------------ -The ``cqtx`` log field has been deprecated, but can be replaced by ``cqhm pqu cqpv``. +The ``cqtx`` log field has been removed, but can be replaced by ``cqhm pqu cqpv``.