Update Amazon bot match to cover Amazonbot user agent (#156)

As seen in the [Amazon developer docs](https://developer.amazon.com/amazonbot), `Amazonbot` is also an user agent used by Amazon crawlers. Did my best to follow existing code, but please let me know if there are any changes that should be made.
alaz · Sep 19, 2024 · fd9ea8a · fd9ea8a
1 parent b4d6dce
commit fd9ea8a
Show file tree

Hide file tree

Showing 4 changed files with 35 additions and 4 deletions.
diff --git a/README.md b/README.md
@@ -50,6 +50,7 @@ end
 ## Supported
 
 - [Ahrefs](https://ahrefs.com/robot)
+- [Amazonbot](https://developer.amazon.com/amazonbot)
 - [Amazon AdBot](https://adbot.amazon.com/index.html)
 - [Applebot](https://support.apple.com/en-us/119829)
 - [Baidu spider](http://help.baidu.com/question?prod_en=master&class=498&id=1000973)

diff --git a/lib/legitbot/amazon.rb b/lib/legitbot/amazon.rb
@@ -2,9 +2,10 @@
 
 module Legitbot # :nodoc:
   # https://adbot.amazon.com/index.html
+  # https://developer.amazon.com/amazonbot
   class Amazon < BotMatch
-    domains 'amazonadbot.com.'
+    domains 'amazon.', 'amazonadbot.com.'
   end
 
-  rule Legitbot::Amazon, %w[AmazonAdBot]
+  rule Legitbot::Amazon, %w[Amazonbot AmazonAdBot]
 end
diff --git a/test/amazon_test.rb b/test/amazon_test.rb
@@ -30,7 +30,7 @@ def test_malicious_ua
     refute_predicate bot, :valid?
   end
 
-  def test_valid_ua
+  def test_user_agent1
     bot = Legitbot.bot(
       'Mozilla/5.0 (compatible; AmazonAdBot/1.0; +https://adbot.amazon.com)',
       '54.166.7.90'
@@ -40,7 +40,19 @@ def test_valid_ua
     assert_predicate bot, :valid?
   end
 
-  def test_valid_name
+  # rubocop:disable Layout/LineLength
+  def test_user_agent2
+    bot = Legitbot.bot(
+      'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML\, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)',
+      '52.70.240.171'
+    )
+
+    assert bot
+    assert_predicate bot, :valid?
+  end
+  # rubocop:enable Layout/LineLength
+
+  def test_valid_name1
     bot = Legitbot.bot(
       'Mozilla/5.0 (compatible; AmazonAdBot/1.0; +https://adbot.amazon.com)',
       '54.166.7.90'
@@ -49,6 +61,17 @@ def test_valid_name
     assert_equal :amazon, bot.detected_as
   end
 
+  # rubocop:disable Layout/LineLength
+  def test_valid_name2
+    bot = Legitbot.bot(
+      'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML\, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)',
+      '52.70.240.171'
+    )
+
+    assert_equal :amazon, bot.detected_as
+  end
+  # rubocop:enable Layout/LineLength
+
   def test_fake_name
     bot = Legitbot.bot(
       'Mozilla/5.0 (compatible; AmazonAdBot/1.0; +https://adbot.amazon.com)',

diff --git a/test/lib/dns_server_mock.rb b/test/lib/dns_server_mock.rb
@@ -29,6 +29,12 @@
   '54.166.7.90' => {
     ptr: %w[crawler-54-166-7-90.amazonadbot.com]
   },
+  '52-70-240-171.crawl.amazonbot.amazon' => {
+    a: %w[52.70.240.171]
+  },
+  '52.70.240.171' => {
+    ptr: %w[52-70-240-171.crawl.amazonbot.amazon]
+  },
 
   # Apple
   '17-58-98-60.applebot.apple.com' => {