Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep on getting the error "Error: Method Not Allowed", after certain number of translations #127

Open
chronos2403 opened this issue Dec 12, 2023 · 1 comment

Comments

@chronos2403
Copy link

import express from 'express';
import fetchChapter from './fetchChapter.js';

import { translate } from '@vitalets/google-translate-api';
import { HttpProxyAgent } from 'http-proxy-agent';

const app = express();
const port = 3000;

// Proxy list for IP rotation
const proxyList = [
    { protocol: 'http', host: '162.248.225.116', port: 80 },
    { protocol: 'http', host: '162.248.225.32', port: 80 },
    { protocol: 'http', host: '207.2.120.58', port: 80 },
    { protocol: 'http', host: '162.248.225.224', port: 80 },
    { protocol: 'http', host: '162.248.225.13', port: 80 },
    { protocol: 'http', host: '162.248.224.103', port: 80 },
    { protocol: 'http', host: '66.225.254.16', port: 80 },
    { protocol: 'http', host: '162.248.225.101', port: 80 },
    { protocol: 'http', host: '162.248.225.106', port: 80 },
    { protocol: 'http', host: '212.107.31.118', port: 80 },
    { protocol: 'http', host: '162.248.225.118', port: 80 }
  ];
let currentProxyIndex = -1; 


const getNextProxy = () => {
  currentProxyIndex = (currentProxyIndex + 1) % proxyList.length;
  return proxyList[currentProxyIndex];
};

app.get('/', async (req, res) => {
  const chapters = [];
  let currentUrl = 'https://www.xxbiqudu.com/52_52835/139293077.html';//sample site for raw chinese novel
  const jspath_txt = "#content";
  const jspath_btn = "#wrapper > div.content_read > div > div.bookname > div.bottem1 > a:nth-child(5)" || "#wrapper > div.content_read > div > div.bookname > div.bottem1 > a:nth-child(6)";

  const fetchAndTranslate = async (url, chapterCount) => {
    const chapterText = await fetchChapter(url, jspath_txt, jspath_btn);
    const originalContent = chapterText.currentContent;

    try {
      // Rotate proxies every 20 chapters
      if (chapterCount % 20 === 0) {
        const proxy = getNextProxy();
        const agent = new HttpProxyAgent(`${proxy.protocol}://${proxy.host}:${proxy.port}`);
        console.log(`Switching to proxy: ${proxy.host}:${proxy.port}`);
        const translatedContent = await translate(originalContent, { to: 'en', fetchOptions: { agent } });
     
        chapters.push(translatedContent.text);
      } else {
        const translatedContent = await translate(originalContent, { to: 'en' });
   
        chapters.push(translatedContent.text);
      }

      if (chapterText.nextUrl) {
        await fetchAndTranslate(chapterText.nextUrl, chapterCount + 1);
      }
    } catch (error) {
      console.error(`Error occurred. Switching to the next proxy. Error: ${error.message}`);
   
      const proxy = getNextProxy();
      console.log(`Switching to proxy: ${proxy.host}:${proxy.port}`);
      await fetchAndTranslate(url, chapterCount); 
    }
  };

  await fetchAndTranslate(currentUrl, 1);

  res.send(chapters);
});

app.listen(port, () => {
  console.log(`Server is running on http://localhost:${port}`);
});

I wrote this code trying to scrape raw chinese light novel chapters and translate them to english, but whenever i run this code, i keep on the getting errors like:
"Switching to proxy: 162.248.225.106:80
Error occurred. Switching to the next proxy. Error: Method Not Allowed"

@vitalets
Copy link
Owner

  1. In the catch clause it's better to check that it's TooManyRequestsError and change proxy only in that case. See example in readme https://github.com/vitalets/google-translate-api?tab=readme-ov-file#limits. Method Not Allowed error is not related to proxy
  2. What is the size of originalContent? Could you try to truncate it to smaller size and re-check?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants