[20210331] Jsoup(Java 크롤링), robots.txt

[JuHyun419]

자바 크롤링 - Jsoup, robots.txt(크롤링 금지 사항)

• robots.txt => 해당 웹사이트/robots.txt
• ex)인프런 => https://www.inflearn.com
• 인프런의 robots.txt => https://www.inflearn.com/robots.txt

package inflearn;

import org.jsoup.Connection;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import java.io.IOException;

public class InflearnCrawling {

    private static final String URL = "https://www.inflearn.com/courses";

    public static void main(String[] args) {
        try {
            Connection conn = Jsoup.connect(URL);

            Document html = conn.get();

            Elements elements = html.getElementsByClass("course_title");

            for (Element element : elements) {
                System.out.println(element);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

• 제목 class