How to get the download link from emulatorsgames.net #181

drpaneas · 2021-02-01T22:17:09Z

drpaneas
Feb 1, 2021
Maintainer

Let's say you have this URL: https://www.emulatorgames.net/roms/neo-geo/king-of-fighters-2002 and you want to get the direct download link so you can wget it. My proposal is:

Open the chrome browser to this page.
Click the "Save Game" button
In the new page, there a timer appears starting to count from 8 down to 0. When this is done, then the browser automatically starts downloading the rom. A seconds (or maybe two or three) later, the download link appears on the screen

Not working? Click Here to start download manually

So that's how you can manually get the direct download link. In order to automate this you can use chromedp. There are two problems:

You don't want to wait for ever, so you need to set a timeout for your automation.
The rom starts downloading automatically, but we don't want that file for various reasons -- e.g. we cannot control via chromedp when the file has been downloaded. So my solution to this is to download the into a temporary directory and delete it as soon as we get the link or another failure has happened (e.g. timeout).

Here's the code:

package main

import (
	"context"
	"fmt"
	"io/ioutil"
	"os"
	"strings"
	"time"

	log "github.com/sirupsen/logrus"

	"github.com/chromedp/cdproto/page"
	"github.com/chromedp/chromedp"
)

func logAction(logStr string) func(context.Context) error {
	return func(context.Context) error {
		log.Printf(logStr)
		return nil
	}
}

func main() {
	link := "https://www.emulatorgames.net/roms/neo-geo/king-of-fighters-2002"
	fmt.Println(GetDownloadLink(link))
}

// GetDownloadLink returns the direct download link for a given game URL
func GetDownloadLink(gameURL string) (donwloadLink string, err error) {

	// Create a temp directory (why? Because it starts downloading automatically after 10 seconds)
	var dir string
	if dirPath, err := os.Getwd(); err != nil {
		log.Println(err)
	} else {
		dir, err = ioutil.TempDir(dirPath, "chromedp-example")
		if err != nil {
			panic(err)
		}
	}
	defer os.RemoveAll(dir) // remove the directory (including the half-finished downloaded file)

	// Create our custom context background
	ctx, cancel := chromedp.NewContext(
		context.Background(),
		chromedp.WithLogf(log.Printf),
	)
	defer cancel()

	// Set the timeout limit as par the context
	ctx, cancel = context.WithTimeout(ctx, 1*time.Minute)
	defer cancel()

	// Open the browser using the previously created context
	chromedp.Run(ctx)

	// Brower settings
	opts := append(chromedp.DefaultExecAllocatorOptions[:],
		chromedp.Flag("disable-background-networking", false),
		chromedp.Flag("disable-renderer-backgrounding", false),
		chromedp.Flag("disable-popup-blocking", false),
		chromedp.Flag("disable-ipc-flooding-protection", false),
		chromedp.Flag("disable-client-side-phishing-detection", false),
		chromedp.Flag("disable-background-timer-throttling", false),
		chromedp.WindowSize(1200, 800),
		chromedp.Flag("headless", false), // just for debugging this code
		chromedp.Flag("hide-scrollbars", false),
		//chromedp.DisableGPU,
		chromedp.UserDataDir(dir),
	)

	// Apply the browser settings into the browser
	allocCtx, cancel := chromedp.NewExecAllocator(ctx, opts...)
	defer cancel()

	// Life is not perfect. This code is triggered when there's either a timeout or error occurs
	taskCtx, cancel := chromedp.NewContext(allocCtx, chromedp.WithLogf(log.Printf))
	defer func() {
		fmt.Println("close browser")
		cancel()
	}()

	// clicky-here-clicky-there-wait-a-bit-and-click-over-there
	var ok bool
	err = chromedp.Run(taskCtx,
		chromedp.Navigate(gameURL),
		chromedp.Click("/html/body/div[3]/div[2]/div[3]/form[1]/button"),
		chromedp.ActionFunc(logAction("Save Game is clicked")),
		chromedp.Sleep(time.Millisecond*600),
		page.SetDownloadBehavior(page.SetDownloadBehaviorBehaviorAllow).WithDownloadPath(dir), // download the file into a specfic dir
		chromedp.WaitVisible("/html/body/div[3]/div[2]/div[1]/p/span[2]/a"),
		chromedp.ActionFunc(logAction(("Download link is available"))),
		chromedp.AttributeValue("/html/body/div[3]/div[2]/div[1]/p/span[2]/a", "href", &donwloadLink, &ok),
	)

	return strings.TrimSpace(donwloadLink), err
}

Output:

$ go run main.go
time="2021-02-01T23:14:52+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:15:08+01:00" level=info msg="Download link is available"
close browser
https://static.emulatorgames.net/roms/neo-geo/kof2002.zip <nil>

Okay, we can remove the <nil> at the end, that's no biggie.

WDYT?

drpaneas · 2021-02-01T22:56:28Z

drpaneas
Feb 1, 2021
Maintainer Author

Embedding this with @zoispag draft PR for PSX:

package main

import (
	"context"
	"fmt"
	"io/ioutil"
	"net/http"
	"os"
	"strings"
	"time"

	"github.com/PuerkitoBio/goquery"
	log "github.com/sirupsen/logrus"

	"github.com/chromedp/cdproto/page"
	"github.com/chromedp/chromedp"
)

func logAction(logStr string) func(context.Context) error {
	return func(context.Context) error {
		log.Printf(logStr)
		return nil
	}
}

func main() {
	Parse("playstation")
}

// GetDownloadLink returns the direct download link for a given game URL
func GetDownloadLink(gameURL string) (donwloadLink string, err error) {

	// Create a temp directory (why? Because it starts downloading automatically after 10 seconds)
	var dir string
	if dirPath, err := os.Getwd(); err != nil {
		log.Println(err)
	} else {
		dir, err = ioutil.TempDir(dirPath, "chromedp-example")
		if err != nil {
			panic(err)
		}
	}
	defer os.RemoveAll(dir) // remove the directory (including the half-finished downloaded file)

	// Create our custom context background
	ctx, cancel := chromedp.NewContext(
		context.Background(),
		chromedp.WithLogf(log.Printf),
	)
	defer cancel()

	// Set the timeout limit as par the context
	ctx, cancel = context.WithTimeout(ctx, 1*time.Minute)
	defer cancel()

	// Open the browser using the previously created context
	chromedp.Run(ctx)

	// Brower settings
	opts := append(chromedp.DefaultExecAllocatorOptions[:],
		chromedp.Flag("disable-background-networking", false),
		chromedp.Flag("disable-renderer-backgrounding", false),
		chromedp.Flag("disable-popup-blocking", false),
		chromedp.Flag("disable-ipc-flooding-protection", false),
		chromedp.Flag("disable-client-side-phishing-detection", false),
		chromedp.Flag("disable-background-timer-throttling", false),
		chromedp.WindowSize(1200, 800),
		chromedp.Flag("headless", false), // just for debugging this code
		chromedp.Flag("hide-scrollbars", false),
		//chromedp.DisableGPU,
		chromedp.UserDataDir(dir),
	)

	// Apply the browser settings into the browser
	allocCtx, cancel := chromedp.NewExecAllocator(ctx, opts...)
	defer cancel()

	// Life is not perfect. This code is triggered when there's either a timeout or error occurs
	taskCtx, cancel := chromedp.NewContext(allocCtx, chromedp.WithLogf(log.Printf))
	defer func() {
		fmt.Println("close browser")
		cancel()
	}()

	// clicky-here-clicky-there-wait-a-bit-and-click-over-there
	var ok bool
	err = chromedp.Run(taskCtx,
		chromedp.Navigate(gameURL),
		chromedp.Click("/html/body/div[3]/div[2]/div[3]/form[1]/button"),
		chromedp.ActionFunc(logAction("Save Game is clicked")),
		chromedp.Sleep(time.Millisecond*600),
		page.SetDownloadBehavior(page.SetDownloadBehaviorBehaviorAllow).WithDownloadPath(dir), // download the file into a specfic dir
		chromedp.WaitVisible("/html/body/div[3]/div[2]/div[1]/p/span[2]/a"),
		chromedp.ActionFunc(logAction(("Download link is available"))),
		chromedp.AttributeValue("/html/body/div[3]/div[2]/div[1]/p/span[2]/a", "href", &donwloadLink, &ok),
	)

	return strings.TrimSpace(donwloadLink), err
}

var (
	emulatorGamesValidConsoles = []string{"playstation"}
	collectedGames             []string
	collectedPages             []string
)

// StringContains searches for a string in a series of strings
func StringContains(baseString string, stringlist ...string) bool {
	if strings.TrimSpace(baseString) == "" {
		log.Debug("Input string is empty")
		return false
	}

	for i, s := range stringlist {
		if strings.TrimSpace(s) == "" {
			continue
		}

		if strings.Contains(
			strings.ToLower(baseString),
			strings.ToLower(s),
		) {
			log.Debug(fmt.Sprint("Input string contains argument number ", i, ": ", s))
			return true
		}
	}

	return false
}

// Parse parses games
func Parse(console string) {
	defer TimeTrack(time.Now(), "EmulatorGames Parser")

	if !StringContains(console, emulatorGamesValidConsoles...) {
		log.Panicf("Platform %s is not yet supported", console)
	}

	uri := fmt.Sprintf("https://www.emulatorgames.net/roms/%s/", console)

	for _, paginationURL := range collectPaginationLinks(uri) {
		parseListPage(paginationURL)
	}

	for _, gameURL := range collectedGames {
		parseGame(gameURL, console)
		log.Printf("Game: %s", gameURL)
	}
}

func parseAndGetDocument(uri string) *goquery.Document {
	// Make HTTP request
	response, err := http.Get(uri)
	if err != nil {
		log.Fatal(err)
	}
	defer response.Body.Close()

	// Create a goquery document from the HTTP response
	document, err := goquery.NewDocumentFromReader(response.Body)
	if err != nil {
		log.Errorf("Error loading HTTP response body (%v)", err)
		return nil
	}

	return document
}

func parseListPage(uri string) {
	document := parseAndGetDocument(uri)
	document.Find("a.eg-box").Each(processGameLink)
}

func processGameLink(_ int, element *goquery.Selection) {
	href, exists := element.Attr("href")
	if exists {
		collectedGames = append(collectedGames, href)
	}
}

func collectPaginationLinks(uri string) []string {
	// Add current page, so even if there are no pagination links
	// the current page will be returned
	_ = append(collectedPages, uri)

	document := parseAndGetDocument(uri)
	document.Find("a.page-link").Each(processPaginationLink)

	return collectedPages
}

// Finds and collects unique pagination links
func processPaginationLink(_ int, element *goquery.Selection) {
	href, exists := element.Attr("href")
	if exists && href != "#" {
		collectedPages = appendIfMissing(collectedPages, href)
	}
}

func appendIfMissing(slice []string, i string) []string {
	for _, ele := range slice {
		if ele == i {
			return slice
		}
	}

	return append(slice, i)
}

func parseGame(gameURL string, console string) {
	document := parseAndGetDocument(gameURL)
	name, _ := document.Find("h1[itemprop='name']").Html()
	lang, _ := document.Find(".eg-meta").Html()
	downloadLink, err := GetDownloadLink(gameURL)
	if err != nil {
		downloadLink = "unknown"
	}
	rom := NewRom(name, console, lang, gameURL, downloadLink)

	log.Println(rom.Stringer())
}

// Rom is the ultimate rom
type Rom struct {
	name         string
	console      string
	language     string
	link         string
	downloadLink string
	version      string
	releaseYear  int
}

// NewRom factory constructor
func NewRom(name string, console string, language string, link string, downloadLink string) *Rom {
	return &Rom{
		name:         name,
		console:      console,
		language:     language,
		link:         link,
		downloadLink: downloadLink,
		version:      "0.0",
		releaseYear:  1970,
	}
}

// Name shit
func (r *Rom) Name() string {
	return r.name
}

// Version shit
func (r *Rom) Version() string {
	return r.version
}

// SetVersion shits
func (r *Rom) SetVersion(version string) {
	r.version = version
}

// ReleaseYear shit
func (r *Rom) ReleaseYear() int {
	return r.releaseYear
}

// SetReleaseYear shit
func (r *Rom) SetReleaseYear(releaseYear int) {
	r.releaseYear = releaseYear
}

// Stringer shit
func (r *Rom) Stringer() string {
	return fmt.Sprintf(`Game: %s (v. %s) for %s. Released in %d [Lang: %s]
Parsed via %s
Download via %s`,
		r.name,
		r.version,
		r.console,
		r.releaseYear,
		r.language,
		r.link,
		r.downloadLink,
	)
}

// TimeTrack tracks the time
func TimeTrack(start time.Time, name string) {
	elapsed := time.Since(start)
	log.Infof("function %s took %s", name, elapsed)
}

The result looks like this:

$ go run main.go
time="2021-02-01T23:50:59+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:51:15+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:51:15+01:00" level=info msg="Game: X Men Vs. Street Fighter [SLUS-00627] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/x-men-vs-street-fighter-slus-00627/\nDownload via https://static.emulatorgames.net/roms/playstation/X Men Vs. Street Fighter [SLUS-00627].7z"
time="2021-02-01T23:51:15+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/x-men-vs-street-fighter-slus-00627/"
time="2021-02-01T23:51:20+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:51:35+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:51:35+01:00" level=info msg="Game: Driver [SLES-01816] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/driver-sles-01816/\nDownload via https://static.emulatorgames.net/roms/playstation/Driver (E) [SLES-01816].7z"
time="2021-02-01T23:51:35+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/driver-sles-01816/"
time="2021-02-01T23:51:41+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:51:56+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:51:56+01:00" level=info msg="Game: Wwf Smackdown 2 Know Your Role [SLUS-01234] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/wwf-smackdown-2-know-your-role-slus-01234/\nDownload via https://static.emulatorgames.net/roms/playstation/WWF SmackDown! 2 - Know Your Role (USA).7z"
time="2021-02-01T23:51:56+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/wwf-smackdown-2-know-your-role-slus-01234/"
time="2021-02-01T23:52:01+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:52:16+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:52:17+01:00" level=info msg="Game: Tekken 3 [SLUS-00402] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/tekken-3-slus-00402/\nDownload via https://static.emulatorgames.net/roms/playstation/Tekken 3 (U) [SLUS-00402].7z"
time="2021-02-01T23:52:17+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/tekken-3-slus-00402/"
time="2021-02-01T23:52:22+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:52:37+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:52:37+01:00" level=info msg="Game: Dino Crisis 2 [SLUS-01279] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/dino-crisis-2-slus-01279/\nDownload via https://static.emulatorgames.net/roms/playstation/Dino Crisis 2 [NTSC-U] [SLUS-01279].7z"
time="2021-02-01T23:52:37+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/dino-crisis-2-slus-01279/"
time="2021-02-01T23:52:42+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:52:57+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:52:58+01:00" level=info msg="Game: Jackie Chan Stuntmaster [SLUS-00684] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/jackie-chan-stuntmaster-slus-00684/\nDownload via https://static.emulatorgames.net/roms/playstation/Jackie Chan Stuntmaster [NTSC-U] [SLUS-00684].7z"
time="2021-02-01T23:52:58+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/jackie-chan-stuntmaster-slus-00684/"
time="2021-02-01T23:53:03+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:53:18+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:53:18+01:00" level=info msg="Game: Tony Hawk S Pro Skater [SLUS-008.60] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/tony-hawk-s-pro-skater-slus-00860/\nDownload via https://static.emulatorgames.net/roms/playstation/Tony Hawk S Pro Skater (U) [SLUS-008.60].7z"
time="2021-02-01T23:53:18+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/tony-hawk-s-pro-skater-slus-00860/"
time="2021-02-01T23:53:23+01:00" level=info msg="Save Game is clicked"
time="2021-02-01T23:53:38+01:00" level=info msg="Download link is available"
close browser
time="2021-02-01T23:53:39+01:00" level=info msg="Game: Tomb Raider Greatest Hits [SLUS-00152] (v. 0.0) for playstation. Released in 1970 [Lang: USA - English]\nParsed via https://www.emulatorgames.net/roms/playstation/tomb-raider-greatest-hits-slus-00152/\nDownload via https://static.emulatorgames.net/roms/playstation/Tomb Raider Greatest Hits (U) [SLUS-00152].7z"
time="2021-02-01T23:53:39+01:00" level=info msg="Game: https://www.emulatorgames.net/roms/playstation/tomb-raider-greatest-hits-slus-00152/"
close browser
....
....

Of course this is a very slow process, yet it seems to be working fine.

2 replies

zoispag Feb 2, 2021
Maintainer

That's actually GREAT! 💪🏼

ge0r Feb 2, 2021
Maintainer

Indeed! \m/

ge0r · 2021-02-02T15:13:01Z

ge0r
Feb 2, 2021
Maintainer

Smart approach!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to get the download link from emulatorsgames.net #181

{{title}}

Replies: 2 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to get the download link from emulatorsgames.net #181

drpaneas Feb 1, 2021 Maintainer

Replies: 2 comments · 2 replies

drpaneas Feb 1, 2021 Maintainer Author

zoispag Feb 2, 2021 Maintainer

ge0r Feb 2, 2021 Maintainer

ge0r Feb 2, 2021 Maintainer

drpaneas
Feb 1, 2021
Maintainer

Replies: 2 comments 2 replies

drpaneas
Feb 1, 2021
Maintainer Author

zoispag Feb 2, 2021
Maintainer

ge0r Feb 2, 2021
Maintainer

ge0r
Feb 2, 2021
Maintainer