Ergast API is being deprecated at the end of 2024

[pesaventofilippo]

As per http://ergast.com/mrd/:

The API will be updated until the end of the 2024 season then shutdown at the end of the year.

It's obviously a long-term problem, but what should we do?
If it's only the API being deprecated, then someone could just run a mirror (which I'm planning to do at some point) by cloning the database every time it updates. If however they stop updating the DB, do we have an alternative to Ergast?

[theOehrly]

Thanks for the hint. I guess there's plenty of time still to tackle this issue. My first reaction is that the community around FastF1 would be perfect to take this over. Maybe get the folks from the Multiviewer community involved as well (although they don't really depend on it, there are a lot of tech-savvy nerds there). I can easily contact the relevant people there.

We don't want Ergast to die. For multiple reasons. First, it's a unique database of Formula 1 history. Second, Chris has spent huge amounts of time to build it and collect the data. It would be a pity if this effort is not retained. I'll go ahead and contact him to make sure it's OK to use his database. And then, it would be nice to build a team to get this done by the end of next year at the latest. Doing this all alone, along with maintaining and updating FastF1, might be a bit of a challenge for me.

An API server for FastF1 was planned anyway, although nothing is really underway yet. It's all in a very rough planning stage still.

From the way I understand it, there will be no further updates to the data. Therefore, it would be necessary to figure out a (preferably) at least semi-automatic way to populate the database with future data.

[pesaventofilippo]

Fully agree. Do you know how the Ergast DB gets populated? Is it still a manual process or does it use data from F1 livetiming (or other sources)?
If it's at least somewhat automatic, it would be great if the current Ergast maintainer could open-source or at least give his system to you, so we don't start from scratch.

What I suspect though is that is a heavily manual process, and in that case I don't know how much we could do automatically, because there aren't many other data sources apart from the F1 livetiming and Ergast itself.

[theOehrly]

Just a very short update here. I've talked to Chris, who runs Ergast. He's totally Ok with it, if we try to build an open source successor to Ergast and use his database.

I currently tend towards rewriting the API server completely. The current Ergast runtime is available on Github, but the code itself is fairly old. I'm positive that a more modern implementation can scale better. One possible option would be Cloudflare's D1 database with Cloudflare Pages functions. When creating this as an open source project, it's possible to apply for project support to get a premium tier for free. This has the advantage that hosting would be free, with virtually unlimited load capacity. Of course, some more thought should be given to all of this before starting anything.
I still have some exams to deal with, but I'd like to start pushing forwards with this somewhere in October.

[mdsakalu]

I'd be interested in helping out with this effort, just let me know what you need.

[theOehrly]

I've set up a Slack workspace for now to better discuss how we might want to tackle this project. Everybody who wants to help out can join here:
https://join.slack.com/t/fastf1/shared_invite/zt-25osjniqp-TsBgxY_7alSMar5wEgWcXQ

(Note that this workspace is only intended for discussions regarding the deprecation of the Ergast API and development work around that, but not for user support questions or similar)

Some other people contacted Chris from Ergast independently, and they will be joining the Slack workspace as well.

(ping for those I know may be interested in helping @mdsakalu @Casper-Guo @harningle @pesaventofilippo)

[pbulsink]

I'm curious as to if there are any updates on this project - over at f1dataR we also rely on Ergast but hope to be able to migrate to a new option by years' end. Neither of us there are strong web/server devs but if you need help with testing I'm sure we could lend a hand.

[theOehrly]

There has been a lot of progress. We are aiming to announce a beta testing phase next. In theory, all the code is already publicly available, and the service is live right now. But we want to fix some more bugs and also add a bit more documentation and information for potential contributors before we tell people where to look. But the plan is to "open" beta testing this summer.

[marcokreeft87]

It seams to be a long term problem but the current season isnt displayed in the same way as it used to be. So its kind off a now problem

[theOehrly]

The problem is actively being worked on. But everybody who is working on it is doing so in their spare time. There's no reasonable way to make things go faster. Aiming for a full replacement in 2025 is the goal and it is realistic. Anything before that is an optional bonus.
And let's see how things really turn out once the season actually starts. Ergast has never supported Pre-Season Testing.

[marcokreeft87]

Yes I know. I wish I had time to help but my current work doesnt leave much room to do stuff after work hours.

Ergast used to have the practice,sprint,qualifying times in the schedule, but they don't have that in the 2024 season data :(

[theOehrly]

I'm aware that those times are still missing. But as I said, the season hasn't started yet. This stuff isn't automated in current Ergast. It's not unlikely that they will be added still.

[quasibasi]

Hello all!
The drivers, constructors, qualifying data seems to be missing for 2024, even though the Bahrain qualy happened today. Any ideas of whether they will be updated?
Thanks in advance!

[marcokreeft87]

Please read this thread

[quasibasi]

Oh I see, you mean it's the same question which has been answered by:

The problem is actively being worked on. But everybody who is working on it is doing so in their spare time.

Thanks!

[SmCTwelve]

Per the Ergast FAQ page: "a season is assumed to begin when results become available for the first race of the season". So just wait and see if the data becomes available after the race results.

[quasibasi]

Thanks a lot @SmCTwelve!

[theOehrly]

Things are moving forward and we've reached alpha state for the Ergast successor. Here's an update, posted by Jess Perry in the Slack Workspace that was mentioned above.

Hey everyone,
We’re excited to announce that the Ergast API replacement that we’ve been working on (jolpica-f1) has reached Alpha and is ready for testing. Our focus has been (and will remain for the 2024 season) to reach full feature parity with the existing API before making improvements. We’ve landed on the Apache 2.0 open source license for this project, which we believe best fits the project’s needs today and into the future.
You can help us out by testing the API and reporting any bugs you find on GitHub. Please have a look at the “How to contribute” notes in the repository as well.
To try it out, simply replace ergast.com/api/f1/ with api.jolpi.ca/ergast/f1/. We currently have an unauthenticated rate limit of 200 requests per hour, which we may adjust based on load, if you have a use case that may need a higher limit please reach out, and we may be able to sort something out.
If you would like to support jolpica-f1 monetarily, we have set up a Ko-fi. This project is not a for-profit venture, but it will have costs to maintain over time and we’d like to share that burden as much as possible. Our intentions are to be as transparent as possible with this project and that includes its finances.
Special thanks to Philipp Schaefer, Xuanyi Wang and Chris Free for all of their hard work getting us to this point.

We have a few intentional differences from ergast documented in the README & issues, the largest being no longer supporting XML.
If you spot anything not mentioned there, it may be a bug.
We plan to get some better documentation in place soon

Please note, jolpica-f1 is a project that is independent of FastF1. I am only a contributor and partially an initiator of the project, and others have done a huge amount of work to make it happen. All issues should be reported in the project's GitHub repository and not here.

Users of FastF1 can manually change the URL for the Ergast API from the default server to the jolpica-f1 server. To do this, your scripts need to contain the following lines of code at the beginning:

import fastf1
fastf1.ergast.interface.BASE_URL = "https://api.jolpi.ca/ergast/f1"

This will cause FastF1 to use the jolpica-f1 server for internal requests and for all requests that you make explicitly through the Ergast API wrapper.

[elidare]

Hey guys,

Firstly, huge thanks for creating and keeping such a database!

I found ergast.com in a list of public APIs, and as a F1 fan decided to build my university Fullstack course project on this. (Done here https://github.com/elidare/elidare-f1-statistics)

I was quite astonished to find the API returning 503 Backend fetch failed today, when it was working well on Saturday. Especially given the project deadline this week :)

So could you please tell me if this API is shut down now for good or there are only some temporary problems?
I see the replacement for the initial API, api.jolpi.ca/ergast/f1/, so would you recommend switching to it in my project? I don't expect huge RPS since it's only a university project.

And again, thanks for your work!

[theOehrly]

As far as I know, Ergast is not shutdown. It should continue at least for the remainder of the year and maybe into the next season, but without any further data updates then.

The problem is that some people do not comply with Ergast rate limits and burst rates may be fairly high (for a single server setup on a minimal budget that is also not fully optimized). The Ergast server is not always able to handle that. You're getting a 503 usually when the cache server is still alive, but the backend is down. So it most likely is a temporary problem, but one that occurs somewhat frequently.

Jolpica has had better uptime than Ergast over the last few months. But it is officially still in alpha. We've had some unexpected downtime as well as intentional downtime for updates while we are trying to get a handle on everything. In the end, it is your call which API you want to use. None of the two is an optimal solution right now.

[elidare]

Thanks a lot for your answer! Now I understand 503.
I'll stay on Jolpica then (hoping no downtime occurs at the exact moment my project will be reviewed) and put some notes about this into my readme.

[theOehrly]

I see, you're using it for a university project. Feel free to open an issue on jolpica/jolpica-f1 if you notice downtime. There's no active monitoring for that yet. So we only notice that the API is down, when someone informs us about it. Maintenance related downtime is usually pretty short.
Some people have switched production apps over to jolpica-f1 already, as they decided that it is more reliable. I think you should be good.

[elidare]

Thank you!