[Usage]: vLLM /score with Mixedbread reranker (Qwen2 seq-cls override): **scores differ vs local Mixedbread**; small payload = same order/different scores; large payload (\~1K chars/doc) = **order diverges** #22983
Description
Title: vLLM /score with Mixedbread reranker (Qwen2 seq-cls override): scores differ vs local Mixedbread; small payload = same order/different scores; large payload (~1K chars/doc) = order diverges
Summary
Using vLLM with hf_overrides to serve mixedbread-ai/mxbai-rerank-base-v2 as Qwen2ForSequenceClassification and querying /score, I observe:
- Scores differ between vLLM and local Mixedbread usage.
- On a small test, ranking matches Mixedbread, but vLLM probabilities are much higher on negatives (calibration mismatch).
- On a larger test (10 docs, ~1000 chars each), the ranking order diverges vs Mixedbread.
Looking for guidance on label tokens, pair template, and any calibration needed for parity.
Environment
-
Server: AWS g6e.xlarge (NVIDIA L40S 48 GB)
-
vLLM command:
vllm serve mixedbread-ai/mxbai-rerank-base-v2 \ --hf_overrides '{"architectures":["Qwen2ForSequenceClassification"],"classifier_from_token":["0","1"],"method":"from_2_way_softmax"}' \ --host 0.0.0.0 --port 8000 -
Endpoint:
/score -
Reference baseline:
from mxbai_rerank import MxbaiRerankV2run locally on Mac M3 Pro (CPU)
Repro 1 — Small payload (same order, different scores)
Query: Who wrote To Kill a Mockingbird?
Observation: Order matches, but vLLM scores are much higher on irrelevant docs.
| Doc (short) | vLLM /score |
MxbaiRerankV2 |
|---|---|---|
| Harper Lee (novel) | 0.9974413 | 0.9941708 |
| Harper Lee (bio) | 0.9837090 | 0.9705565 |
| Jane Austen | 0.9364685 | 0.2971900 |
| Harry Potter | 0.7970418 | 0.0699584 |
| Moby-Dick | 0.6106243 | 0.0519386 |
| Gatsby | 0.5890393 | 0.0290230 |
Repro 2 — Large payload (~1000 chars/doc, 10 docs; order diverges)
Query: Who wrote the novel 1984?
Full JSON payload (10 docs ≈1000 chars each):
{
"model": "mixedbread-ai/mxbai-rerank-base-v2",
"query": "Who wrote the novel 1984?",
"documents": [
"“Nineteen Eighty-Four” (commonly printed as 1984) is a dystopian novel written by George Orwell and first published in 1949 by Secker & Warburg. The book follows Winston Smith, a minor Party member in Airstrip One, as he struggles against the totalitarian rule of the Party and its leader, Big Brother. Orwell’s prose introduces concepts like Newspeak, Doublethink, Thoughtcrime, and the Ministry of Truth, all of which have entered everyday language as shorthand for propaganda and authoritarian manipulation. The novel’s bleak finale, in which personal loyalty, love, and memory are crushed by the state, was intended as a warning about the corrosive effects of unchecked power. Because 1984 was written by George Orwell—born Eric Arthur Blair—the question of authorship is settled explicitly in the historical record and in every standard edition’s title page, author bio, and critical apparatus that attributes the work to Orwell, the English essayist, journalist, and novelist best known alongside “Animal Farm.”",
"George Orwell (1903–1950), the pen name of the English writer and journalist Eric Arthur Blair, wrote two of the twentieth century’s most widely read political fables: “Animal Farm” (1945) and “Nineteen Eighty-Four” (1949). Before becoming synonymous with the adjective “Orwellian,” he worked as a colonial police officer in Burma, fought on the Republican side in the Spanish Civil War, and wrote reportage and criticism that probed poverty, empire, and propaganda. In countless biographical summaries and library catalogs, Orwell is credited as the author of 1984; bibliographies list the novel under his surname, and first printings from Secker & Warburg name “George Orwell” on the jacket. When people ask who wrote 1984, the answer is unambiguous: it was authored by George Orwell. The book’s enduring relevance lies in its precise, economical style and its clinical depiction of surveillance and language control—concerns that Orwell had sharpened across essays like “Politics and the English Language.”",
"Eric Arthur Blair published under the name George Orwell to separate his literary persona from his private life. Under that pen name he wrote novels, essays, and reportage that triangulate ethics, language, and power. “Nineteen Eighty-Four” is one of his last works, completed when his health was failing on the Isle of Jura, and released in 1949 to immediate debate and acclaim. Publishing records, copyright notices, and critical editions uniformly identify Orwell as the book’s author; in university courses and popular reading lists, the entry for 1984 always pairs the title with “George Orwell.” The text’s motifs—Big Brother, the telescreen, the Thought Police—are regularly cited back to Orwell as their creator. Thus, if the task is to determine who wrote the novel 1984, the correct attribution is to George Orwell (Eric Arthur Blair), whose career also includes “Down and Out in Paris and London,” “Homage to Catalonia,” and the satirical allegory “Animal Farm.”",
"The adjective “Orwellian” has become a byword for policies and practices that normalize surveillance, historical revision, and euphemistic language. The term derives directly from George Orwell’s body of work, especially his novel “Nineteen Eighty-Four,” which he wrote and published in 1949. In that novel, authored by Orwell, the Party’s manipulation of truth through Newspeak and the rewriting of archives shows how language can be engineered to curtail dissent. Journalists, scholars, and policymakers routinely cite “Orwellian” measures when discussing censorship, mass data collection, or propaganda campaigns. The etymology here is instructive for authorship: the word points back to Orwell the writer, and to 1984 as a text by George Orwell rather than by any contemporary figure. The cultural afterlife of the novel—film adaptations, stage versions, and countless essays—always credits Orwell as the author.",
"Apple’s famous “1984” Super Bowl commercial—directed by Ridley Scott to introduce the Macintosh—uses the imagery of a gray, regimented audience and a single athlete shattering a giant screen. While the ad does not name the novel outright, it trades on the public’s recognition of the dystopia described in “Nineteen Eighty-Four,” written by George Orwell. Contemporary news coverage of the spot, Apple’s own retrospectives, and marketing histories all anchor the reference in Orwell’s authorship, contrasting the conformity of Big Brother’s world with the personal computing freedom Apple wished to symbolize. The advertisement’s power requires the audience to know that 1984 is a novel by George Orwell; the metaphor works because Orwell authored a text about surveillance and control that marketers could invert. Thus, even in advertising contexts, the association between the title and its author is explicit: 1984 is by George Orwell.",
"The title “1984” has been used outside literature—for example, David Bowie’s song “1984” on the album “Diamond Dogs,” and Van Halen’s record titled “1984.” These works are often discussed in relation to the same set of themes (dystopia, control, spectacle), but music historians typically note that the song and album titles are allusions rather than sources. Crucially, when distinguishing among these references, guides and encyclopedias clarify that the **novel** “Nineteen Eighty-Four” was written by George Orwell in 1949, whereas the song and album are separate artistic creations. Librarians and teachers advising students emphasize this authorship distinction to prevent citation errors: if a research prompt asks, “Who wrote the novel 1984?” the correct response is not the musician or band but the English author George Orwell. Contextual notes in study guides often add that Orwell’s 1984 helped shape the cultural lexicon that later artists riffed on.",
"Ray Bradbury’s “Fahrenheit 451” (1953) portrays firefighters who burn books and citizens lulled by wall-sized televisions. Although readers frequently compare Bradbury’s story to the world depicted in “Nineteen Eighty-Four,” the two novels have different authors and project distinct anxieties. Bradbury—a Californian who came up through pulp magazines—wrote his book to protest censorship and the numbing effects of mass media. In contrast, 1984 was written by the English author George Orwell and addresses the political psychology of totalitarian rule. Literary surveys pair the works to teach how dystopias can arise from different causes, but they are careful about attribution: 1984 is by Orwell; “Fahrenheit 451” is by Bradbury. For a question seeking the author of the novel 1984, citing Bradbury would be incorrect despite thematic overlap. Correct authorship belongs to Orwell.",
"Aldous Huxley’s “Brave New World” (1932) is often taught alongside George Orwell’s “Nineteen Eighty-Four” to contrast two visions of social control: Huxley’s through pleasure and conditioning, Orwell’s through fear and coercion. Essays routinely compare Huxley and Orwell on technology, biology, and language. Yet the authorship remains distinct: Huxley wrote “Brave New World,” while the novel 1984 was written by George Orwell. In classroom discussions and exam rubrics, instructors explicitly remind students not to conflate these writers when answering authorship questions. A comparative paragraph might say, “Orwell’s ‘1984’ depicts state terror and surveillance,” which implicitly attributes the book to Orwell; however, the safest and clearest statement is that 1984 is a novel by George Orwell, published in 1949, whereas “Brave New World” is by Aldous Huxley, published in 1932.",
"Harper Lee, an American novelist born in Monroeville, Alabama, wrote “To Kill a Mockingbird” (1960), which won the Pulitzer Prize and became a staple of U.S. school curricula. Though sometimes discussed in surveys of twentieth-century fiction that also include George Orwell, Lee’s work concerns race, justice, and childhood in the American South rather than totalitarianism. If asked who wrote 1984, the correct answer is not Harper Lee, because 1984 is the title of a dystopian novel written by George Orwell, not of Lee’s courtroom drama. Annotated reading lists and library subject headings keep these authors separate: “Lee, Harper—To Kill a Mockingbird,” and “Orwell, George—Nineteen Eighty-Four.” Any answer that attributes 1984 to Lee would be a misattribution, even though both writers produced widely taught classics of the twentieth century.",
"J. R. R. Tolkien created Middle-earth in “The Hobbit” and “The Lord of the Rings,” pioneering modern high fantasy with invented languages and deep mythic backstory. His concerns—philology, myth-making, the corrupting lure of power embodied in a magic ring—differ entirely from the political and linguistic critique in George Orwell’s “Nineteen Eighty-Four.” Literary encyclopedias index Tolkien under fantasy and medievalism, not under dystopian literature. For a factual question—who wrote the novel 1984?—the Tolkien corpus offers no relevant authorship information; the correct attribution is to George Orwell, whose 1949 novel examines surveillance, language, and authoritarian control. Confusing Tolkien with Orwell would be akin to crediting “The Silmarillion” to Orwell, which would be plainly incorrect. The author of 1984 is George Orwell."
]
}Result: vLLM ranking differs from the Mixedbread library’s ranking on this larger set. Looking for guidance on label tokens, pair template, and any calibration needed for parity.
How would you like to use vllm
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.