Issue with scale_color_manual

[chris-prener]

I have just updated to the most recent ggplot2 version (somewhat belatedly) and have noticed a change in how scale_colour_manual behaves. This seems along the same lines as #4511 but with a different issue.

The reprex below generates a plot where the legend order follows the values of y for the most recent date. This works as expected plot 1 below, where scale_colour_brewer is used to define the palette.

Until I upgraded ggplot2, I had the same result with scale_colour_manual. Now, however, the legend items are ordered alphabetically instead. This is reproduced in plot 2 below.

I can't find anything to suggest that this was an intended change - apologies if I have missed that. I really appreciate folks taking a look at this and helping sort out what is going on here!

# dependencies
suppressPackageStartupMessages({
  library(dplyr)
  library(forcats)
  library(ggplot2)
  library(readr)
  
  library(RColorBrewer)
})
  
# load data
metro_data <- read_csv("https://raw.githubusercontent.com/slu-openGIS/MO_HEALTH_Covid_Tracking/master/data/metro_all/metro_full.csv")
#> Rows: 4792 Columns: 14
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr   (2): short_name, full_name
#> dbl  (11): geoid, cases, case_rate, new_cases, case_avg, case_avg_rate, deat...
#> date  (1): report_date
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
# prep data
## subset
metro_data <- filter(metro_data, report_date >= "2020-03-10")

## create factors
metro_data <- mutate(metro_data, factor_var = fct_reorder2(short_name, report_date, case_rate))

# plot data
## define colors
pal <- brewer.pal(n = 8, name = "Set1")
cols <- c("Cape Girardeau" = pal[6], "Columbia" = pal[3], 
          "Jefferson City" = pal[4], "Joplin" = pal[7],
          "Kansas City" = pal[2], "Springfield" = pal[5], 
          "St. Joseph" = pal[8], "St. Louis" = pal[1])

## create plot 1 (legend ordered by y values)
ggplot() +
  geom_line(metro_data, mapping = aes(x = report_date, y = case_rate, color = factor_var), size = 2) +
  scale_colour_brewer(palette = "Set1")

## create plot 2 (legend ordered alphabetically)
ggplot() +
  geom_line(metro_data, mapping = aes(x = report_date, y = case_rate, color = factor_var), size = 2) +
  scale_colour_manual(values = cols)

^{Created on 2021-09-14 by the reprex package (v2.0.1)}

[teunbrand]

I think because the limits are taken from the names of the cols vector (which coincidentally is alphanumerically sorted, but needn't be), your cols vector determines the order.

[chris-prener]

Sure @teunbrand - that makes sense in terms of current behavior - but it represents a change from prior versions of ggplot2.

[teunbrand]

Ah, I see: apologies for my misinterpreting the issue. I think the change is a consequence of the following point documented in the 3.3.4 release note:

• Manual scales now allow named vectors passed to values to contain fewer elements than existing in the data. Elements not present in values will be set to NA (@thomasp85, scale_manual throws unexpected error when na.values given and insufficient values provided #3451)

The functional change that was introduced is in the following lines:

ggplot2/R/scale-manual.r

Lines 135 to 137 in ffcfaf7

	if (is.null(limits)) {
	limits <- names(values)
	}

Since the limits now become the names of the values vector, the order of the names in the values vector affect in what order the legend keys are displayed. This is a departure from previous behaviour where the limits were inferred from the data when values had names. Reading the release note text doesn't imply that this departure is intended.

[chris-prener]

Thanks for flagging the cause and submitting a PR, @teunbrand!

[yutannihilation]

Should be fixed by #4619

[chris-prener]

thanks for the update @yutannihilation - I'll test this out tonight!

[chris-prener]

Worked great - thanks @yutannihilation and @teunbrand!

[yutannihilation]

Thanks for confirming!