Issue of removeObjects API only delete maximum of 2000 objects

[zhihan-wei-exa]

When applying batch delete by calling the removeObjects API, it only deletes 2000 objects per API call. It shouldn't work that way by design, since there is nothing mentioned in the official document about the limit of removeObjects API.

Several versions (6.0.7, 7.0.0, 7.1.3) have been tested, and all have the same issue. Suspect it's caused by a bug in the iterable implementation of the API.

[kannappanr]

import io.minio.MinioClient;
import io.minio.RemoveObjectsArgs;
import io.minio.Result;
import io.minio.errors.MinioException;
import io.minio.messages.DeleteError;
import io.minio.messages.DeleteObject;
import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.LinkedList;
import java.util.List;

public class RemoveObjects {
/** MinioClient.removeObject() example removing multiple objects. */
public static void main(String[] args)
throws IOException, NoSuchAlgorithmException, InvalidKeyException {
try {

  MinioClient minioClient =
      MinioClient.builder()
          .endpoint("http://localhost:9000")
          .credentials("minio", "minio123")
          .build();


  List<DeleteObject> objects = new LinkedList<>();
  for (int i = 0; i < 3000; i++) {
  objects.add(new DeleteObject("object" + i));
}  
  
  Iterable<Result<DeleteError>> results =
      minioClient.removeObjects(
          RemoveObjectsArgs.builder().bucket("test").objects(objects).build());
  for (Result<DeleteError> result : results) {
    DeleteError error = result.get();
    System.out.println(
        "Error in deleting object " + error.objectName() + "; " + error.message());
  }
} catch (MinioException e) {
  System.out.println("Error occurred: " + e);
}

}
}

[balamurugana]

@zhihan-wei-exa How do you know the API stops removing after 2000 objects? Please enable MinioClient.traceOn() and share the output along with your code populating > 2000 objects for deletion.

[kannappanr]

@balamurugana was able to reproduce the issue with the above code. I observed it with mc admin trace in the backend. I have also attached the trace for the code that I was able to reproduce it with.

[kannappanr]

`
$ java -cp "minio-7.1.4-all.jar:." RemoveObjects
---------START-HTTP---------
POST /test?delete= HTTP/1.1
Host: localhost:9000
Accept-Encoding: identity
User-Agent: MinIO (amd64; amd64) minio-java/7.1.4
Content-MD5: q1BFUhVQKJNHoBga3en9Zg==
x-amz-content-sha256: 6b77ec69c82964ec0295dcffbee3524c4de639381a3e02744798acfbd272373a
x-amz-date: 20201017T032310Z
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20201017/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=REDACTED

falseobject0object1object2object3object4object5object6object7object8object9object10object11object12object13object14object15object16object17object18object19object20object21object22object23object24object25object26object27object28object29object30object31object32object33object34object35object36object37object38object39object40object41object42object43object44object45object46object47object48object49object50object51object52object53object54object55object56object57object58object59object60object61object62object63object64object65object66object67object68object69object70object71object72object73object74object75object76object77object78object79object80object81object82object83object84object85object86object87object88object89object90object91object92object93object94object95object96object97object98object99object100object101object102object103object104object105object106object107object108object109object110object111object112object113object114object115object116object117object118object119object120object121object122object123object124object125object126object127object128object129object130object131object132object133object134object135object136object137object138object139object140object141object142object143object144object145object146object147object148object149object150object151object152object153object154object155object156object157object158object159object160object161object162object163object164object165object166object167object168object169object170object171object172object173object174object175object176object177object178object179object180object181object182object183object184object185object186object187object188object189object190object191object192object193object194object195object196object197object198object199object200object201object202object203object204object205object206object207object208object209object210object211object212object213object214object215object216object217object218object219object220object221object222object223object224object225object226object227object228object229object230object231object232object233object234object235object236object237object238object239object240object241object242object243object244object245object246object247object248object249object250object251object252object253object254object255object256object257object258object259object260object261object262object263object264object265object266object267object268object269object270object271object272object273object274object275object276object277object278object279object280object281object282object283object284object285object286object287object288object289object290object291object292object293object294object295object296object297object298object299object300object301object302object303object304object305object306object307object308object309object310object311object312object313object314object315object316object317object318object319object320object321object322object323object324object325object326object327object328object329object330object331object332object333object334object335object336object337object338object339object340object341object342object343object344object345object346object347object348object349object350object351object352object353object354object355object356object357object358object359object360object361object362object363object364object365object366object367object368object369object370object371object372object373object374object375object376object377object378object379object380object381object382object383object384object385object386object387object388object389object390object391object392object393object394object395object396object397object398object399object400object401object402object403object404object405object406object407object408object409object410object411object412object413object414object415object416object417object418object419object420object421object422object423object424object425object426object427object428object429object430object431object432object433object434object435object436object437object438object439object440object441object442object443object444object445object446object447object448object449object450object451object452object453object454object455object456object457object458object459object460object461object462object463object464object465object466object467object468object469object470object471object472object473object474object475object476object477object478object479object480object481object482object483object484object485object486object487object488object489object490object491object492object493object494object495object496object497object498object499object500object501object502object503object504object505object506object507object508object509object510object511object512object513object514object515object516object517object518object519object520object521object522object523object524object525object526object527object528object529object530object531object532object533object534object535object536object537object538object539object540object541object542object543object544object545object546object547object548object549object550object551object552object553object554object555object556object557object558object559object560object561object562object563object564object565object566object567object568object569object570object571object572object573object574object575object576object577object578object579object580object581object582object583object584object585object586object587object588object589object590object591object592object593object594object595object596object597object598object599object600object601object602object603object604object605object606object607object608object609object610object611object612object613object614object615object616object617object618object619object620object621object622object623object624object625object626object627object628object629object630object631object632object633object634object635object636object637object638object639object640object641object642object643object644object645object646object647object648object649object650object651object652object653object654object655object656object657object658object659object660object661object662object663object664object665object666object667object668object669object670object671object672object673object674object675object676object677object678object679object680object681object682object683object684object685object686object687object688object689object690object691object692object693object694object695object696object697object698object699object700object701object702object703object704object705object706object707object708object709object710object711object712object713object714object715object716object717object718object719object720object721object722object723object724object725object726object727object728object729object730object731object732object733object734object735object736object737object738object739object740object741object742object743object744object745object746object747object748object749object750object751object752object753object754object755object756object757object758object759object760object761object762object763object764object765object766object767object768object769object770object771object772object773object774object775object776object777object778object779object780object781object782object783object784object785object786object787object788object789object790object791object792object793object794object795object796object797object798object799object800object801object802object803object804object805object806object807object808object809object810object811object812object813object814object815object816object817object818object819object820object821object822object823object824object825object826object827object828object829object830object831object832object833object834object835object836object837object838object839object840object841object842object843object844object845object846object847object848object849object850object851object852object853object854object855object856object857object858object859object860object861object862object863object864object865object866object867object868object869object870object871object872object873object874object875object876object877object878object879object880object881object882object883object884object885object886object887object888object889object890object891object892object893object894object895object896object897object898object899object900object901object902object903object904object905object906object907object908object909object910object911object912object913object914object915object916object917object918object919object920object921object922object923object924object925object926object927object928object929object930object931object932object933object934object935object936object937object938object939object940object941object942object943object944object945object946object947object948object949object950object951object952object953object954object955object956object957object958object959object960object961object962object963object964object965object966object967object968object969object970object971object972object973object974object975object976object977object978object979object980object981object982object983object984object985object986object987object988object989object990object991object992object993object994object995object996object997object998object999
HTTP/1.1 200
Accept-Ranges: bytes
Content-Length: 155006
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Server: MinIO/DEVELOPMENT.2020-10-16T17-50-21Z
Vary: Origin
X-Amz-Request-Id: 163EA998BB1B1083
X-Xss-Protection: 1; mode=block
Date: Sat, 17 Oct 2020 03:23:10 GMT

----------END-HTTP----------
---------START-HTTP---------
POST /test?delete= HTTP/1.1
Host: localhost:9000
Accept-Encoding: identity
User-Agent: MinIO (amd64; amd64) minio-java/7.1.4
Content-MD5: fZkdnhKAFymcqETXjoZy+A==
x-amz-content-sha256: 77fee35625088ce40ed7c42fd9163efb743d877841c301ae8b05fd78db603f06
x-amz-date: 20201017T032311Z
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20201017/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;content-md5;host;x-amz-content-sha256;x-amz-date, Signature=REDACTED

falseobject1000object1001object1002object1003object1004object1005object1006object1007object1008object1009object1010object1011object1012object1013object1014object1015object1016object1017object1018object1019object1020object1021object1022object1023object1024object1025object1026object1027object1028object1029object1030object1031object1032object1033object1034object1035object1036object1037object1038object1039object1040object1041object1042object1043object1044object1045object1046object1047object1048object1049object1050object1051object1052object1053object1054object1055object1056object1057object1058object1059object1060object1061object1062object1063object1064object1065object1066object1067object1068object1069object1070object1071object1072object1073object1074object1075object1076object1077object1078object1079object1080object1081object1082object1083object1084object1085object1086object1087object1088object1089object1090object1091object1092object1093object1094object1095object1096object1097object1098object1099object1100object1101object1102object1103object1104object1105object1106object1107object1108object1109object1110object1111object1112object1113object1114object1115object1116object1117object1118object1119object1120object1121object1122object1123object1124object1125object1126object1127object1128object1129object1130object1131object1132object1133object1134object1135object1136object1137object1138object1139object1140object1141object1142object1143object1144object1145object1146object1147object1148object1149object1150object1151object1152object1153object1154object1155object1156object1157object1158object1159object1160object1161object1162object1163object1164object1165object1166object1167object1168object1169object1170object1171object1172object1173object1174object1175object1176object1177object1178object1179object1180object1181object1182object1183object1184object1185object1186object1187object1188object1189object1190object1191object1192object1193object1194object1195object1196object1197object1198object1199object1200object1201object1202object1203object1204object1205object1206object1207object1208object1209object1210object1211object1212object1213object1214object1215object1216object1217object1218object1219object1220object1221object1222object1223object1224object1225object1226object1227object1228object1229object1230object1231object1232object1233object1234object1235object1236object1237object1238object1239object1240object1241object1242object1243object1244object1245object1246object1247object1248object1249object1250object1251object1252object1253object1254object1255object1256object1257object1258object1259object1260object1261object1262object1263object1264object1265object1266object1267object1268object1269object1270object1271object1272object1273object1274object1275object1276object1277object1278object1279object1280object1281object1282object1283object1284object1285object1286object1287object1288object1289object1290object1291object1292object1293object1294object1295object1296object1297object1298object1299object1300object1301object1302object1303object1304object1305object1306object1307object1308object1309object1310object1311object1312object1313object1314object1315object1316object1317object1318object1319object1320object1321object1322object1323object1324object1325object1326object1327object1328object1329object1330object1331object1332object1333object1334object1335object1336object1337object1338object1339object1340object1341object1342object1343object1344object1345object1346object1347object1348object1349object1350object1351object1352object1353object1354object1355object1356object1357object1358object1359object1360object1361object1362object1363object1364object1365object1366object1367object1368object1369object1370object1371object1372object1373object1374object1375object1376object1377object1378object1379object1380object1381object1382object1383object1384object1385object1386object1387object1388object1389object1390object1391object1392object1393object1394object1395object1396object1397object1398object1399object1400object1401object1402object1403object1404object1405object1406object1407object1408object1409object1410object1411object1412object1413object1414object1415object1416object1417object1418object1419object1420object1421object1422object1423object1424object1425object1426object1427object1428object1429object1430object1431object1432object1433object1434object1435object1436object1437object1438object1439object1440object1441object1442object1443object1444object1445object1446object1447object1448object1449object1450object1451object1452object1453object1454object1455object1456object1457object1458object1459object1460object1461object1462object1463object1464object1465object1466object1467object1468object1469object1470object1471object1472object1473object1474object1475object1476object1477object1478object1479object1480object1481object1482object1483object1484object1485object1486object1487object1488object1489object1490object1491object1492object1493object1494object1495object1496object1497object1498object1499object1500object1501object1502object1503object1504object1505object1506object1507object1508object1509object1510object1511object1512object1513object1514object1515object1516object1517object1518object1519object1520object1521object1522object1523object1524object1525object1526object1527object1528object1529object1530object1531object1532object1533object1534object1535object1536object1537object1538object1539object1540object1541object1542object1543object1544object1545object1546object1547object1548object1549object1550object1551object1552object1553object1554object1555object1556object1557object1558object1559object1560object1561object1562object1563object1564object1565object1566object1567object1568object1569object1570object1571object1572object1573object1574object1575object1576object1577object1578object1579object1580object1581object1582object1583object1584object1585object1586object1587object1588object1589object1590object1591object1592object1593object1594object1595object1596object1597object1598object1599object1600object1601object1602object1603object1604object1605object1606object1607object1608object1609object1610object1611object1612object1613object1614object1615object1616object1617object1618object1619object1620object1621object1622object1623object1624object1625object1626object1627object1628object1629object1630object1631object1632object1633object1634object1635object1636object1637object1638object1639object1640object1641object1642object1643object1644object1645object1646object1647object1648object1649object1650object1651object1652object1653object1654object1655object1656object1657object1658object1659object1660object1661object1662object1663object1664object1665object1666object1667object1668object1669object1670object1671object1672object1673object1674object1675object1676object1677object1678object1679object1680object1681object1682object1683object1684object1685object1686object1687object1688object1689object1690object1691object1692object1693object1694object1695object1696object1697object1698object1699object1700object1701object1702object1703object1704object1705object1706object1707object1708object1709object1710object1711object1712object1713object1714object1715object1716object1717object1718object1719object1720object1721object1722object1723object1724object1725object1726object1727object1728object1729object1730object1731object1732object1733object1734object1735object1736object1737object1738object1739object1740object1741object1742object1743object1744object1745object1746object1747object1748object1749object1750object1751object1752object1753object1754object1755object1756object1757object1758object1759object1760object1761object1762object1763object1764object1765object1766object1767object1768object1769object1770object1771object1772object1773object1774object1775object1776object1777object1778object1779object1780object1781object1782object1783object1784object1785object1786object1787object1788object1789object1790object1791object1792object1793object1794object1795object1796object1797object1798object1799object1800object1801object1802object1803object1804object1805object1806object1807object1808object1809object1810object1811object1812object1813object1814object1815object1816object1817object1818object1819object1820object1821object1822object1823object1824object1825object1826object1827object1828object1829object1830object1831object1832object1833object1834object1835object1836object1837object1838object1839object1840object1841object1842object1843object1844object1845object1846object1847object1848object1849object1850object1851object1852object1853object1854object1855object1856object1857object1858object1859object1860object1861object1862object1863object1864object1865object1866object1867object1868object1869object1870object1871object1872object1873object1874object1875object1876object1877object1878object1879object1880object1881object1882object1883object1884object1885object1886object1887object1888object1889object1890object1891object1892object1893object1894object1895object1896object1897object1898object1899object1900object1901object1902object1903object1904object1905object1906object1907object1908object1909object1910object1911object1912object1913object1914object1915object1916object1917object1918object1919object1920object1921object1922object1923object1924object1925object1926object1927object1928object1929object1930object1931object1932object1933object1934object1935object1936object1937object1938object1939object1940object1941object1942object1943object1944object1945object1946object1947object1948object1949object1950object1951object1952object1953object1954object1955object1956object1957object1958object1959object1960object1961object1962object1963object1964object1965object1966object1967object1968object1969object1970object1971object1972object1973object1974object1975object1976object1977object1978object1979object1980object1981object1982object1983object1984object1985object1986object1987object1988object1989object1990object1991object1992object1993object1994object1995object1996object1997object1998object1999
HTTP/1.1 200
Accept-Ranges: bytes
Content-Length: 156116
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Server: MinIO/DEVELOPMENT.2020-10-16T17-50-21Z
Vary: Origin
X-Amz-Request-Id: 163EA998CE2C248B
X-Xss-Protection: 1; mode=block
Date: Sat, 17 Oct 2020 03:23:11 GMT

----------END-HTTP----------

`

[balamurugana]

@balamurugana was able to reproduce the issue with the above code. I observed it with mc admin trace in the backend

@kannappanr Do you have > 2000 objects in the server as per the code?

[kannappanr]

You do not need any object to be present on the server to issue DeleteObjects API, I just have an empty bucket named test

[zhihan-wei-exa]

Code:

import io.minio.Result;
import io.minio.errors.*;
import io.minio.messages.DeleteError;
import io.minio.messages.Item;
import org.xmlpull.v1.XmlPullParserException;

import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.LinkedList;
import java.util.List;

public class Test {
    public static void main(String[] args) throws InvalidPortException, InvalidEndpointException, XmlPullParserException, InsufficientDataException, NoSuchAlgorithmException, IOException, NoResponseException, InvalidKeyException, InternalException, InvalidBucketNameException, ErrorResponseException {
        System.out.println("Start ......");    
       String endpoint = "http://localhost:9000";
        String accessKey = "accesskey";
        String secretKey = "secretkey";
        String bucketName = "bucket";
        String cloudPathPrefix = "prefix";

        MinioClient minioClient = new MinioClient(endpoint, accessKey, secretKey);

        List<String> objectNames = new LinkedList<String>();

        int beforeDelete = 0;
        for (Result<Item> resultItem : minioClient.listObjects(bucketName, cloudPathPrefix)) {
            Item item = resultItem.get();
            objectNames.add(item.objectName());
            beforeDelete += 1;
        }
        System.out.println("No. of files before delete: " + beforeDelete);

        int deletedFiles = 0;
        try {
            for (Result<DeleteError> errorResult : minioClient.removeObjects(bucketName, objectNames)) {
                DeleteError error = errorResult.get();
                System.out.println("Failed to remove" + error.objectName() + "'.Error:" + error.message());
                deletedFiles += 1;
            }
            System.out.println("No. of files have been deleted: " + deletedFiles);
        } catch (MinioException e) {
           System.out.println("Error occurred: " + e);
        }

        int afterDelete = 0;
        for (Result<Item> result : minioClient.listObjects(bucketName, cloudPathPrefix)) {
            Item item = result.get();
            objectNames.add(item.objectName());
            afterDelete += 1;
        }
        System.out.println("No. of files after delete: " + afterDelete);
    }
}

Result:

No. of files before delete: 4800
No. of files have been deleted: 2000
No. of files after delete: 2800

[balamurugana]

Fixed by #1076