Skip to content
This repository has been archived by the owner on Apr 16, 2020. It is now read-only.

MDSConnect raw data (Library of Congress) #152

Open
ghost opened this issue May 18, 2017 · 11 comments
Open

MDSConnect raw data (Library of Congress) #152

ghost opened this issue May 18, 2017 · 11 comments

Comments

@ghost
Copy link

ghost commented May 18, 2017 •
edited by ghost
Loading

The Library of Congress announced today that it is making 25 million records in its online catalog available for free bulk download at loc.gov/cds/products/marcDist.php. This is the largest release of digital records in the Library’s history.
-- https://www.loc.gov/item/prn-17-068?loclr=twloc

I grabbed a raw dump of the files and added it on the storage host nihal (QmepgFW7BHEtU4pZJdxaNiv75mKLLRQnPi1KaaXmQN4V1a)

> screen bash -c "wget --mirror --convert-links --no-parent -nv https://loc.gov/cds/downloads/MDSConnect/ 2>&1 | tee loc.log"
> ipfs add -r loc.gov/cds/downloads/MDSConnect/
added QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9 loc.gov/cds/downloads/MDSConnect

Result: https://ipfs.io/ipfs/QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9/ https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1JAA7HX5
@ghost
Copy link
Author

ghost commented May 18, 2017

It's slightly more than 100 GB:

> ipfs object stat QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9/
NumLinks: 301
BlockSize: 21623
LinksSize: 21621
DataSize: 2
CumulativeSize: 105535285885

@ghost
Copy link

ghost commented May 29, 2017

I'm only seeing 18GB after running that same wget command. Any ideas?

@ghost
Copy link
Author

ghost commented May 29, 2017

I don't know :) Here's what I downloaded:

> du -sh loc.gov/cds/downloads/MDSConnect/
99G	loc.gov/cds/downloads/MDSConnect/
> ls -la loc.gov/cds/downloads/MDSConnect/
total 103037888
drwxr-xr-x 1 lars users       15944 May 17 23:03 .
drwxr-xr-x 1 lars users          20 May 17 03:31 ..
-rw-r--r-- 1 lars users    69715040 Mar  6 18:23 BooksAll.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   241298365 Mar  6 18:25 BooksAll.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   699767853 Mar  6 18:26 BooksAll.2014.part01.xml.gz
-rw-r--r-- 1 lars users   203707455 Mar  6 18:27 BooksAll.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   203934030 Mar  6 18:29 BooksAll.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   574933045 Mar  6 18:30 BooksAll.2014.part02.xml.gz
-rw-r--r-- 1 lars users   196930346 Mar  6 18:31 BooksAll.2014.part03.marc8.gz
-rw-r--r-- 1 lars users   197207496 Mar  6 18:32 BooksAll.2014.part03.utf8.gz
-rw-r--r-- 1 lars users   572514807 Mar  6 18:33 BooksAll.2014.part03.xml.gz
-rw-r--r-- 1 lars users   181828576 Mar  6 18:34 BooksAll.2014.part04.marc8.gz
-rw-r--r-- 1 lars users   182253854 Mar  6 18:35 BooksAll.2014.part04.utf8.gz
-rw-r--r-- 1 lars users   545114876 Mar  6 18:36 BooksAll.2014.part04.xml.gz
-rw-r--r-- 1 lars users   173423594 Mar  6 18:37 BooksAll.2014.part05.marc8.gz
-rw-r--r-- 1 lars users   173986807 Mar  6 18:38 BooksAll.2014.part05.utf8.gz
-rw-r--r-- 1 lars users   529003733 Mar  6 18:39 BooksAll.2014.part05.xml.gz
-rw-r--r-- 1 lars users   182129529 Mar  6 18:40 BooksAll.2014.part06.marc8.gz
-rw-r--r-- 1 lars users   182671671 Mar  6 18:41 BooksAll.2014.part06.utf8.gz
-rw-r--r-- 1 lars users   542825762 Mar  6 18:43 BooksAll.2014.part06.xml.gz
-rw-r--r-- 1 lars users   187315378 Mar  6 18:44 BooksAll.2014.part07.marc8.gz
-rw-r--r-- 1 lars users   187912394 Mar  6 18:45 BooksAll.2014.part07.utf8.gz
-rw-r--r-- 1 lars users   555739812 Mar  6 18:46 BooksAll.2014.part07.xml.gz
-rw-r--r-- 1 lars users   189680872 Mar  6 18:47 BooksAll.2014.part08.marc8.gz
-rw-r--r-- 1 lars users   190342088 Mar  6 18:49 BooksAll.2014.part08.utf8.gz
-rw-r--r-- 1 lars users   562044482 Mar  6 18:50 BooksAll.2014.part08.xml.gz
-rw-r--r-- 1 lars users   193086744 Mar  6 18:51 BooksAll.2014.part09.marc8.gz
-rw-r--r-- 1 lars users   193790793 Mar  6 18:52 BooksAll.2014.part09.utf8.gz
-rw-r--r-- 1 lars users   563448752 Mar  6 18:54 BooksAll.2014.part09.xml.gz
-rw-r--r-- 1 lars users   195052177 Mar  6 18:55 BooksAll.2014.part10.marc8.gz
-rw-r--r-- 1 lars users   195750552 Mar  6 18:56 BooksAll.2014.part10.utf8.gz
-rw-r--r-- 1 lars users   571278098 Mar  6 18:57 BooksAll.2014.part10.xml.gz
-rw-r--r-- 1 lars users   195386784 Mar  6 18:59 BooksAll.2014.part11.marc8.gz
-rw-r--r-- 1 lars users   196300104 Mar  6 19:00 BooksAll.2014.part11.utf8.gz
-rw-r--r-- 1 lars users   571091283 Mar  6 19:01 BooksAll.2014.part11.xml.gz
-rw-r--r-- 1 lars users   194946672 Mar  6 19:03 BooksAll.2014.part12.marc8.gz
-rw-r--r-- 1 lars users   195849864 Mar  6 19:04 BooksAll.2014.part12.utf8.gz
-rw-r--r-- 1 lars users   570395829 Mar  6 19:05 BooksAll.2014.part12.xml.gz
-rw-r--r-- 1 lars users   203152504 Mar  6 19:07 BooksAll.2014.part13.marc8.gz
-rw-r--r-- 1 lars users   204472578 Mar  6 19:08 BooksAll.2014.part13.utf8.gz
-rw-r--r-- 1 lars users   592712132 Mar  6 19:09 BooksAll.2014.part13.xml.gz
-rw-r--r-- 1 lars users   200994841 Mar  6 19:11 BooksAll.2014.part14.marc8.gz
-rw-r--r-- 1 lars users   202046295 Mar  6 19:12 BooksAll.2014.part14.utf8.gz
-rw-r--r-- 1 lars users   583943437 Mar  6 19:13 BooksAll.2014.part14.xml.gz
-rw-r--r-- 1 lars users   203542905 Mar  6 19:15 BooksAll.2014.part15.marc8.gz
-rw-r--r-- 1 lars users   204589201 Mar  6 19:16 BooksAll.2014.part15.utf8.gz
-rw-r--r-- 1 lars users   590033819 Mar  6 19:17 BooksAll.2014.part15.xml.gz
-rw-r--r-- 1 lars users   221243670 Mar  6 19:19 BooksAll.2014.part16.marc8.gz
-rw-r--r-- 1 lars users   221115040 Mar  6 19:20 BooksAll.2014.part16.utf8.gz
-rw-r--r-- 1 lars users   641499947 Mar  6 19:22 BooksAll.2014.part16.xml.gz
-rw-r--r-- 1 lars users   214644431 Mar  6 19:23 BooksAll.2014.part17.marc8.gz
-rw-r--r-- 1 lars users   214918525 Mar  6 19:25 BooksAll.2014.part17.utf8.gz
-rw-r--r-- 1 lars users   618703199 Mar  6 19:26 BooksAll.2014.part17.xml.gz
-rw-r--r-- 1 lars users   224531239 Mar  6 19:27 BooksAll.2014.part18.marc8.gz
-rw-r--r-- 1 lars users   224156107 Mar  6 19:29 BooksAll.2014.part18.utf8.gz
-rw-r--r-- 1 lars users   456765866 May 17 05:59 BooksAll.2014.part18.xml.gz
-rw-r--r-- 1 lars users    71469761 Mar  6 19:32 BooksAll.2014.part19.marc8.gz
-rw-r--r-- 1 lars users   226774478 Mar  6 19:33 BooksAll.2014.part19.utf8.gz
-rw-r--r-- 1 lars users   659834069 Mar  6 19:35 BooksAll.2014.part19.xml.gz
-rw-r--r-- 1 lars users   222810801 Mar  6 19:36 BooksAll.2014.part20.marc8.gz
-rw-r--r-- 1 lars users   222901634 Mar  6 19:37 BooksAll.2014.part20.utf8.gz
-rw-r--r-- 1 lars users   646174224 Mar  6 19:39 BooksAll.2014.part20.xml.gz
-rw-r--r-- 1 lars users   228088426 Mar  6 19:40 BooksAll.2014.part21.marc8.gz
-rw-r--r-- 1 lars users   227911669 Mar  6 19:41 BooksAll.2014.part21.utf8.gz
-rw-r--r-- 1 lars users   665956191 Mar  6 19:43 BooksAll.2014.part21.xml.gz
-rw-r--r-- 1 lars users   225335764 Mar  6 19:44 BooksAll.2014.part22.marc8.gz
-rw-r--r-- 1 lars users   225286411 Mar  6 19:45 BooksAll.2014.part22.utf8.gz
-rw-r--r-- 1 lars users   657160123 Mar  6 19:47 BooksAll.2014.part22.xml.gz
-rw-r--r-- 1 lars users   228905706 Mar  6 19:48 BooksAll.2014.part23.marc8.gz
-rw-r--r-- 1 lars users   228945550 Mar  6 19:50 BooksAll.2014.part23.utf8.gz
-rw-r--r-- 1 lars users   668859353 Mar  6 19:51 BooksAll.2014.part23.xml.gz
-rw-r--r-- 1 lars users   231010312 Mar  6 19:53 BooksAll.2014.part24.marc8.gz
-rw-r--r-- 1 lars users   231475659 Mar  6 19:54 BooksAll.2014.part24.utf8.gz
-rw-r--r-- 1 lars users   671986563 Mar  6 19:56 BooksAll.2014.part24.xml.gz
-rw-r--r-- 1 lars users   241602985 Mar  6 19:57 BooksAll.2014.part25.marc8.gz
-rw-r--r-- 1 lars users   240955227 Mar  6 19:59 BooksAll.2014.part25.utf8.gz
-rw-r--r-- 1 lars users   703426547 Mar  6 20:00 BooksAll.2014.part25.xml.gz
-rw-r--r-- 1 lars users   237474386 Mar  6 20:01 BooksAll.2014.part26.marc8.gz
-rw-r--r-- 1 lars users   238178024 Mar  6 20:03 BooksAll.2014.part26.utf8.gz
-rw-r--r-- 1 lars users   688003265 Mar  6 20:04 BooksAll.2014.part26.xml.gz
-rw-r--r-- 1 lars users   237290429 Mar  6 20:06 BooksAll.2014.part27.marc8.gz
-rw-r--r-- 1 lars users   237710635 Mar  6 20:07 BooksAll.2014.part27.utf8.gz
-rw-r--r-- 1 lars users   690960392 Mar  6 20:08 BooksAll.2014.part27.xml.gz
-rw-r--r-- 1 lars users   241833691 Mar  6 20:10 BooksAll.2014.part28.marc8.gz
-rw-r--r-- 1 lars users   242377031 Mar  6 20:11 BooksAll.2014.part28.utf8.gz
-rw-r--r-- 1 lars users   700636950 Mar  6 20:13 BooksAll.2014.part28.xml.gz
-rw-r--r-- 1 lars users   252353493 Mar  6 20:14 BooksAll.2014.part29.marc8.gz
-rw-r--r-- 1 lars users   252763519 Mar  6 20:15 BooksAll.2014.part29.utf8.gz
-rw-r--r-- 1 lars users   724849542 Mar  6 20:17 BooksAll.2014.part29.xml.gz
-rw-r--r-- 1 lars users   254035211 Mar  6 20:18 BooksAll.2014.part30.marc8.gz
-rw-r--r-- 1 lars users   254643556 Mar  6 20:20 BooksAll.2014.part30.utf8.gz
-rw-r--r-- 1 lars users   731563945 Mar  6 20:21 BooksAll.2014.part30.xml.gz
-rw-r--r-- 1 lars users   258590257 Mar  6 20:22 BooksAll.2014.part31.marc8.gz
-rw-r--r-- 1 lars users   258920866 Mar  6 20:24 BooksAll.2014.part31.utf8.gz
-rw-r--r-- 1 lars users   742418454 Mar  6 20:25 BooksAll.2014.part31.xml.gz
-rw-r--r-- 1 lars users   272235356 Mar  6 20:27 BooksAll.2014.part32.marc8.gz
-rw-r--r-- 1 lars users   272071674 Mar  6 20:28 BooksAll.2014.part32.utf8.gz
-rw-r--r-- 1 lars users   774985570 Mar  6 20:30 BooksAll.2014.part32.xml.gz
-rw-r--r-- 1 lars users   278033342 Mar  6 20:31 BooksAll.2014.part33.marc8.gz
-rw-r--r-- 1 lars users   277343192 Mar  6 20:33 BooksAll.2014.part33.utf8.gz
-rw-r--r-- 1 lars users   780125418 Mar  6 20:34 BooksAll.2014.part33.xml.gz
-rw-r--r-- 1 lars users   282031930 Mar  6 20:35 BooksAll.2014.part34.marc8.gz
-rw-r--r-- 1 lars users   280731807 Mar  6 20:37 BooksAll.2014.part34.utf8.gz
-rw-r--r-- 1 lars users   790682376 Mar  6 20:39 BooksAll.2014.part34.xml.gz
-rw-r--r-- 1 lars users   277102937 Mar  6 20:40 BooksAll.2014.part35.marc8.gz
-rw-r--r-- 1 lars users   277720141 Mar  6 20:42 BooksAll.2014.part35.utf8.gz
-rw-r--r-- 1 lars users   776713285 Mar  6 20:44 BooksAll.2014.part35.xml.gz
-rw-r--r-- 1 lars users   288079691 Mar  6 20:45 BooksAll.2014.part36.marc8.gz
-rw-r--r-- 1 lars users   287548806 Mar  6 20:47 BooksAll.2014.part36.utf8.gz
-rw-r--r-- 1 lars users   808790988 Mar  6 20:49 BooksAll.2014.part36.xml.gz
-rw-r--r-- 1 lars users   295612804 Mar  6 20:50 BooksAll.2014.part37.marc8.gz
-rw-r--r-- 1 lars users   295801774 Mar  6 20:52 BooksAll.2014.part37.utf8.gz
-rw-r--r-- 1 lars users   819919343 Mar  6 20:54 BooksAll.2014.part37.xml.gz
-rw-r--r-- 1 lars users   309400161 Mar  6 20:56 BooksAll.2014.part38.marc8.gz
-rw-r--r-- 1 lars users   309675475 Mar  6 20:58 BooksAll.2014.part38.utf8.gz
-rw-r--r-- 1 lars users   850580276 Mar  6 20:59 BooksAll.2014.part38.xml.gz
-rw-r--r-- 1 lars users   305520335 Mar  6 21:01 BooksAll.2014.part39.marc8.gz
-rw-r--r-- 1 lars users   303987823 Mar  6 21:03 BooksAll.2014.part39.utf8.gz
-rw-r--r-- 1 lars users   855291792 Mar  6 21:05 BooksAll.2014.part39.xml.gz
-rw-r--r-- 1 lars users   297834423 Mar  6 21:06 BooksAll.2014.part40.marc8.gz
-rw-r--r-- 1 lars users   297543947 Mar  6 21:08 BooksAll.2014.part40.utf8.gz
-rw-r--r-- 1 lars users   839434664 Mar  6 21:09 BooksAll.2014.part40.xml.gz
-rw-r--r-- 1 lars users    72981354 Mar  6 21:10 BooksAll.2014.part41.marc8.gz
-rw-r--r-- 1 lars users    73301759 Mar  6 21:10 BooksAll.2014.part41.utf8.gz
-rw-r--r-- 1 lars users   214206332 Mar  6 21:10 BooksAll.2014.part41.xml.gz
-rw-r--r-- 1 lars users        1000 Mar  6 22:01 Class.thru.d130228.label.gz
-rw-r--r-- 1 lars users   248527814 Mar  6 22:02 Class.thru.d130228.records.marc8.gz
-rw-r--r-- 1 lars users   248561562 Mar  6 22:02 Class.thru.d130228.records.utf8.gz
-rw-r--r-- 1 lars users   729950005 Mar  6 22:03 Class.thru.d130228.records.xml.gz
-rw-r--r-- 1 lars users    50765920 Mar  6 22:03 Class.thru.d130228.report.gz
-rw-r--r-- 1 lars users    14113402 Mar  6 21:15 Computer.Files.2014.part01.marc8.gz
-rw-r--r-- 1 lars users    14124087 Mar  6 21:15 Computer.Files.2014.part01.utf8.gz
-rw-r--r-- 1 lars users    34579155 Mar  6 21:15 Computer.Files.2014.part01.xml.gz
-rw-r--r-- 1 lars users   343670846 Mar  6 21:16 Maps.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   344230905 Mar  6 21:18 Maps.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   971330020 Mar  6 21:19 Maps.2014.part01.xml.gz
-rw-r--r-- 1 lars users   313887801 Mar  6 21:21 Music.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   314584973 Mar  6 21:23 Music.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   843214097 Mar  6 21:25 Music.2014.part01.xml.gz
-rw-r--r-- 1 lars users   319653660 Mar  6 21:26 Music.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   320326381 Mar  6 21:28 Music.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   853042577 Mar  6 21:30 Music.2014.part02.xml.gz
-rw-r--r-- 1 lars users  4886872448 Mar  6 23:56 Names.2014.combined.marc8.gz
-rw-r--r-- 1 lars users  4911180917 Mar  7 00:24 Names.2014.combined.utf8.gz
-rw-r--r-- 1 lars users 12388696929 Mar  7 00:53 Names.2014.combined.xml.gz
-rw-r--r-- 1 lars users   143881637 Mar  6 22:21 Names.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   144557890 Mar  6 22:22 Names.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   376256048 Mar  6 22:22 Names.2014.part01.xml.gz
-rw-r--r-- 1 lars users   147732769 Mar  6 22:23 Names.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   148369572 Mar  6 22:24 Names.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   372804530 Mar  6 22:25 Names.2014.part02.xml.gz
-rw-r--r-- 1 lars users   146881502 Mar  6 22:25 Names.2014.part03.marc8.gz
-rw-r--r-- 1 lars users   147761163 Mar  6 22:26 Names.2014.part03.utf8.gz
-rw-r--r-- 1 lars users   373275684 Mar  6 22:27 Names.2014.part03.xml.gz
-rw-r--r-- 1 lars users   145373939 Mar  6 22:28 Names.2014.part04.marc8.gz
-rw-r--r-- 1 lars users   146127016 Mar  6 22:28 Names.2014.part04.utf8.gz
-rw-r--r-- 1 lars users   369282192 Mar  6 22:29 Names.2014.part04.xml.gz
-rw-r--r-- 1 lars users   145212695 Mar  6 22:30 Names.2014.part05.marc8.gz
-rw-r--r-- 1 lars users   145953182 Mar  6 22:30 Names.2014.part05.utf8.gz
-rw-r--r-- 1 lars users   371973849 Mar  6 22:31 Names.2014.part05.xml.gz
-rw-r--r-- 1 lars users   145844340 Mar  6 22:32 Names.2014.part06.marc8.gz
-rw-r--r-- 1 lars users   146731035 Mar  6 22:33 Names.2014.part06.utf8.gz
-rw-r--r-- 1 lars users   379654492 Mar  6 22:33 Names.2014.part06.xml.gz
-rw-r--r-- 1 lars users   139730108 Mar  6 22:34 Names.2014.part07.marc8.gz
-rw-r--r-- 1 lars users   140542064 Mar  6 22:35 Names.2014.part07.utf8.gz
-rw-r--r-- 1 lars users   353551399 Mar  6 22:35 Names.2014.part07.xml.gz
-rw-r--r-- 1 lars users   136203133 Mar  6 22:36 Names.2014.part08.marc8.gz
-rw-r--r-- 1 lars users   136783970 Mar  6 22:37 Names.2014.part08.utf8.gz
-rw-r--r-- 1 lars users   350890835 Mar  6 22:37 Names.2014.part08.xml.gz
-rw-r--r-- 1 lars users   131315036 Mar  6 22:38 Names.2014.part09.marc8.gz
-rw-r--r-- 1 lars users   131805101 Mar  6 22:39 Names.2014.part09.utf8.gz
-rw-r--r-- 1 lars users   327613055 Mar  6 22:39 Names.2014.part09.xml.gz
-rw-r--r-- 1 lars users   134580163 Mar  6 22:40 Names.2014.part10.marc8.gz
-rw-r--r-- 1 lars users   135348785 Mar  6 22:41 Names.2014.part10.utf8.gz
-rw-r--r-- 1 lars users   335096157 Mar  6 22:41 Names.2014.part10.xml.gz
-rw-r--r-- 1 lars users   127972301 Mar  6 22:42 Names.2014.part11.marc8.gz
-rw-r--r-- 1 lars users   129035112 Mar  6 22:43 Names.2014.part11.utf8.gz
-rw-r--r-- 1 lars users   316462193 Mar  6 22:43 Names.2014.part11.xml.gz
-rw-r--r-- 1 lars users   125377432 Mar  6 22:44 Names.2014.part12.marc8.gz
-rw-r--r-- 1 lars users   126177972 Mar  6 22:44 Names.2014.part12.utf8.gz
-rw-r--r-- 1 lars users   311720539 Mar  6 22:45 Names.2014.part12.xml.gz
-rw-r--r-- 1 lars users   119734364 Mar  6 22:46 Names.2014.part13.marc8.gz
-rw-r--r-- 1 lars users   120467491 Mar  6 22:46 Names.2014.part13.utf8.gz
-rw-r--r-- 1 lars users   303515618 Mar  6 22:47 Names.2014.part13.xml.gz
-rw-r--r-- 1 lars users   118218580 Mar  6 22:48 Names.2014.part14.marc8.gz
-rw-r--r-- 1 lars users   118981650 Mar  6 22:48 Names.2014.part14.utf8.gz
-rw-r--r-- 1 lars users   299863852 Mar  6 22:49 Names.2014.part14.xml.gz
-rw-r--r-- 1 lars users   127482102 Mar  6 22:49 Names.2014.part15.marc8.gz
-rw-r--r-- 1 lars users   128212778 Mar  6 22:50 Names.2014.part15.utf8.gz
-rw-r--r-- 1 lars users   316284204 Mar  6 22:50 Names.2014.part15.xml.gz
-rw-r--r-- 1 lars users   120132655 Mar  6 22:51 Names.2014.part16.marc8.gz
-rw-r--r-- 1 lars users   120990890 Mar  6 22:52 Names.2014.part16.utf8.gz
-rw-r--r-- 1 lars users   305355746 Mar  6 22:52 Names.2014.part16.xml.gz
-rw-r--r-- 1 lars users   121497164 Mar  6 22:53 Names.2014.part17.marc8.gz
-rw-r--r-- 1 lars users   122268210 Mar  6 22:53 Names.2014.part17.utf8.gz
-rw-r--r-- 1 lars users   306183075 Mar  6 22:54 Names.2014.part17.xml.gz
-rw-r--r-- 1 lars users   121604332 Mar  6 22:54 Names.2014.part18.marc8.gz
-rw-r--r-- 1 lars users   122514902 Mar  6 22:55 Names.2014.part18.utf8.gz
-rw-r--r-- 1 lars users   304168406 Mar  6 22:56 Names.2014.part18.xml.gz
-rw-r--r-- 1 lars users   118993765 Mar  6 22:56 Names.2014.part19.marc8.gz
-rw-r--r-- 1 lars users   119824991 Mar  6 22:57 Names.2014.part19.utf8.gz
-rw-r--r-- 1 lars users   293192329 Mar  6 22:57 Names.2014.part19.xml.gz
-rw-r--r-- 1 lars users   118402725 Mar  6 22:58 Names.2014.part20.marc8.gz
-rw-r--r-- 1 lars users   118993483 Mar  6 22:58 Names.2014.part20.utf8.gz
-rw-r--r-- 1 lars users   289940225 Mar  6 22:59 Names.2014.part20.xml.gz
-rw-r--r-- 1 lars users   123808537 Mar  6 22:59 Names.2014.part21.marc8.gz
-rw-r--r-- 1 lars users   124213381 Mar  6 23:00 Names.2014.part21.utf8.gz
-rw-r--r-- 1 lars users   312402021 Mar  6 23:01 Names.2014.part21.xml.gz
-rw-r--r-- 1 lars users   109872886 Mar  6 23:01 Names.2014.part22.marc8.gz
-rw-r--r-- 1 lars users   110191969 Mar  6 23:01 Names.2014.part22.utf8.gz
-rw-r--r-- 1 lars users   285322141 Mar  6 23:02 Names.2014.part22.xml.gz
-rw-r--r-- 1 lars users   134687884 Mar  6 23:03 Names.2014.part23.marc8.gz
-rw-r--r-- 1 lars users   135188738 Mar  6 23:03 Names.2014.part23.utf8.gz
-rw-r--r-- 1 lars users   349312546 Mar  6 23:04 Names.2014.part23.xml.gz
-rw-r--r-- 1 lars users   139656833 Mar  6 23:05 Names.2014.part24.marc8.gz
-rw-r--r-- 1 lars users   140429254 Mar  6 23:05 Names.2014.part24.utf8.gz
-rw-r--r-- 1 lars users   343070107 Mar  6 23:06 Names.2014.part24.xml.gz
-rw-r--r-- 1 lars users   130367583 Mar  6 23:07 Names.2014.part25.marc8.gz
-rw-r--r-- 1 lars users   130896729 Mar  6 23:07 Names.2014.part25.utf8.gz
-rw-r--r-- 1 lars users   333671234 Mar  6 23:08 Names.2014.part25.xml.gz
-rw-r--r-- 1 lars users   134496047 Mar  6 23:09 Names.2014.part26.marc8.gz
-rw-r--r-- 1 lars users   135319895 Mar  6 23:09 Names.2014.part26.utf8.gz
-rw-r--r-- 1 lars users   343687721 Mar  6 23:10 Names.2014.part26.xml.gz
-rw-r--r-- 1 lars users   131694295 Mar  6 23:10 Names.2014.part27.marc8.gz
-rw-r--r-- 1 lars users   132405240 Mar  6 23:11 Names.2014.part27.utf8.gz
-rw-r--r-- 1 lars users   337884658 Mar  6 23:12 Names.2014.part27.xml.gz
-rw-r--r-- 1 lars users   131068213 Mar  6 23:13 Names.2014.part28.marc8.gz
-rw-r--r-- 1 lars users   131739682 Mar  6 23:13 Names.2014.part28.utf8.gz
-rw-r--r-- 1 lars users   336182701 Mar  6 23:14 Names.2014.part28.xml.gz
-rw-r--r-- 1 lars users   128283863 Mar  6 23:14 Names.2014.part29.marc8.gz
-rw-r--r-- 1 lars users   128834138 Mar  6 23:15 Names.2014.part29.utf8.gz
-rw-r--r-- 1 lars users   328680794 Mar  6 23:16 Names.2014.part29.xml.gz
-rw-r--r-- 1 lars users   128980689 Mar  6 23:16 Names.2014.part30.marc8.gz
-rw-r--r-- 1 lars users   129466370 Mar  6 23:17 Names.2014.part30.utf8.gz
-rw-r--r-- 1 lars users   329206568 Mar  6 23:17 Names.2014.part30.xml.gz
-rw-r--r-- 1 lars users   129785810 Mar  6 23:18 Names.2014.part31.marc8.gz
-rw-r--r-- 1 lars users   130178917 Mar  6 23:19 Names.2014.part31.utf8.gz
-rw-r--r-- 1 lars users   328737993 Mar  6 23:19 Names.2014.part31.xml.gz
-rw-r--r-- 1 lars users   131094474 Mar  6 23:20 Names.2014.part32.marc8.gz
-rw-r--r-- 1 lars users   131502729 Mar  6 23:20 Names.2014.part32.utf8.gz
-rw-r--r-- 1 lars users   330980471 Mar  6 23:21 Names.2014.part32.xml.gz
-rw-r--r-- 1 lars users   135871719 Mar  6 23:21 Names.2014.part33.marc8.gz
-rw-r--r-- 1 lars users   136263324 Mar  6 23:22 Names.2014.part33.utf8.gz
-rw-r--r-- 1 lars users   341795239 Mar  6 23:23 Names.2014.part33.xml.gz
-rw-r--r-- 1 lars users   159542440 Mar  6 23:24 Names.2014.part34.marc8.gz
-rw-r--r-- 1 lars users   160035758 Mar  6 23:24 Names.2014.part34.utf8.gz
-rw-r--r-- 1 lars users   410421877 Mar  6 23:25 Names.2014.part34.xml.gz
-rw-r--r-- 1 lars users   145741314 Mar  6 23:26 Names.2014.part35.marc8.gz
-rw-r--r-- 1 lars users   146204757 Mar  6 23:26 Names.2014.part35.utf8.gz
-rw-r--r-- 1 lars users   370756365 Mar  6 23:27 Names.2014.part35.xml.gz
-rw-r--r-- 1 lars users   136505530 Mar  6 23:27 Names.2014.part36.marc8.gz
-rw-r--r-- 1 lars users   137112741 Mar  6 23:28 Names.2014.part36.utf8.gz
-rw-r--r-- 1 lars users   348055966 Mar  6 23:28 Names.2014.part36.xml.gz
-rw-r--r-- 1 lars users   119243589 Mar  6 23:29 Names.2014.part37.marc8.gz
-rw-r--r-- 1 lars users   119750038 Mar  6 23:29 Names.2014.part37.utf8.gz
-rw-r--r-- 1 lars users   301444099 Mar  6 23:29 Names.2014.part37.xml.gz
-rw-r--r-- 1 lars users  1574745266 Mar  6 21:38 Serials.2014.combined.marc8.gz
-rw-r--r-- 1 lars users  1577294477 Mar  6 21:45 Serials.2014.combined.utf8.gz
-rw-r--r-- 1 lars users   153000046 Mar  6 21:46 Serials.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   153339863 Mar  6 21:46 Serials.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   161216715 Mar  6 21:48 Serials.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   161394544 Mar  6 21:48 Serials.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   156260609 Mar  6 21:49 Serials.2014.part03.marc8.gz
-rw-r--r-- 1 lars users   156436261 Mar  6 21:50 Serials.2014.part03.utf8.gz
-rw-r--r-- 1 lars users   159641872 Mar  6 21:50 Serials.2014.part04.marc8.gz
-rw-r--r-- 1 lars users   159703811 Mar  6 21:51 Serials.2014.part04.utf8.gz
-rw-r--r-- 1 lars users   161607090 Mar  6 21:51 Serials.2014.part05.marc8.gz
-rw-r--r-- 1 lars users   161700029 Mar  6 21:52 Serials.2014.part05.utf8.gz
-rw-r--r-- 1 lars users   159352935 Mar  6 21:53 Serials.2014.part06.marc8.gz
-rw-r--r-- 1 lars users   160014445 Mar  6 21:54 Serials.2014.part06.utf8.gz
-rw-r--r-- 1 lars users   139575613 Mar  6 21:54 Serials.2014.part07.marc8.gz
-rw-r--r-- 1 lars users   140115302 Mar  6 21:55 Serials.2014.part07.utf8.gz
-rw-r--r-- 1 lars users   127774934 Mar  6 21:56 Serials.2014.part08.marc8.gz
-rw-r--r-- 1 lars users   127861052 Mar  6 21:56 Serials.2014.part08.utf8.gz
-rw-r--r-- 1 lars users   118919822 Mar  6 21:57 Serials.2014.part09.marc8.gz
-rw-r--r-- 1 lars users   119018637 Mar  6 21:57 Serials.2014.part09.utf8.gz
-rw-r--r-- 1 lars users   132217133 Mar  6 21:58 Serials.2014.part10.marc8.gz
-rw-r--r-- 1 lars users   132383827 Mar  6 21:59 Serials.2014.part10.utf8.gz
-rw-r--r-- 1 lars users   105178497 Mar  6 21:59 Serials.2014.part11.marc8.gz
-rw-r--r-- 1 lars users   105326706 Mar  6 22:00 Serials.2014.part11.utf8.gz
-rw-r--r-- 1 lars users   204494374 Mar  6 22:15 Subjects.2014.marc8.gz
-rw-r--r-- 1 lars users    96991643 Mar  6 22:16 Subjects.2014.part01.marc8.gz
-rw-r--r-- 1 lars users    97125788 Mar  6 22:16 Subjects.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   271018292 Mar  6 22:17 Subjects.2014.part01.xml.gz
-rw-r--r-- 1 lars users   107502731 Mar  6 22:17 Subjects.2014.part02.marc8.gz
-rw-r--r-- 1 lars users   107761308 Mar  6 22:18 Subjects.2014.part02.utf8.gz
-rw-r--r-- 1 lars users   262773760 Mar  6 22:18 Subjects.2014.part02.xml.gz
-rw-r--r-- 1 lars users   204887096 Mar  6 22:19 Subjects.2014.utf8.gz
-rw-r--r-- 1 lars users   533792052 Mar  6 22:20 Subjects.2014.xml.gz
-rw-r--r-- 1 lars users   309641400 Mar  6 21:12 Visual.Materials.2014.part01.marc8.gz
-rw-r--r-- 1 lars users   309869230 Mar  6 21:13 Visual.Materials.2014.part01.utf8.gz
-rw-r--r-- 1 lars users   810529940 Mar  6 21:14 Visual.Materials.2014.part01.xml.gz
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=D;O=A
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=D;O=D
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=M;O=A
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=M;O=D
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=N;O=A
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=N;O=D
-rw-r--r-- 1 lars users       41614 May 17 22:59 index.html?C=S;O=A
-rw-r--r-- 1 lars users       46087 May 17 17:57 index.html?C=S;O=D
-rw-r--r-- 1 lars users   328737993 May 17 17:41 test-Names.2014.part31.xml
-rw-r--r-- 1 lars users           0 Apr  4 19:47 text.txt

@ghost
Copy link

ghost commented May 29, 2017

We have completely different file sizes. What's the output of file BooksAll.2014.part01.utf8.gz?

@ghost
Copy link
Author

ghost commented May 29, 2017 

> file BooksAll.2014.part01.utf8.gz
BooksAll.2014.part01.utf8.gz: MARC21 Bibliographic
> sha256sum BooksAll.2014.part01.utf8.gz 
6a3bb078b8a905282419caa2e43f83ab6c93ea08cad6878ca8f2f0f3dd9a2029  BooksAll.2014.part01.utf8.gz
> ipfs add -n BooksAll.2014.part01.utf8.gz 
added QmP6Fnp7pJJ6R8ZXQ77c18HY9oF1nLnhF8mBXWrSuwyGTz BooksAll.2014.part01.utf8.gz

@ghost
Copy link

ghost commented May 29, 2017 •
edited by ghost
Loading

BooksAll.2014.part01.utf8.gz: gzip compressed data, was "BooksAll.part01.utf8", last modified: Wed Dec 9 14:21:29 2015, from Unix

As I suspected, your data is uncompressed - I don't know how.

root@ip-172-30-0-114:/tmp# gzip -d BooksAll.2014.part01.utf8.gz
root@ip-172-30-0-114:/tmp# file BooksAll.2014.part01.utf8
BooksAll.2014.part01.utf8: MARC21 Bibliographic

@ghost
Copy link
Author

ghost commented May 29, 2017

Mmmh that is weird. Maybe that was wget?

@ghost
Copy link

ghost commented May 29, 2017

Unknown but it'd be better to host a compressed copy for bandwidth reasons.

@ghost
Copy link
Author

ghost commented May 29, 2017

Okay I'm making another download and this time it seems correct:

> file loc.gov/cds/downloads/MDSConnect/BooksAll.2014.part01.utf8.gz 
loc.gov/cds/downloads/MDSConnect/BooksAll.2014.part01.utf8.gz: gzip compressed data, was "BooksAll.part01.utf8", from Unix, last modified: Wed Dec  9 15:21:29 2015

@ghost
Copy link
Author

ghost commented May 30, 2017

Okay, fixed dataset is here: https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1JAA7HX5

@ghost
Copy link
Author

ghost commented May 30, 2017

@voltagex did you get the same hash?

@ghost ghost mentioned this issue Jul 30, 2018
Open
13 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
0 participants