Add State Attorney General Opinion Scrapers

[mlissner]

Two top level tasks here:

• Trawl the internet and find all the available sources.

• Make the scrapers.

I'll develop a list below of all scrapers we want to build.

[arderyp]

Maybe this should be a living ticket where we update the list of new scrapers to add. If so, maybe we could rename this issue, or create a new issue to handle the living list. Do you think we should create separate child issues for each new scraper we want to add, so we can close those when done? It would be helpful when adding to our living list to indicate:

1. Resource URL
2. Desired module name for new scraper (juriscraper.opinions.whatever.new_scraper_name)
3. Link to child issue for this specific new scraper, if you think child issues is something you want

Maybe it could be a 3 column table that we add new desired scrapers to with the info above. If this sounds reasonable, could you update this issue (or create new issue for living list) to include this new format/info for your Maryland example above? And if we want this to be a living issue with links to sub issues, could you also add the new scrapers from #167 to the list here with the pertinent 1-3 info mentioned above?

[mlissner]

I like the idea of having one ticket per court, but obviously I occasionally make bigger tickets like this one. We've also used the github wiki in the past for this purpose (I think there's still a page with a gazillion court links).

Dunno. I'd say let's keep this one focused on State AG. Closing tickets is nice, and keeping them short is nice too.

So, in that vein, here's the info we need:

Done	State	Link	Module Name
☐	Alabama	http://www.ago.state.al.us/Opinions.aspx	alaag
☐	Alaska	http://www.law.state.ak.us/doclibrary/opinions-index/opinions_chron.html	arkag
☐	Arizona	https://www.azag.gov/ag-opinions	azag
☐	Arkansas	http://www.arkansasag.gov/opinions/index.php	arizag
✓	California	https://oag.ca.gov/opinions	calag
☐	Colorado	https://coag.gov/resources/formal-ag-opinions	coloag
☐	Connecticut	http://www.ct.gov/ag/cwp/browse.asp?a=1770	connag
☐	Delaware	http://opinions.attorneygeneral.delaware.gov/	delag
☐	D.C.	https://oag.dc.gov/page/oagcorporation-counsel-opinions	dcag
☐	Florida	http://myfloridalegal.com/opinions	flaag
☐	Georgia	http://law.ga.gov/opinions	gaag
☐	Guam	http://www.guamag.org/ (but down at present)	guamag
☐	Hawaii	http://ag.hawaii.gov/publications/opinions/	haag
☐	Idaho	http://www.ag.idaho.gov/publications/op-guide-cert_index.html	idahoag
☐	Illinios	http://www.ag.state.il.us/opinions/	illag
☐	Indiana	http://www.in.gov/attorneygeneral/2352.htm	indag
☐	Iowa	https://www.iowaattorneygeneral.gov/about-us/attorney-general-opinions/	iowaag
☐	Kansas	http://ag.ks.gov/media-center/ag-opinions	kanag
☐	Kentucky	http://ag.ky.gov/civil/civil-enviro/opinions/Pages/default.aspx	kyag
☐	Louisiana	https://www.ag.state.la.us/Opinions	laag
☐	Maine	https://www.maine.gov/ag/about/ag_opinions.html	meag
✓	Maryland	http://www.marylandattorneygeneral.gov/Pages/Opinions/index.aspx	mdag
☐	Mass.	http://www.mass.gov/ago/government-resources/ags-opinions/	massag
☐	Michigan	http://www.ag.state.mi.us/opinion/opinions.aspx	michag
✓	Minnesota	http://www.ag.state.mn.us/office/opinions/	minnag
☐	Mississsippi	https://govt.westlaw.com/msag/Index?__lrTS=20170120220617535 (review TOS?)	missag
☐	Missouri	https://ago.mo.gov/other-resources/ag-opinions	moag
☐	Montana	https://dojmt.gov/agooffice/attorney-generals-opinions/	montag
☐	Nebraskta	https://ago.nebraska.gov/ag_opinion	nebag
☐	Nevada	http://ag.nv.gov/Publications/Opinions/	nevag
☐	New Hampshire	http://www.doj.nh.gov/media-center/opinions.htm	nhag
☐	New Jersey	http://nj.gov/oag/ag-opinions.htm	njag
☐	New Mexico	http://public-records.nmag.gov/opinions	nmag
✓	New York	https://ag.ny.gov/appeals-and-opinions/numerical-index	nyag
☐	N. Carolina	http://www.ncdoj.gov/getdoc/683dc0b7-ad27-4dbd-8ea5-c9109ff016a0/Legal-Opinions.aspx	ncag
☐	N. Dakota	https://attorneygeneral.nd.gov/attorney-generals-office/legal-opinions/opinion-search	ndag
☐	N. Mariana Islands	No online presence found	nmiag
☐	Ohio	http://www.ohioattorneygeneral.gov/About-AG/Service-Divisions/Opinions/Opinions-Archive	ohag
☐	Oklahoma	https://www.ok.gov/oag/Legal_Resources/AG_Opinions.html	oklaag
☐	Oregon	http://www.doj.state.or.us/agoffice/pages/index.aspx	orag
☐	Penn.	https://www.attorneygeneral.gov/The_Office/Official_Attorney_General_Opinions/	paag
☐	Puerto Rico	http://www.justicia1.pr.gov/ordenesa/opiniones.aspx (spanish)	prag
☐	Rhode Island	Homepage, but cannot find opinions: http://www.riag.ri.gov/	riag
☐	S. Carolina	http://www.scag.gov/opinions	scag
☐	S. Dakota	http://atg.sd.gov/OurOffice/OfficialOpinions/opinions.aspx	sdag
☐	Tennessee	https://www.tn.gov/attorneygeneral/topic/attorney-general-opinions	tennag
✓	Texas	https://texasattorneygeneral.gov/opinion/index-to-opinions	texag
☐	Utah	1969-present on microfiche, but that's it! https://archives.utah.gov/research/inventories/20369.html	utahag
☐	Vermont	There's 12 between 2000 and today: http://ago.vermont.gov/divisions/about-the-attorney-generals-office/attorney-general-opinions.php More here: http://libguides.vermontlaw.edu/vermontlawguide/vermontadministrativelaw and here: http://llmc.com/titledescfull.aspx?type=6&coll=49&div=195&set=08140 (paywall?)	vtag
☐	Virgin Islands	Homepage, but no opinions: http://usvidoj.codemeta.com/DivisionContent_1.php?divId=84	viag
☐	Virginia	http://ag.virginia.gov/citizen-resources/opinions/official-opinions	vaag
☐	Washington	http://www.atg.wa.gov/AGOOpinions/opinion	washag
☐	West Virginia	http://www.ago.wv.gov/publicresources/Attorney%20General%20Opinions/Pages/default.aspx	wvaag
☐	Wisconsin	https://docs.legis.wisconsin.gov/misc/oag	wisag
☐	Wyoming	http://ag.wyo.gov/formal-opinions	wyoag

I'll also just point out that the module name will almost always correspond with the ID in CourtListener.

[mlissner]

If you're new here and can help, please say which scraper you're able to work on, and check out the readme to get started.

[arderyp]

dear lord! Is there any strategy here, or just start working from the top?

I'll take care of the others in #167 first so we can close that ticket.

opinions/united_states/federal_special/ag.py
opinions/united_states/state_special/mdag.py

[mlissner]

The right way to do it is to start with the most populous states. The fun way is to find easy ones with big archives that are easy to traverse.

On Fri, Jan 20, 2017, 21:21 Philip Ardery ***@***.***> wrote: dear lord! Is there any strategy here, or just start working from the top? I'll take care of the others in #167 <#167> first so we can close that ticket. opinions/united_states/federal_special/ag.py opinions/united_states/state_special/mdag.py — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOdqkV4UHrRG7GbmzF7goHrjiJHyVrgks5rUZW-gaJpZM4LTKNz> .

-- Mike Lissner Executive Director Free Law Project https://free.law

[dmvj]

I don't know if this is useful or not, but the Alabama AG has opinions going back to 1979. They're numbered differently depending on period:

When searching by opinion number use the following formats:
Opinions 79-00001 to 95-00338, use format yynnnnn (ex: 8500001)
Opinions 96-00001 to 99-00290, use format yy-nnnnn (ex: 96-00331)
Opinions 2000-001 to Present, use format yyyy-nnn (ex: 2000-025)

And these are the correct ranges from 1994-2000:

9400001 - 9400267
9500001 - 9500338
96-00001 - 96-00331
97-00001 - 97-00298
98-00001 - 98-00225
99-00001 - 99-00290
2000-001 - 2000-252

The PDFs are named according to the above scheme. So, examples from the three different date formats:

https://www.alabamaag.gov/Documents/opin/9400001.pdf
https://www.alabamaag.gov/Documents/opin/97-00001.pdf
https://www.alabamaag.gov/Documents/opin/2000-003.pdf

[mlissner]

That's great. One day, perhaps, we'll get on this, but absent a volunteer picking it up, it's outside of our budget to do this work for the moment.

[dmvj]

I'm just learning Python, so maybe if I ever get to a place where I understand *args and **kwargs, I can help. But at least the information's there now. :)

[flooie]

Notes on AGs

Missing

• Massachusetts, - no longer issued

• Mississippi - Westlaw

• Iowa - Westlaw

• Rhode Island - None Found

• Utah - None Found

• Wyoming - Google drive

• New Mexico - None Found

• Virgin Islands - None found

• Puerto Rico - None found

• Guam - Google drive

I suppose the google drives could maybe be tackled with selenium but I wasn't up for figuring that out.
Massachusetts does issue opinions on meetings - but its not the same thing.

Rhode Island is a mystery because I do think they exists but I dont know where.

Future Notes

This was the bare minimum - and didnt set up for back scraping opinions.
Also It looks to me that lots of offices have scaled back - or they post these very intermittently.
Some may not post in years and some haven't posted in years - but I set up the scrapers anyway.

[flooie]

I also moved all the AG scrapers into a new folder juriscraper/opinions/united_states/attorney_general

And moved all the previously added ones into that directory