forked from nZEDb/nZEDb
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Changelog
executable file
·371 lines (365 loc) · 30.1 KB
/
Changelog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
* Note *
No pull requests will be accepted without this file being updated to reflect what you changed and when.
No pull requests will be accepted to any branch except the dev branch.
Each update goes to the top of the file
2013-11-16 -> jonnyboy -> disabled misc/testing/DB_scripts/backfill_predb.php
added requestid lookup to releases_threaded.py, both per group and without
changed output for threaded postprocess additional [release.id] instead of [1], removed extra '!' echo in grabnzbs for replaced dupe
added maximum parts to download when usign GrabNZBs
added check for valid xml in check_nzbs.php
added unique index on predb md5
possible fix for utf-8 encode errors in py postprocess, added half second sleep after making an nntp connection, to allow nntpproxy time to wake up
2013-11-15 -> jonnyboy -> Added more regexes to namecleaning, added 2 functions to colorcli, fixed bugs
2013-11-14 -> niel -> Changed dates to use international format
2013-11-14 -> ThePeePs -> Added Script in testing/DB_scripts to convert to TPG.
2013-11-14 -> niel -> Conversion of require(_once) constructs to use path constants.
2013-11-13 -> niel -> Providing dynamic application base directory for shell scripts.
2013-11-11 -> jonnyboy -> moved all nntp connection responsiblity to the calling script, this allows for the same connection to be used throughout the process, it is closed only on error or process ends
2013-11-10 -> jonnyboy -> added clean option to movies/tv pp in tmux
added getAttribute function to db, removed rows_examined until I can determine db brand
added pane that could be used as a live 'slow query monitor' to tmux
2913-11-10 -> nuthead -> rewritten sql in file releases.php to give speed boost
2013-11-08 -> ThePeePs -> added option to choose more then one type of Crap Release rather then None/All/One (see tmux-edit page)
2013-11-06 -> jonnyboy -> added script to grab all shows from tvrage at 1 time
added option to add unreceived parts to partrepair when running safe threaded scripts, added partrepair run when running binary safe thread script
updated populate_anidb.php, but since i can not open anidb.net, I can not test the script, I assume I am permabanned
merged egant's pull request, it is untested, I spent an hour changing and fixing the pull request before I could merge it
merged Bart39's push
added option to allow grabnzbs to replace releases currently in the db
2013-11-05 -> jonnyboy -> added script to test a release name against all regexes, show regex and match
added more groups and regexes
2013-11-04 -> jonnyboy -> updated patchDB to handle errors and ignore certain errors
fixed schema.mysql and bad patches
2013-11-03 -> jonnyboy -> added some of the changes presented by niel
added option to set update releases threads when using tablepergroup
hopefully a fix for postdate, error found by tuxeh
added more regexes and groups to release cleaner
added missing lines to schema
added some indexes, added to the trigger, both to improve query performance, once I'm done going through the queries, some indexes will be removed
you may need to run the patch manually if something fails, the patch may not work with pgsql. I will try to set up a vm for pgsql to test the schema and all queries.
2013-10-25 -> jonnyboy -> changed renametopre.php to use releaseCleaner before added regexes
2013-10-24 -> jonnyboy -> begin importing nntpproxy by greenbender
2013-10-21 -> jonnyboy -> begin - adding option to use table per group
2013-10-19 -> jonnyboy -> added dry run option to clean_nzbs.php
begin using threaded fixreleasenames in tmux
changed monitor newest release to newest release posted, not newest release added
updated threaded scripts to process all of -1 before processing any -2
removed part repair threaded, it was slower than non threaded
removed nzb-import-bulk.php, it does not function properly with duplicates and pdo
added changes to part repair by Whitelighter
2013-10-16 -> jonnyboy -> created new table for safe scripts
improved output for check_usenet_groups.php
changed guid creation to improve speed
changed import scripts dupe detection, was copying nzbs even if release not added
fixed a couple of bugs in releases
2013-10-15 -> jonnyboy -> updated python script for category in pp additional
2013-10-14 -> jonnyboy -> added update to nzbstatus
fixed a few bugs
2013-10-13 -> jonnyboy -> allow requestid to work with relnamestatus 0 and 1
added new db connection to nzb-reog
fixed a few bugs in pgsql schema, added some regexes to renametopre.php
2013-10-11 -> jonnyboy -> fixed a few bugs(kaibosh)
2013-10-06 -> jonnyboy -> reverted my previous patch
removed check for dupe before release creation
2103-10-05 -> jonnyboy -> added my rename script, would be better that these reges were part of namefixer/namecleaner
2013-10-03 -> ThePeePs -> Added Script to update article numbers for groups when changing USP's rather than resetting all groups.
2013-10-03 -> jonnyboy -> changed update/insert queries with last_record and first_record from dec to string
stage 7b query 3 not needed to run every loop, it is slow, so added check to only run sometimes
fixed bug in releases.php not catching duplicates properly
hide getid3.php line 1641 errors
fixed a couple bugs in python scripts
2013-09-30 -> jonnyboy -> fixed binaries_safe_threaded.py
fixed many bugs
2013-09-25 -> jonnyboy -> added table for country codes, tvrage does not always return an abbreviation
added yesterdays patch to schema, correct patch for pgsql
2013-09-24 -> jonnyboy -> numerous changes queries, improved query speed, fixed some bugs
2013-09-21 -> jonnyboy -> fixed a few bugs
updated python scripts to use Psycopg, untested
updated install texts to reflect changes for Psycopg
updated binaries safe to start new groups
2013-09-19 -> jonnyboy -> fixed stage 4 thx kevin123
2013-09-18 -> jonnyboy -> bugfixes
2013-09-17 -> jonnyboy -> moved triggers out of schema and into separate file, fixed step2.php
made color pane optional
fixed bug in fixreleasenames par2
2013-09-16 -> jonnyboy -> added pron to monitor
changed python script to escape strings
bug fixes
fixed update releases showing insert release greater than actual inserted
2013-09-15 -> jonnyboy -> fixed a few bugs
added changes by bblanchon
added categorize to requestid, fixed a few bugs, added set relnamestatus = 6 when releaseCleaner cleans name
2013-09-14 -> jonnyboy -> added "S1 D1" names to regex for TV renaming
added par2 filecheck to fixreleasenames, fixed bug in parsePAR2
added option to set maximum per run in threaded fixreleasenames
removed typo from FAQ
2013-09-12 -> jonnyboy -> added update tv/movies and tmux_colors to tmux
added script to show colors available to tmux, this should help you remove bad colors
2013-09-11 -> jonnyboy -> changed query for pp to speed up query on large databases
2013-09-10 -> jonnyboy -> added option to postprocess movie/tv only renamed releases
added Safe Binary Threaded update, this maximizes thread usage for all groups, instead of running 1 thread to grab all from 1 group, this should be considered testing only, for now
added check for $nntp in parsePAR2
2013-09-09 -> jonnyboy -> updated backfill_groups.php and active_groups.php
rewrote the trigger to add another check
2013-09-08 -> jonnyboy -> fixed bad patch 119, added triggers for and added index
changed pp additonal first query to passwordstatus BETWEEN -2 and -1, seems that -1 and -1 or = -1 takes a very long time
fixed bug in grabnzbs where failed to get article deleted collection from collections, added more output to grabnzbs, added unique index to nzbs
fixed python script echo properlly when run from php script, other minor tweaks
2103-09-07 -> jonnyboy -> fixed bug where update_binaries not using compression after first grab
fixed bug where update_binaries not getting correct set of headers, when postdate fails initial attempt, may need to reset_truncate.php to be safe
fixed bug where update_binaries not running correctly on new groups(i hope)
fixed bug where parsePAR2 was not using alternate nntp, other minor fixes
2013-09-02 -> jonnyboy -> various, numberous little bug fixes
2013-08-31 -> jonnyboy -> added threaded versions for fixreleasesnames and update_releases, these should be considered alpha and could create invalid releases
cleaned up a few bugs
2013-08-29 -> jonnyboy -> changed queries for pg
update for changes to decrypt_hashes.php
typo in monitor.php
removed unneeded column from nzbs table
2013-08-29 -> kevin -> August 23 to 29:
Convert to php database objects.
Almost full support for postgresql (still needs work).
Now there is a seperate schema / patches for both database types, make sure to read the readme in db if you are submiting patches.
The install has been modified to support postgresql, but it has been locked for now while I keep working on it.
Start working on comment sharing throughout sites (ThePeePs will work on the rest with our help).
Fixes to par2 and toggle option to turn on and off adding the files to the filelist.
Changes to the install page (more checks and clearer text).
Changes to namefixer to not rerun the same release, also changes to relnamestatus, consolidating relstatus and adding more types.
Changes to backfill's postdate to fallback on the last known date or try a different article to get a date.
Other random stuff.
Various bug fixes.
2013-08-28 -> jonnyboy -> patch rename column
2013-08-27 -> jonnyboy -> reverted nntplib.py to one that works with both 2.* and 3.*
changed nzbs.group to nzbs.groupname
fixed clean_nzbs.php to remove nzbs and releases
2013-08-25 -> jonnyboy -> added regex to releaseCleaner that gets most predb style names
changed remove creap releases password to all negative search items and not remove releases that are applications/misc
added code to renumber content if new content ordinal is 1
2013-08-22 -> jonnyboy -> removed "added and r.nzbstatus = 0 to all stage 4.5 queries, since it is for deleting before the nzbs are created" and moved stage 4.5 out of the loop
updated stage4567 to run max 3 loops, would run indefinately before
updated missing items from alhpa theme site-edit/tmux-edit
added and r.nzbstatus = 0 to all stage 4.5 queries, since it is for deleting before the nzbs are created
added option to site-edit to allow user control of of delay time
updated predb matched to get from releases table
2013-08-21 -> kevin -> Predb changes, not the release holds the preid, the nzb details page has pre info and the browse/search pages have pre info.
2013-08-21 -> jonnyboy -> added color to h4 tag
possible fix for db lose connection in tmux when db is not local
added option for alternate sequential run in tmux
2013-08-21 -> kevin -> Add memcache support. Add the following under define('NNTP_SSLENABLED_A', false); in your config.php. Set to true to enable. (for ubuntu -> sudo apt-get install memcached php5-memcache)
// Wether to use memcached or not.
define('MEMCACHE_ENABLED', false);
define('MEMCACHE_HOST', '127.0.0.1');
define('MEMCACHE_PORT', '11211');
// Amount of time to keep a query in ram in seconds.
define('MEMCACHE_EXPIRY', '900');
// To compress the queries using zlib or not (more cpu usage and less ram usage if set to true, inverse for false);
define('MEMCACHE_COMPRESSION', true);
Also edit /etc/php5/apache2/php.ini, add extension=memcache.so to the dynamic extensions section and restart apache2.
2013-08-20 -> kevin -> 08 17 to 08 20 : More clean up, convert the simple theme, more regex for matching etc.., fix queries for poster/subject searching, change movie.php to catch movies better.
2013-08-16 -> kevin -> Fix more issues in postprocess.php
More checks on hidden nfos to see if it's an NFO or not.
2013-08-15 -> kevin -> 08 13 to 08 15 : Clean up files in www/lib, add fetching names from PAR2 files.
2013-08-15 -> ThePeePs -> Site stats for alpha theme.
2013-08-15 -> Josh -> Update themes.
2013-08-12 -> bodrick -> Fixed bug with namefixer, if print changes only, status was still being set to updated.
2013-08-12 -> kevin -> Catgorize S01E01E02 / S01E01A properly, clean up more files.
2013-08-11 -> kevin -> Clean up/fix bugs in a bunch of files in www/lib
2013-08-10 -> kevin -> Clean up postprocess.php
2013-08-09 -> kevin -> More regex in namecleaning.
2013-08-08 -> kevin -> Fix a bug in releases.php where releases under minimum file/size setting were not deleted.
Add more regex for matching/name cleaning.
2013-08-07 -> kevin -> 7zip to decompress nfos froms zips during post process (/usr/bin/7za on ubuntu, 7z command line windows(untested on windows)).
Fix many bugs in post proc.
2013-08-06 -> sinfulj -> Add a new theme (currently in alpha testing).
2013-08-05 -> kevin -> Clean up postprocess.php, make sure there are no duplicate names for releaseID's in the releasefiles table, more regex in namecleaning.
2013-08-05 -> wtf911 -> Attempt to fix duplicate tvrage entries when running threaded.
2013-08-04 -> sinfulj -> Fix up some theme issues.
2013-08-04 -> kevin -> Add more regex, fix up a few issues, reset collection hash again with these changes.
2013-08-03 -> kevin -> Update namecleaning, requires a reset of the collection hash.
2013-08-02 -> kevin -> When many releases are added at the same time the hash for the guid can be the same, so add mt_rand while hashing.
Clean up postprocess.php a bit.
2013-08-01 -> ThePeePs -> Add logging of failed site login attempts.
2013-08-01 -> kevin -> Add more regex to namecleaning.
2013-07-30 -> kevin -> Clean up binaries.php and backfill.php and attempting to fix the connection errors.
Modify files that use namecleaning to send the groupID.
Modify namecleaning collectionsCleaner to be notified if there are files or not.
Drop the name column on the collections table since it is useless now.
2013-07-20 -> jonnyboy -> added release name to output of decrypt_hashes.php
added release count per group to active_groups.php
fixed bug in predb with zenet
added 5 indexes to releases to improve postprocess queries
improved query time in monitor
2013-07-19 -> jonnyboy -> fixed bug in postprocess(windows)
fixed bug in safe backfill, was adding parts to partrepair, you many need to run: truncate table partrepair;
fixed bug in predb, you will need to run this query: update predb set title = replace (title, ' - omgwtfnzbs.org', '');
2013-07-18 -> jonnyboy -> added alternate nntp to install
fixed bug in orlydb
fixed bug in threaded postprocess
attempt to fix sab view queue
2013-07-16 -> jonnyboy -> changed output for nzb errors during nfo processing
fixed bug in threaded import using tmux
fixed pass by reference error
added group check in binaries scan
typo in postprocess.php
fixed bug in grabnzbs
fixed bug in nfo nntp connection
added option to import gzipped nzbs, not working in www/admin/nzb-import.php
changed import copy file to save stream, increased import speed
fixed typo in namecleaning.php
2013-07-15 -> jonnyboy -> replaced external decrypt hashes script with internal one
added preg_replace to grabnzbs and nzb-import to remove part number from end of subject
2013-07-15 -> kevin -> update namecleaning Added some regex to name cleaning.
Make sure that yEnc is before part count in binaries.php (some High Compressed virus posts were getting through).
2013-07-14 -> jonnyboy -> reverted last
changed default user view to list view
updated rarinfo libs
added additional monitor paths in tmux - request
added limit to requestID per run
added powerline segment for NNTP connections - ThePeePs
added lines to monitor to show NNTP connections - Mike1
2013-07-13 -> jonnyboy -> gave GrabNZBs its own thread count
fixed type errors
typos
cleanup backfill.php
2013-07-13 -> Junkman690 -> readd RSS namespace to make the rss feeds work again. Make it point to nZEDb install for info
2013-07-12 -> jonnyboy -> added NNTP error checking in postprocess and nfo
nzb-import and grabNZBs now use site edit Crosspost time
fixed a bug when switch nntp provider and groups not available or invalid first_post
2013-07-11 -> jonnyboy -> tmux -added option to use removecrapreleases all or just 1 setting
fixed bug in postprocess_threaded.py
2013-07-11 -> jonnyboy -> added 3 loop limit to releases stage4567
updated nntplib.py, fixed a bug when switch nntp provider and groups not available
fixed bug in tmux-seq when not running any backfill
fixed bug in postback, when reconnecting to nntp, need to set group name first
2013-07-11 -> kevin -> added the new regexes to namecleaning releaseCleaner, removed the "universal" one from the groups.
changed the group regexes in releaseCleaner to make sure they don't match on other groups.
make sure exe doesn't get into tv.
Added alt.binaries.multimedia.anime to releaseCleaner.
Changed collectionsCleaner so in the future it can do groups.
2013-07-11 -> jonnyboy -> added 'trim' to releases name in nzb-import and grabNZBs
fixed bug in postprocess.php
beginning the move of pre/requestID matching to release creation, instead of separate process later
2013-07-10 -> jonnyboy -> fixed bug in backfill causing some issue with nntp stall
moved python lib files
2013-07-09 -> jonnyboy -> added limit clause when running populate gids from tmux
added query to grabnzbs that appear incomplete but are older than 2 hours
fixed bug in grabznbs creating gid at same time as update_releases
changed poster to fromname delete_releases.php
added nzbstatus check for gid creation
added db connect string to start of loop in monitor.php, should help the with disabled by running issues
updated start.php to latest patch
2013-07-07 -> jonnyboy -> added update_relases run when have exceeded limits in seq
updated all python scripts, put closing stuffin in main(), removed thread deamon's was causing shutdown issues with 2.*
added gid to grabzbs and import threaded scripts, still need to add to non threaded
updated monitor predb to match the other totals, fixed typo in populate_nzb_guid.php
tmux - removed unneeded/unused timers
2013-07-07 -> Miatrix -> Added populate_nzb_guid.php in the misc/testing/DB_scripts folder to update existing releases with the guid
2013-07-07 -> Miatrix -> updated the creation of releases to populate the nzg_guid column with a md5 hash based on the 1st post. This will allow possible spotnab integration at a later date.
2013-07-07 -> jonnyboy -> updated monitor pane for requestID
integrated threaded requestID into releases.php, no need to run separate python script
fixed testing/nzb-import.php not deleting the nzbs
fixed skipping releases when importing by filename
added more rss feeds for predb.me
2013-07-06 -> jonnyboy -> fixed patch
Added threaded requestID for initial pull
2013-07-06 -> Miatrix -> Added Request ID Lookup. This looks up requests from the URL we provide to fix naming where possible.
This is very usefull for pron, but helps sometimes on tv/movies/console releases.
2013-07-06 -> jonnyboy -> removed remove stale folders from threaded pp additional, wrong place
2013-07-05 -> jonnyboy -> fixed bug in tmux-edit and monitor
fixed bug in postprocess
a little cleanup, removed optimize/patch from tmux-edit
added formatting to numbers in safe backfill
tmux - split pp addtional and pp nfo
added partrepair threaded
added remove stale folders from threaded pp additional
2013-07-05 -> jonnyboy -> really fixed it, updated tmux-edit with an example of the formula used
fixed issue with safe backfill threaded, not using the correct number of threads
fixed issue with postprocess threaded nfo
2013-07-05 -> Miatrix -> Added 2 options to control the video sample to the site edit screen.
One is segments to download. If you find your samples arn't long enough you can increase this though it will mean that you
download more from your usenet provider.
The other option is the max length (in seconds) for the preview.
Added 1 option to control the sample image. When in seconds to try to grab the image.
Fixed an issue where the same files were listed multiple times in the releasefiles table. This
was due to the code processing the 1st and last rar file of the archive.
2013-07-05 -> jonnyboy -> updated schema for missing patches
2013-07-04 -> kevin -> Update the FAQ.
2013-07-04 -> devmem -> Reverted changes in categorisation and added HD movie check to a.b.movee
Altered ordering of if() in namefixer to improve efficiency
2013-07-04 -> jonnyboy -> removed nntp lib from pp python script
removed optimize and patchdb from tmux scripts
cymysql has been updated to use sockets, python scripts also, you will need to:
please run everything, not just the next line
run sudo pip install --upgrade cymysql -or-
run sudo easy_install --upgrade cymysql
run sudo pip-3.2 install --upgrade cymysql
run sudo pip-3.3 install --upgrade cymysql
to use sockets, add (be sure the path is correct)
define('DB_SOCKET', '/var/run/mysqld/mysqld.sock'); to you www/config.php
or, not use sockets, add
define('DB_SOCKET', ''); to you www/config.php
2013-07-03 -> jonnyboy -> fixed imported nzbs not pp
fixed usefilename to use filename instead of clean name
fixed bug in omport threadd when path has no subfolders
fixed bug in import true
added patch to update for new setting
added option to use filenames with nzb-import - tmux
added option to use alternate nntp provider for grabnzbs
updated python nnpt lib
added new connection, if error
removed if connected when definately not connected
2013-07-02 -> jonnyboy -> added small delay around python connection
removed nntp connect where not needed, removed delay between doQuit and doConnect
added a small delay between doQuit and doConnect, in a attempt to allow the connection to terminate before trying a new connection.
2013-07-01 -> jonnyboy -> added variable timeout to nntp.php
2013-06-30 -> jonnyboy -> changed nzb retention for GrabNZBs to partretentionhours
added script to show sleeptime using Consoletools
added dateadded column to nzbs table, added code to delete incomplete nzbs older than 4 hours
added nzbs to monitor, total(distinct), pending
started adding code to install scripts for socket and alternate nntp
2013-06-29 -> ugo -> added language flags to movie page
updateded test_misc_sorter.php
2013-06-29 -> jonnyboy -> added script to create a ramdisk and move mysql tables to it
modified file to delete releases based on poster, searchname, name or guid, using = or like
added simple script to run all threaded scripts sequentially
removed hard coded db name from clean_nzb, added safety argv
made tmux colors array user defineable
changed query for music to process
added output to start of threaded scripts
separated decrypt hashes and predb
fixed grabnzbs not setting haspreview = -1
fixed backfill safe loop over completed groups
changed progessive sleep time to use min sleep timer if timer would be less than that
renamed patch
added blank line to robot.txt
replaced && with ; because python scripts exit and sleeps were not working
changed echo output in postprocess doecho and/or debug
changed patch numbers
added cats to predb
removed hard coded db name from patch
2013-06-28 -> ugo -> added code to check by the user for broken releases
added robot.txt files to deny all indexers by default
some minor changes to other parts
2013-06-28 -> jonnyboy -> backfill safe will loop to next group, if available posts is less than 10000
2013-06-28 -> devmem -> Fixed removeCrapReleases to also process when argv[1] is false
2013-06-28 -> devmem -> Added extra column 'relstatus' to table 'releases' to enable bitwise ops on releases
Edited lib/framework/db.php to start adding constants for relstatus
Updated namefixer to only process releases not processed before
Corrected jonnyboys dates (he was 1 month ahead - it's still June)
2013-06-28 -> jonnyboy -> backfill safe will loop to next group, if available posts is less than 10000
oops, removed sys.exit from safe backfill
added back changes to site-edit
rewrote all python scripts
added back changes to releases.php
added output back to postprocess
reverted some changes looking for nntp hang bug
2013-06-27 -> jonnyboy -> may have fixed some connection issues
added legend to output in postprocess
removed second rm -rf, bad placement
changed fixreleasenames to use inner join insteaqd of left join
changed tmpunrar folder creation to chmod 0777 from 0764, just in case
moved delete stale tmpunrar from threaded script to monitor
* Note *
No future pull requests will be accepted without this file being updated to reflect what you changed and when.
No pull requests will be accepted to any branch except the dev branch.
Each update goes to the top of the file