Command line vs config file

[Joeweav]

I may have a complicates set up here, I have a bunch of groups that I pick from every day out of a scheduled task. I have been around the block with this a few times and while it is less efficient I find it works better if I use a for loop to cycle through a text file of the subs that I want to harvest. That way if something hangs and I have to manually kill it, I only lose that sub from that point on. Stacking thing up on the command line it seemed that I never made it to the end of all the stacked subs. So that problem seems to be solved and I am happy with the solution.

Next, I figured I would start logging and perhaps figure out what the offenders are so I created a sub directory and used the --log option to have bdfr leave log file for each of the subs in there. I figured when it hung I could go to the end of the current log file and figure out what the offending post id was and exclude it. The logging seems to work well though I am not sure when the logs will roll or if I am going to have to deal with that manually. For now I am happy and if I have to trim them off after some number manually that too is not a big deal.

However, with the way I really only have one command line and I loop, if I put the exclusions on the command line itself, that is going to get out of hand pretty quickly. I may also want to play with some other options so I started looking at the --config option. And I found one sub where it hung, I went to the bottom of the list and found the item it was trying to grab and added one line to that subs config file, --exclude-id XXXX and I figured that was the last time it would hang on that one. But no, it just did it again. Am I missing something here? I created empty config files for all of the subs and only one of them has one --exclude-id XXXX line in it. Why is this not working? Can you not mix a config file with command line options?

For the curious this is what the command line looks like...

for /f %%g in (\192.168.1.123\reddit\subreddit.txt) do python -m bdfr download \192.168.1.123\reddit --subreddit %%g --no-dupes --config \192.168.1.123\reddit\bdfr\cfgs%%g --log \192.168.1.123\reddit\bdfr\logs%%g.log

Where..
\192.168.1.123\reddit is a share with a big disk on a fast network
\192.168.1.123\reddit\subreddit.txt is a list of subs with one per line
\192.168.1.123\reddit us were all the stuff goes in it's default folder and file naming format.
\192.168.1.123\reddit\bdfr\cfgs\ is were all of the config files live for each of the subs
\192.168.1.123\reddit\bdfr\logs is where all the log files live for all of the subs

I also have a solution that builds up one monster command line from a list of subs, if anybody wants that to poke at, but because of the hanging issues I can not recommend it as a good solution.

[Serene-Arc]

First off, we have an option called --exclude-id-file which will read all the IDs from a text file, one per line, and then remove all of them from the download process so any errors don't occur again and the time is saved.

Config files can only have a certain few fields in it. --exclude-id is not one of them, it is a command-line option. If you add it to the command, then that submission will be skipped, but in the config file it will do nothing.

[Joeweav]

Thank you for the reply!

I will have to read over the docs again and try and parse out the options that are viable for download and which ones can go in the config file and which ones need to be on the command line.

Is --exclude-id-file a command line or a confg file option or both? I guess for now given I only want to exclude ID's so far I could change the config file to and exclude file, but it would seem more proper, and no doubt slower over the net, to have the config file point to the exclude file.

[Serene-Arc]

There are very few things that can go in the config file, pretty much nothing that you need to worry about. If it starts with two hyphens, then it's a command-line option.