Import data from long-term database #208
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ed it in database.c) Signed-off-by: DL6ER <dl6er@dl6er.de>
…TL. Fall back to log interpretation if database is not available Signed-off-by: DL6ER <dl6er@dl6er.de>
Signed-off-by: DL6ER <dl6er@dl6er.de>
…rvers for the actual IPs. In think case we have to decrease the "cached" counter Signed-off-by: DL6ER <dl6er@dl6er.de>
…s counted Signed-off-by: DL6ER <dl6er@dl6er.de>
Signed-off-by: DL6ER <dl6er@dl6er.de>
Signed-off-by: DL6ER <dl6er@dl6er.de>
DL6ER
changed the title
AzureMarker
suggested changes
Jan 20, 2018
| queries[queryID].timeidx = timeidx; | ||
| queries[queryID].valid = true; // Mark this as a valid query (false = it has been garbage collected and should be skipped) | ||
| queries[queryID].db = true; // Mark this as already present in the database | ||
| queries[queryID].id = 0; // This is dnsmasq's internal ID. We don't store it in the database |
There was a problem hiding this comment.
Just to make sure, the dnsmasq query IDs don't start at 0, right?
| @@ -587,27 +578,21 @@ void process_pihole_log(int file) | |||
| continue; | |||
parser.c
Outdated
| @@ -587,27 +578,21 @@ void process_pihole_log(int file) | |||
| continue; | |||
| } | |||
|
|
|||
| // Count only if current query has not been counted so far | |||
| if(queries[i].complete) | |||
| continue; | |||
| extracttimestamp(readbuffer, &querytimestamp, &overTimetimestamp); | ||
|
|
||
| // Check if this query has already been imported from the database | ||
| if(querytimestamp < lastDBimportedtimestamp) continue; |
There was a problem hiding this comment.
There are more of this line below, but since this is run on every line, the others would just be redundant?
There was a problem hiding this comment.
Oh yes, they shouldn't be there, seems like I forgot to remove them at some places.
Signed-off-by: DL6ER <dl6er@dl6er.de>
AzureMarker
approved these changes
Jan 20, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By submitting this pull request, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your PR:
Please submit all pull requests against the
developmentbranch. Failure to do so will delay or deny your requestHow familiar are you with the codebase?:
10
This pull requests implements some general improvements for
FTL.Changes:
pihole.logandpihole.log.1each timeFTLstarts up, we read the already analyzed and categorized data from the long term database. We only read those queries frompihole.logthat are not already present in the database. This enhances the startup speed ofFTLas well as provides some functionality we will be using withFTLin the future.Forward Destinations (integrated)plot and is implemented to acknowledge the fact that we store only one forward destination per query in the database.Minor fixes:
<CNAME>record but still have to ask the upstream servers for the actual IP addresses, we don't count this query as being cached but as being forwarded. Note that this will reduce thelocalforward destination count (it has been too high so far).validate_access()calls at some places for a slight performance increase.This template was created based on the work of
udemy-dl.