Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use shared memory objects for data #429

Merged
merged 59 commits into from
Dec 12, 2018
Merged

Use shared memory objects for data #429

merged 59 commits into from
Dec 12, 2018

Commits on Aug 1, 2018

  1. Store forward destination IP address and host name in contiguous stri… 

    …ng buffer

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 1, 2018
    Configuration menu
    Copy the full SHA
    3e88490 View commit details
    Browse the repository at this point in the history

  2. Use strbuffer also for client IPs and host names 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 1, 2018
    Configuration menu
    Copy the full SHA
    b7da12b View commit details
    Browse the repository at this point in the history

  3. Use strbuffer also for domains 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 1, 2018
    Configuration menu
    Copy the full SHA
    96928c7 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2018

  1. Rename last_pos to next_pos and use struct for string buffer 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    1a508fc View commit details
    Browse the repository at this point in the history

  2. Store size in string buffer + use long long for string position indices 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    d7d8518 View commit details
    Browse the repository at this point in the history

  3. Add shared memory components to store strings in shared memory 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    1b7e084 View commit details
    Browse the repository at this point in the history

  4. Improve shmem error outputs 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    bb51f4e View commit details
    Browse the repository at this point in the history

  5. Allocate shared memory in units of pages 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    1 Configuration menu
    Copy the full SHA
    2e12558 View commit details
    Browse the repository at this point in the history

  6. Unify two if statements into one + allocate already one page size ini… 

    …tially

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    f1ba139 View commit details
    Browse the repository at this point in the history

  7. Also store domains struct in shared memory object 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    0ced77c View commit details
    Browse the repository at this point in the history

  8. Also store clients struct in shared memory object 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    249d236 View commit details
    Browse the repository at this point in the history

  9. Also store forwarded struct in shared memory object 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    2 Configuration menu
    Copy the full SHA
    584c23d View commit details
    Browse the repository at this point in the history

  10. Also store queries struct in shared memory object 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 2, 2018
    Configuration menu
    Copy the full SHA
    5d22aa7 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2018

  1. Fix wrong pointer reference 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    edf7461 View commit details
    Browse the repository at this point in the history

  2. Fix wrong error message, remove duplicated resized message, and don't… 

    … assume char* pointer type (use void* and cast at the right places if needed)

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    74d25d1 View commit details
    Browse the repository at this point in the history

  3. Use munmap + mmap instead of mremap when enlarging shared memory objects 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    0003a41 View commit details
    Browse the repository at this point in the history

  4. Move query type determination further up 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    77f932b View commit details
    Browse the repository at this point in the history

  5. First munmap() the shared memory object, thereafter resize it (using … 

    …ftruncate), and only then mmap() it again

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    f40d8b1 View commit details
    Browse the repository at this point in the history

  6. man shm_open recommends to use shared memory object names starting wi… 

    …th "/" for portability

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    f72a4cd View commit details
    Browse the repository at this point in the history

  7. Re-open shared memory object when resizing the object using ftruncate… 

    …(). shm_open() automatically sets FD_CLOEXEC which closes the file handler automatically whenever something is calling system() to avoid leaking the file descriptors to random programs. The file descriptor becomes invalid and later mmap operations fail. We always use shm_open() (without the O_CREAT attribute) every time before we run mmap to ensure we always have a valid file descriptor.

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 3, 2018
    Configuration menu
    Copy the full SHA
    9f275a0 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2018

  1. Allocate more struct memory earlier 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Aug 4, 2018
    Configuration menu
    Copy the full SHA
    eb1d851 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2018

  1. Merge branch 'development' into new/shmem 

    # Conflicts:
#	api.c
#	dnsmasq_interface.c
    @AzureMarker
    AzureMarker committed Sep 2, 2018
    Configuration menu
    Copy the full SHA
    bbfe186 View commit details
    Browse the repository at this point in the history

  2. Fix compile error due to merge 

    The merged in code tried to use normal strings instead of shared memory
strings.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 2, 2018
    Configuration menu
    Copy the full SHA
    992726d View commit details
    Browse the repository at this point in the history

  3. Fix getstr parameter warning 

    unsigned int is smaller than long long, which all of the string position
data was using. This generated a lot of warnings.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 2, 2018
    Configuration menu
    Copy the full SHA
    99c4e40 View commit details
    Browse the repository at this point in the history

  4. Use unsigned long long for string positions 

    Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 2, 2018
    Configuration menu
    Copy the full SHA
    3585043 View commit details
    Browse the repository at this point in the history

  5. Only use a single NULL terminator in strings shared memory 

    Previously two NULL bytes were used between strings, which was not
intended.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 2, 2018
    Configuration menu
    Copy the full SHA
    d96c009 View commit details
    Browse the repository at this point in the history

  6. Fix shared memory resize not being remembered 

    The new size was not remembered after resizing, causing it to never
resize larger than two 4k blocks of items.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 2, 2018
    Configuration menu
    Copy the full SHA
    e91369f View commit details
    Browse the repository at this point in the history

  7. Add stdbool include to shmem.h 

    Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 2, 2018
    Configuration menu
    Copy the full SHA
    8401c9b View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2018

  1. Move counters to shared memory 

    This requires that it be accessed through a pointer, so most of the
changes are just changing `counters.` to `counters->`

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 3, 2018
    Configuration menu
    Copy the full SHA
    ae1bb5b View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2018

  1. Merge remote-tracking branch 'origin/development' into new/shmem 

    # Conflicts:
#	dnsmasq_interface.c
#	resolve.c
    @AzureMarker
    AzureMarker committed Sep 17, 2018
    Configuration menu
    Copy the full SHA
    3d439ad View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2018

  1. Use negative query IDs for queries loaded from the database 

    This is necessary for all queries to have unique IDs. Previously,
all queries loaded from the database had an ID of zero.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 18, 2018
    Configuration menu
    Copy the full SHA
    e1c8ccc View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2018

  1. Merge pull request #382 from pi-hole/feature/negative-database-query-ids 

    Use negative query IDs for queries loaded from the database
    @AzureMarker
    AzureMarker authored Sep 19, 2018
    Configuration menu
    Copy the full SHA
    9e4d77a View commit details
    Browse the repository at this point in the history

  2. Put overTime data into shared memory 

    The array of overTime structs is separate from the client data, because
the client data is dynamically sized. Each client has a separate shared
memory block named `/FTL-client-ID` where `ID` is the numerical ID of
the client. The client data is an array of ints, which is indexed using
the overTime time indexes.

Before, the code referenced the client data via
`overTime[timeIndex].clientdata[clientID]`
Now, the code references the data via
`overTimeClientData[clientID][timeIndex]`

The client data is updated whenever a new client or overTime slot is
added, so referencing the client data should always be valid.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 19, 2018
    Configuration menu
    Copy the full SHA
    cd41aae View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2018

  1. Merge pull request #385 from pi-hole/feature/shmem-overTime 

    Put overTime data into shared memory
    @AzureMarker
    AzureMarker authored Sep 21, 2018
    Configuration menu
    Copy the full SHA
    45832da View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2018

  1. Fix error freeing client shared memory 

    The client name was freed before the shared memory was freed, so it was
used after free in delete_shm.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    b936e2b View commit details
    Browse the repository at this point in the history

  2. Fix interchanged indizes in GC routine 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Sep 25, 2018
    Configuration menu
    Copy the full SHA
    325fc64 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2018

  1. Merge branch 'development' into new/shmem

    @DL6ER
    DL6ER committed Sep 26, 2018
    Configuration menu
    Copy the full SHA
    ea7cdfb View commit details
    Browse the repository at this point in the history

  2. Allocate individual client overTime objects guaranteed pagesize-aligned 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Sep 26, 2018
    Configuration menu
    Copy the full SHA
    776f91c View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2018

  1. Add shared memory lock 

    This replaces the previous mutex managed by threads.c. The lock is
stored in shared memory with the name "/FTL-lock", and any clients
connecting to the shared memory are expected to use the lock when
accessing memory.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Oct 5, 2018
    Configuration menu
    Copy the full SHA
    8c68aac View commit details
    Browse the repository at this point in the history

  2. Remove threads.o from Makefile 

    Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Oct 5, 2018
    Configuration menu
    Copy the full SHA
    3283d68 View commit details
    Browse the repository at this point in the history

  3. Fix lock initializers for arm build 

    The arm compiler does not like the `{ 0 }` initializers for some reason.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Oct 5, 2018
    Configuration menu
    Copy the full SHA
    fa9e7b1 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2018

  1. Review comments 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Oct 7, 2018
    Configuration menu
    Copy the full SHA
    215fa14 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #389 from pi-hole/new/shmem-clients-pagesize 

    Allocate individual client overTime objects guaranteed pagesize-aligned
    @AzureMarker
    AzureMarker authored Oct 7, 2018
    Configuration menu
    Copy the full SHA
    c3934eb View commit details
    Browse the repository at this point in the history

  3. Store database ID in queries struct. This information will be availab… 

    …e through shmem to the API to support transparent overlap with the SQlite3 database

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Oct 7, 2018
    Configuration menu
    Copy the full SHA
    1279b93 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2018

  1. Obtain ID before insertion transaction and store IDs only when a quer… 

    …y has actually been saved instead of trying to do this in a post-processing step

Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Oct 11, 2018
    Configuration menu
    Copy the full SHA
    601722d View commit details
    Browse the repository at this point in the history

  2. Initialize db rowID with zero 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Oct 11, 2018
    Configuration menu
    Copy the full SHA
    ca2e38b View commit details
    Browse the repository at this point in the history

  3. Remove read lock and simplify lock names 

    Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Oct 11, 2018
    Configuration menu
    Copy the full SHA
    d7b4ce8 View commit details
    Browse the repository at this point in the history

  4. Change shm lock to a robust mutex 

    Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Oct 11, 2018
    Configuration menu
    Copy the full SHA
    a3e9677 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2018

  1. Use a condition variable and boolean to notify FTL is waiting for a lock 

    This is checked by accessors of shared memory to defer to FTL so they do
not starve FTL of access to shared memory (starving DNS resolution).

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Oct 13, 2018
    Configuration menu
    Copy the full SHA
    b0c8a54 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2018

  1. Remove condition variable from shared memory lock 

    It is no longer needed.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Oct 24, 2018
    Configuration menu
    Copy the full SHA
    ece89a5 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2018

  1. Set queryID to zero for queries imported from the database on startup 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Oct 25, 2018
    Configuration menu
    Copy the full SHA
    100d92b View commit details
    Browse the repository at this point in the history

  2. Remove unused variable queryID 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Oct 25, 2018
    Configuration menu
    Copy the full SHA
    f8ba173 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #398 from pi-hole/new/shmem_databaseID 

    Store database ID in queries struct
    @AzureMarker
    AzureMarker authored Oct 25, 2018
    Configuration menu
    Copy the full SHA
    f0dabf6 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2018

  1. Merge branch 'development' into new/shmem 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Nov 21, 2018
    Configuration menu
    Copy the full SHA
    85290bb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'new/shmem' into feature/shm-lock 

    # Conflicts:
#	database.c
#	request.c
    @AzureMarker
    AzureMarker committed Nov 21, 2018
    Configuration menu
    Copy the full SHA
    ff4d985 View commit details
    Browse the repository at this point in the history

  3. Fix reference to old lock functions 

    Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
    @AzureMarker
    AzureMarker committed Nov 21, 2018
    Configuration menu
    Copy the full SHA
    befc0f8 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2018

  1. Merge pull request #396 from pi-hole/feature/shm-lock 

    Add shared memory lock
    @DL6ER
    DL6ER authored Dec 9, 2018
    Configuration menu
    Copy the full SHA
    0760461 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'development' into new/shmem 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Dec 9, 2018
    Configuration menu
    Copy the full SHA
    a595f21 View commit details
    Browse the repository at this point in the history

  3. Remove obsolete constants 

    Signed-off-by: DL6ER <dl6er@dl6er.de>
    @DL6ER
    DL6ER committed Dec 9, 2018
    Configuration menu
    Copy the full SHA
    120aeb4 View commit details
    Browse the repository at this point in the history