Problem importing data from Immich #270
Comments
|
Mine worked... I think. though it seems low on the number of data points, and there is not status updates. i am expecting much more than 26k datapoints. How do i check that it has worked? |
|
Same issue here with the NoMethodError; Immich version 1.117 |
|
Mine are imported but the reverse_geocoding doesn't work. I'm using Immich v1.118.2 It would be quite helpful if we could get the payload that is causing these errors but for this we need to manually trigger the job and use a breakpoint. There's a section on the docs that might be helpful: https://dawarich.app/docs/FAQ |
this means that the point, that was attempted to be reverse geocoded, was not found in the database. In this case, there would be nothing to log because there is no point in the first place. |
|
I have the same problem with an import from Immich |
|
@daggr8 @ilijamt @Millio345 in the 0.15.11 I introduced writing the response received from Immich to the
Example of JSON for a single image produced by Immich: {"id"=>"b3f77c49-5335-4ad2-9032-afe04b8843f2",
"deviceAssetId"=>"D04881DA-F413-494D-80CB-8B4CBC8BD37D.jpg-131770",
"ownerId"=>"f579f328-c355-438c-a82c-fe3390bd5f08",
"deviceId"=>"CLI",
"libraryId"=>nil,
"type"=>"IMAGE",
"originalPath"=>"upload/library/admin/2001/2001-01-01/D04881DA-F413-494D-80CB-8B4CBC8BD37D.jpg",
"originalFileName"=>"D04881DA-F413-494D-80CB-8B4CBC8BD37D.jpg",
"originalMimeType"=>"image/jpeg",
"thumbhash"=>"JPgVHAR394iniHeHZ3h4iOD4Bg==",
"fileCreatedAt"=>"2000-12-31T23:00:00.000Z",
"fileModifiedAt"=>"2001-01-01T00:00:00.000Z",
"localDateTime"=>"2001-01-01T00:00:00.000Z",
"updatedAt"=>"2024-05-21T14:39:22.205Z",
"isFavorite"=>false,
"isArchived"=>false,
"isTrashed"=>false,
"duration"=>"0:00:00.00000",
"exifInfo"=>
{"make"=>nil,
"model"=>nil,
"exifImageWidth"=>1024,
"exifImageHeight"=>1820,
"fileSizeInByte"=>131770,
"orientation"=>"1",
"dateTimeOriginal"=>"2000-12-31T23:00:00.000Z",
"modifyDate"=>"2000-12-31T23:00:00.000Z",
"timeZone"=>"Europe/Berlin",
"lensModel"=>nil,
"fNumber"=>nil,
"focalLength"=>nil,
"iso"=>nil,
"exposureTime"=>nil,
"latitude"=>11.111,
"longitude"=>22.222,
"city"=>"Hohenschwangau",
"state"=>"Swabia, Bavaria",
"country"=>"Germany",
"description"=>"",
"projectionType"=>nil,
"rating"=>nil},
"livePhotoVideoId"=>nil,
"people"=>[],
"checksum"=>"7PGkxQcyGjRNy82gV9vOKcWZy94=",
"stack"=>nil,
"isOffline"=>false,
"hasMetadata"=>true,
"duplicateId"=>nil,
"resized"=>true}If you could send me your file(s) in Discord, I could work on the fix. My importing process works without any errors, so I need an example of a payload that causes errors to be raised. |
|
Hi, For me the .json file doesn't exist. |
|
Immich import is also failing for me. Immich import has worked in the past from the same Immich server, so either I have a new bad photo or something changed in Immich or Dawarich recently. I notice the OP had this error, as did a subsequent poster:
But now others and myself are seeing this instead: |
|
It appears that the Sidekiq container needs to have a I fixed the issue by doing this: And then run the import again. This fix will only last until you restart the container(s), but it should get the import to work. I'm not sure why we are missing that folder, perhaps because we didn't set up a "watch" folder and so it doesn't get auto-created? Seems like the process should create the |
|
I followed the tip above and i get a json containing this. I tried it multiple times with the same result. [[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]] |
This one is really interesting, it means Dawarich receives a list of empty arrays so there are no points to get data from. I think I'll change approach to writing the debugging file and will log whole response from Immich, so we could investigate it further |
In the default docker-compose.yml there are these lines: last one assumes your instance should have this directory, but I guess if you didn't use watched functionality, Dawarich should create it on its own. I'll look into it too |
|
Here is a script for you guys to see what data is being returned by your Immich instance: #!/bin/bash
# Set your API key and base URL here
IMMICH_API_KEY="YOUR_API_KEY"
IMMICH_API_BASE_URL="http://immich.instance.com/api"
# Output file
OUTPUT_FILE="immich_data.json"
# Temporary file for storing intermediate responses
TEMP_FILE="immich_data_temp.json"
echo "[]" > $TEMP_FILE # Initialize as an empty JSON array
# Calculate total requests for progress tracking
START_YEAR=1970
END_YEAR=$(date +"%Y")
TOTAL_REQUESTS=$(( (END_YEAR - START_YEAR + 1) * 12 ))
# Initialize counters
current_request=0
# Loop through each year and month, retrieve data, and append to TEMP_FILE
for year in $(seq $START_YEAR $END_YEAR); do
for month in $(seq -w 1 12); do
# Construct the URL
url="${IMMICH_API_BASE_URL}/timeline/bucket?size=MONTH&timeBucket=${year}-${month}-01"
# Send the request and get the response
response=$(curl -s -H "x-api-key: $IMMICH_API_KEY" -H "accept: application/json" "$url")
# Check if the response is valid JSON and append it to TEMP_FILE
if echo "$response" | jq empty 2>/dev/null; then
# Add the response to TEMP_FILE JSON array
jq ". += [$response]" $TEMP_FILE > tmp.json && mv tmp.json $TEMP_FILE
else
echo "Failed to retrieve data for ${year}-${month}"
fi
# Increment the counter and update progress bar
current_request=$((current_request + 1))
progress=$((current_request * 100 / TOTAL_REQUESTS))
printf "\rProgress: [%-50s] %d%%" $(printf '#%.0s' $(seq 1 $((progress / 2)))) $progress
done
done
# Move the final JSON array from TEMP_FILE to OUTPUT_FILE
mv $TEMP_FILE $OUTPUT_FILE
# Complete the progress bar and show final message
echo -e "\nData collection complete. Output written to $OUTPUT_FILE."
Save it as The result will be written into Then feel free to send me the results via DM in discord and I'll be able to debug it. Hope this helps! |
|
I don't use Discord unfortunately. I hope it's okay to send it here. [],
[],
[], |
In the immich interface, do your photos have info on geodata? Are they shown on immich map? |
|
Also, the resulting file might contain a lot of empty arrays in the beginning and then the actual data is way below them, so scrolling might help :) |
Yepp they do. And the file is empty, no data besides hundreds of rows of [],. I've double checked :) |
|
Hmm so |
Same problem and yes, empty result, even though there is geodata in the photos for the day specified. |
|
Using baseurl/api/timeline returns Using baseurl/timeline returns a page I've tried multiple dates. I can add that it takes a minute or two for the script to run and while it does the CPU on my Immich server is spiking, so it definitely talks to the API. |
|
Then I'm afraid it's something with Immich, because if there is data in your instance, it should be returned, but it's not :( |
I'm not confident that's the only problem, i think there are two:
If i add a hosts entry in the sidekiq container and use the local IP and port then it's able to connect and then subsequently, but it fails over https. I think your code may not be able to handle https? The subsequent error is the one predominantly experienced in this issues ticket:
I created an issue within the immich github for this: |
|
This tool should not use the timeline endpoints to extract/download gps data from immich. The timeline endpoints are optimized for specific use cases, like hiding archived photos, showing only trashed photos, etc., and are subject to change. Please use the "search metadata" endpoint for bulk operations like this. |
|
In 0.16.5 I reworked the integration to use |
|
At first run i got an error that However the "Jobs in queue" is at 0. I'll update when it's finished. |
|
this seems to be working! Thank you :) |
Describe the bug
When using the "Import Immich Data" nothing is imported.
Version
0.14.3
Immich version 1.115
To Reproduce
Go to /imports and click "import Immich Data"
Expected behavior
Data imported
Logs
Logs from dawarich_sidekiq
The text was updated successfully, but these errors were encountered: