I recently updated my macbook and as a part of that I had to update foto’s. Since then I found out that none of the files I have imported into tropy can be shown via tropy. In essence I now have metadata without the corresponding files.
I suspect Photos.app moved the photos in its library; there’s a good change that you will be able to fix this by consolidating a single photo (unless Apple also renamed the originals). Can you tell us which version of Photos.app you had been using and which one you’re using now? We’ll take a look to see what changed.
1 Like
My photo’s programme had to be updated after I updated my Mac from Mac Os 10.14 Mojave to Mac Os 10.15 Catalina. The current version of Photo’s is 5.0 (101.15.190) But I don’t have a clue from which one it had to be updated. The version right before that I guess?
It looks like Photos.app in macOS Catalina renames and moves all the photos in its library into a new folder structure. I’m afraid this means our best option will be writing a script that searches for each missing photo in your Tropy library by using a checksum.
Can you collect some information about the photos in Tropy and Photos.app for us? To do this, you need to open Terminal.app and query the Tropy project database. For example:
$ sqlite3 -csv ~/Documents/project.tpy "select size, checksum, path from photos order by path"
Will print out the size, checksum and path of all the photos in the project file at ~/Documents/project.tpy. To save the results to a CSV file instead of printing them to screen you can add > tropy-photos.csv at the end of the command to save the results to this file.
Next, we need some information about the Photos.app library. When you look at the library file in Finder (by default it’s in your Pictures folder) you can right click on the file and select ‘Show package contents’. This will open a new Finder window with your photos library. There should be a folder ‘originals’ in this folder: please look up the full path to this folder because we’ll need it next (for example, if you drag the folder into a Terminal.app window, the full path should be printed there). Using the full path to the originals folder instead of the PHOTOS_ORIGINALS, the following two commands should print out all the file sizes and check-sums of each file in your Photos library. Again, if you add > and a filename at the end of these, the results will be printed into the respective file:
$ find PHOTOS_ORIGINALS -type f -print0 | xargs -0 ls -l
The command above would print file size and dates for each photo. And:
$ find PHOTOS_ORIGINALS -type f -print0 | xargs -0 md5
This one should print the MD5 checksums. If you could send us the results of these commands (here or, as a private message if you prefer), we’ll look into resolving the missing files.
1 Like
Thank you, I will collect the data for you tomorrow, and hopefully this can be resolved. Otherwide I’ll lose weeks worth of work. How can I send it to you via private message?
I seriously hope that Apple did not alter the files themselves (the folder is called ‘originals’ so I would be surprised if they had!). As long as we have check-sums to compare we can definitely recover the mapping.
Does you workflow include working on the photos in parallel in Photos and Tropy? If not, it might be safer, going forward to make copies of the files used by Tropy so that Photos can’t change them (the downside of this is that you’d need twice the disk space).
If you click on my name/avatar at the top of each message there should be a popup with a ‘Message’ button. You can use this to switch to a direct thread that is not public.
1 Like
Hello, I am having the exact same problem.
When I tried to run the check-sum in Terminal it gave me the error “command not found”.
About 75% of my almost 750 previously uploaded photos now appear as gray boxes, with an error triangle next to them in the metadata. I cannot see any pattern with the photos that are still properly rendered versus the ones that are not.
@Lauren did you also import directly from Photos.app? And update to Catalina recently? If not this might be a different that might be easier to fix using the auto-consolidation.
However, if this is indeed caused by Photos.app moving and renaming the originals (or if the photos have been moved/renamed in a way that makes auto-consolidation impossible) I have written a script find-photos.sh to help deal with this.
Before you start, please make a note of where your project file and your Photos.app library are located. For the following instructions, I’ll assume my Tropy project is at /Users/ariadne/Documents/project.tpy and my photo library is at /Users/ariadne/Pictures/My Photos.photoslibrary.
To be safe, it’s best to be working with a copy of my project file. So, as a first step:
- Close Tropy
- Make a copy of the project file. In my case, I’ll copy my project file to the Desktop:
/Users/ariadne/Desktop/project.tpy.
Next, download find-photos.sh (direct link) and also save it to the Desktop. Now we need to make the script executable. To do this, open Terminal.app and run the following commands:
cd Desktop
chmod 755 find-photos.sh
xattr -d com.apple.quarantine find-photos.sh
To make sure this worked, we try to run the script without any input:
./find-photos.sh
This should print a short usage message and means that everything’s setup correctly.
Finally, let’s also check that we have the right location of the project and photos library:
ls -l project.tpy
ls -l "/Users/ariadne/Pictures/My Photos.photoslibrary"
These two commands should print out the file info for my project file and the contents of the photos library folder. Notice that I’ve used quotes around the path to the photos library: this is necessary, because the path contains a space. In the latest version, the photos library should contain a folder called ‘originals’: this is the folder we’re interested in.
To run the script for real, we need to point it at our project and the photos library. The script will print all necessary update commands to the screen (it currently does not alter the project file directly). This is just a precaution, because it let’s us review the changes first. Instead of printing the update commands to the screen, we can also pipe them into text file, which makes it easier to use it later, so that’s what we’ll do next:
./find-photos.sh project.tpy "/Users/ariadne/Pictures/My Photos.photoslibrary/originals" > update.sql
The > update.sql at the end means that we’ll write all output to this file. If you leave it off, the results will be printed to the screen instead. Running this script may take a while: it has to compute a checksum for each file in your photos library; as rough estimate you can assume this takes about one second per photo in your library.
If the script has terminated, there should now be a file update.sql on your Desktop. When you open the file, it should contain only lines like:
UPDATE photos SET path=’/Users/ariadne/Pictures/My Photos.photoslibrary/originals/3/39F2A58E-AAE2-5984-2C9B-9995724450BF.jpeg’ WHERE id=422;
If this looks good, we can apply the update on our copy of the project database:
sqlite3 project.tpy < update.sql
After this, the copy of project.tpy on the Desktop should be up to date. To see if it works:
- Open Tropy. This will open your original project file by default, which is not fixed yet.
- Select
File -> Open...and select the updated project file on the Desktop (or just drag the file from the Desktop into Tropy) and check if everything looks good.
If the update was successful, close Tropy again; move your original project file to a backup folder (or delete it) and move the updated file back to the original location.
I’ve had the same problem and I’m having major issues following the directions for the solution (find-photos.sh link goes to a text file—how am I supposed to download this? Command lines for Terminal not working, etc.). Are there any other solutions or could you clarify some of the directions? And will this same problem happen every time I update my OS?
This only happens if Apple changes the way Photos.app structures its data. The issue is simply that the Photos.app update moved your files to a different place – it’s the first time this happened in a number of years, but it’s really Apple’s prerogative to change their app’s internal structure.
The script file is just a text file: if you follow the direct link above, you can then just ‘save’ the file with your browser (e.g., File -> Save) and follow the rest of the directions. However, it’s true that this is fairly advanced stuff – do you have someone at your institution who could assist you?
1 Like