Consolidating photos after moving project files to external HD

Troubleshooting
1

I recently moved my project files to an external HD as my computer had run out of space. Before the move, thousands of photos were regularly automatically uploaded to ICloud to save HD space, and would therefore show up as missing in Tropy.

Although the files on the new external are organized exactly the same as the computer HD, it seems that Tropy is still looking for the files on my computer when I run Consolidate Photo Library, causing the same problem with randomly missing files across my project.

Is the way past that to delete the original folder on my computer? Or is there a way to tell Tropy to find the Photo Library on the external?

Thanks for your help!

2

As long as your photos are still organized in the same way, it should be relatively easy to consolidate them. There is more info in the user manual here, but basically, what you can try to do is to consolidate a single photo (either consolidate via the context menu in the photo panel, or, if you’re using version 1.10 you can also consolidate an item that only has a single photo). Tropy will prompt you to find the missing photo at its new location; afterwards you will be asked whether Tropy should try to look for similarly missing photos. As long as the photos are still relatively located to each other, Tropy should find them. Depending on the number of photos it can take a long time to consolidate the entire library. If you have more than a few thousand photos we can also fix the paths in the project database manually – this takes only seconds but is a little more advanced.

3

Thank you for your help!

I think I’ve discovered my problem. My project files are organized into different folders and subfolders.

Although I moved that entire organizational structure into my external HD, when I ask Tropy to search for other similarly missing photos, it only will look for them within the specific subfolder, meaning I’ll have to run “consolidate” for each of these folders across the whole of a very long project (dozens - at least - folders - and about 13,000 files).

Does that seem like a plausible explanation of what’s going on here? If so, is there a way out of the predicament?

Thanks,
Mitch

4

Yes, Tropy will only be able to find photos if the necessary change to the path of the missing photo can be deduced, in some way, from the photo path you consolidated. In practice, if you moved all your photos together there will be a common sub-path that needs to be changed in all the paths so Tropy would normally be able to detect them easily. However, since you moved the photos to a different drive, I’m guessing that you’re using Windows and so that paths have a new drive letter. Unfortunately, this means that there is no valid relative path from the old to the new location (Tropy tries to compute this relative path and apply it to other missing photos) and for this reason the auto-consolidation fails.

The best way to move the photos to an external drive on Windows is to first move the project file to the same folder as the photos, open the project, and change the preferences to use paths relative to the project file, close the project and move both project file and photos to the new drive. But it’s too late for that now (you could move the photos back temporarily, but with 13k files this takes a long time) so I think the best solution is to fix the paths manually in your project file.

The project file is a SQLite database, you can open it with the sqlite3 command-line tool; on Windows I’ve also recommended DB Browser in the past. Before you go ahead and inspect your database file, please make a backup copy, in case something goes wrong!

In the database, the paths to your photos are stored under photos.path. So for example, to list all your photo paths you could use the command

  • select path from photos order by path;

To change a common prefix in all your paths instantly, you can use a command like

  • update photos set path = replace(path, 'C:\\OLD PATH', 'E:\\NEW PATH');

If you tell me from where to where you moved your photos I can post the exact command you can use.

5

Thanks for this. I thought it had solved things for me, but sadly I’m still encountering problems.

First, I should note that I am on a Mac, so some of your suggestions about how to move photos to an external HD didn’t seem applicable. Perhaps that is part of my problem.

Here’s what I am encountering now: when I go to consolidate my photos, approx. half of them still try to use old pathways, e.g.:

/Users/isabell/Desktop/research/Tropy files/Wilson Library, UNC/Gaston Papers, UNC, fol 4 …

The rest follow the new pathway to the external, e.g.:

Users/isabell/Volumes/One Touch/Tropy files/St. Mary_s research/St. Mary’s research, summer 2019

When I try to consolidate the half looking to the HD still, that makes other parts of the project get misaligned. It’s turned into a vicious cycle.

My question is this: is there a way that I can manually change the file paths of only the photos that are still directing themselves toward the HD? For the life of me I can’t figure out how to properly use consolidation. Thanks for your help!

6

If you’re on macOS this should be quite easy. I think you should be able to let Tropy fix the issue by consolidating just a single photo through the UI; but if things got mixed up it should also be easy to do this manually in the Terminal.

To try to let Tropy figure out the paths, you would need to consolidate a single photo by pointing Tropy to its current location. You would do this as follows:

  1. Find one of the photos that fails to load
  2. Right-click on that photo in the photo panel and select to consolidate it
  3. This should prompt you to find the appropriate file: at the top of the dialog it should print the original file name, so you would navigate to the folder on the external HD and select the matching file there
  4. When you’ve selected the file, Tropy will prompt you whether or not to consolidate other missing photos automatically

In your scenario, I’m relatively certain that Tropy should then be able to find all the photos again. If not, you can change the paths manually, but first it’s a good idea to print out all the paths Tropy knows about, just to make sure everything is actually the way we expect it to.

To print all photo paths, you need to open the Terminal.app and you need to know where your Tropy project file is.

  1. Before you start, please make a backup copy of the project file.
  2. Open Terminal.app
  3. Next, open the project file. To do this, you need to know the full path to your project file. At the Terminal prompt you can print a list of all the photos in your project like this:
sqlite3 "project" "select path from photos order by path"

Only, where it says "project" you insert the path to your project file. For example:

sqlite3 "/Users/isabell/Desktop/research/My Tropy Project.tpy" "select path from photos order by path"

This should print a long, sorted list of the files in your project. If you scroll through the list I would assume that the first half of paths starts with /Users/isabell/Desktop/research/Tropy files and the other half with /Volumes/One Touch/Tropy files. If this is the case, then everything is the way you described it above and you can change all the old paths with a single command like this (again, replacing the "project" at the start there with the path to your project file):

sqlite3 "project" "update photos set path = replace(path, '/Users/isabell/Desktop/research', '/Volumes/One Touch')"

That should do the trick. If you now repeat the previous command to list all the paths in the project, they should all start with /Volumes/One Touch/Tropy files.

7

Thanks for your reply!

After consolidation did not fix the problem I followed your directions with Terminal in order to see the state of my file paths. I’m confused by the results.

While you’re right, the paths are all different… they look pretty wildly so.

Here are all the different paths, each leading to hundreds – sometimes thousands – of files:

…/…/…/…/…/Volumes/One Touch/Tropy files/

…/…/…/…/Volumes/One Touch/Tropy files/

…/…/…/Downloads/

…/…/Research/Tropy files/

and perhaps most baffling of all, another set of paths do not contain /Users, /volume, or /research. Instead they just begin with the subfolder.

For example, this is complete file name (which still works for some reason!):

Wilson Library, UNC/Gaston Papers, UNC, fol 4/IMG_3052.jpeg

These are my questions:

  1. what should I make of the repeating …/…/ that begin so many of my files?
  2. is my collection now entirely too much of a mess for an easy fix?
  3. is there a way for me to rename individual pathways? It seems my efforts to apply broad instructions only lead to larger problems.

Thanks for your help. I would be even more lost without it!

8

You seem to have switched your project to use paths relative to your project file (in the project preferences). This is normally the best thing to do if you keep both your project file and the photos together on the same external hard drive (using relative paths this way allows you to open the project file on a different device if you connect the external drive there).

The way relative paths work, is that Tropy will find each photo by combining the current location of the project file with the path it has stored for the photo. So in your example above, it depends on the where your project file is. I’m assuming it is currently at /Users/isabell/Desktop/research/Tropy files, and then it would find the photo above at /Users/isabell/Desktop/research/Tropy files/Wilson Library, UNC/Gaston Papers, UNC, fol 4/IMG_3052.jpeg.

In any case, when you change how photos are linked in the project preferences (using absolute or relative paths), Tropy will re-write the paths. It’s important that all paths are correct when you do this; I suspect you might have changed the setting previously, while the project was not fully consolidated.

As long as you have not moved your photos around too much this should still be easy to fix. But before you do this, it’s important that you decide how you want to organize the project: that is, where to keep the photos and the project and whether to use absolute or relative paths.

If you want to keep all the photos and the project file on the external drive, then it’s best to keep using relative paths. If you want to do this, then you should first make sure that you have all your photos below the /Volumes/One Touch/Tropy files/ folder. Then you can also place the project file in that folder and next you’ll have to fix all the paths in the project using the rename command I posted earlier. Basically, you need to remove everything up to the ‘Tropy files’ folder. To do this, you need to look at the listing of files and count all the different path beginnings and then replace them. For example:

sqlite3 "project" "update photos set path = replace(path, '../../../../../Volumes/One/Touch/Tropy files/', '')"

However, if you prefer to keep the project file on the main drive (e.g. to keep working on the project while the external drive is not connected); or if you want to store only some of the photos on the external drive and some others on the main drive, then it will be best to use absolute paths. Renaming all the paths is a bit more difficult in this case, because it depends on where your photos actually are located, but it should still be relatively easy. If you send me a link to your project file (you can send me a direct message if you prefer not publicly share the project) I can probably do this for you more quickly than it takes to explain it. (You can also send me the file if you’d prefer to keep using relative paths as in the scenario above.)

9

Thanks! It took a good part of the day, but I was able to change the paths, and now I can open all of them.

However – a new problem immediately arose: when I open the project and try to add notes I now get the following message:

Error: SQLITE_READONLY: attempt to write a readonly database

Is this a common problem?

Thanks again,
Mitch

10

Here’s the full error message if that helps.

{“msg”:“unhandled error: SQLITE_READONLY: attempt to write a readonly database”,“stack”:“Error: SQLITE_READONLY: attempt to write a readonly database”,“system”:“Darwin 19.6.0 (x64)”,“time”:1642187628108,“version”:“1.11.1”}

11

And one more thing. Per your instructions in other threads, I made sure that sharing and permissions grants access to read and write on the file. It still appears as read only.

12

Does Tropy indicate that the file is locked (i.e., read-only) when you open the project with a lock symbol next to the project name? In that case, the file (or the external drive) is read-only or Tropy does not have write permissions for other reasons.

Can you post the full log file here? It might help us figure out what’s going on.

13

I wasn’t able to make it read the file through the external. However, when I moved the project (back) to my laptop harddrive, it works!

Considering the trouble it’s taken me to get to this point. I think I’ll leave it there. Thanks for all your advice!

14

It’s likely an OS-level permission that is missing. You could check the System Preferences: under the ‘Privacy’ tab in the ‘Security & Privacy’ section you could test is adding Tropy to the ‘Full Disk Access’ list solves the problem/

15

Ah, thanks. I’ll try that if I attempt moving it to the external again. For now, I’m just satisfied it’s working at all!

16

Hello. My photos and Tropy project file are saved in OneDrive and on my original laptop everything works fine, but I am trying to consolidate all the photos to a second laptop. Right now I can only consolidate each photo individually on the second laptop. Following a few of these threads led me to change “link photos” in preferences to “relative to your home directory” but automatic consolidation still did not work. I ran the code you suggested in this thread in my terminal and saw a similar problem that Tropy was pulling photos from two different paths. I ran the code to replace one with the other, but auto consolidation still does not work. Any suggestions on how to proceed? Many thanks in advance for your help!

17

If you link the files relative to your home directory, opening the project from multiple devices should work as long as your OneDrive folder has the exact same path relative to your home directory on both devices.

However, because the project file itself is also on OneDrive, I think the best option is to link photos relative to the project file itself. This way the paths are guaranteed to be the same across devices. When you change the way the files are linked, Tropy re-writes the paths, but this will fail if you change the settings while the project is not fully consolidated. I think this is what probably happened in your case. It should still be easy to fix, especially if we link the files relative to the project.

I’d try the following steps: make a backup copy of the project file (just in case!); then open the project on the device where all paths work (ideally – but we should be able to manually fix the paths if that’s not possible) and change the link settings to be relative to the project file.

Once that’s done quit Tropy and review the paths from the terminal with the command:

sqlite3 project.tpy "select path from photos order by path"

This should print all the paths. If everything worked then all these paths should be relative to your project file. I.e., if the project file and photos are all in the same folder, these should just be the list of filenames; if the photos are in a sub-folder then there should only be that folder at the beginning of the path, etc. If that’s not the case we can update the paths manually. I can tell you the exact command if you post the paths and tell me where the files are located relative to the project file.

18

Thank you very much for the quick and helpful response! I relinked photos relative to the project file and printed my photo orders by path, which showed that there are still multiple paths for photos. I realized this is probably because I moved a few sub-folders of photos from my local desktop into OneDrive, as I did not originally start the project file in OneDrive. I also recently changed the main folder name, which is reflected nowhere in the printed paths.

The photos are now all in the same folder (under sub-folders) that the project file is in OneDrive. There are quite a few, so I can message you privately if you prefer.

19

Right. So, just to make sure your project is ready, it should now be set to link relative to the project file. And all paths (if you review the entire list) are either the correct path relative to the project file or a wrong path. But (and this is the important bit), every wrong path only has a bad/superfluous left-hand side. That is to say, every wrong path can be turned into the correct path (relative to the project file) by removing a prefix. Unless you completely re-organized your photos there should only be a handful of these bad prefixes in the list and you should be able to remove them using this command:

sqlite3 project.tpy "update photos set path = replace(path, 'the-bad-prefix-goes-here', '')"
20

Yes, thanks. It seems that the closest thing to my “correct” path is the past name I used for the project folder originally that no longer exists. There are no paths for the new folder name that everything is currently under. In this case, should I replace those paths as well, or replace only the bad prefixes?

More details here: The original folder name was “Tropy photos - Diss” but now it is “tropyprojectdiss” and all the bulk of the paths (what I’m counting as “correct” as of now) is “Tropy photos - Diss.” But should I replace all prefixes to start with “tropyprojectdiss”? Also, in the command you sent above would “path” be “tropyprojectdiss” or just “path”?

Will Tropy only automatically consolidate photos if every single photo has a correct path? Realizing there are a few really old paths that are not relevant to the project anymore that might just be worth deleting if that is the case.

I hope that makes sense and thanks so much in advance for all your help.