Constant Consolidation

[emanley1]

Greetings-
I am having an issue very similar to the one here: Continually consolidating photos. I decided to create a new post, but I am essentially having the same issue - despite not moving the site of the .tpy folder (or any of my images), Tropy keeps consolidating the entire library. Thumbnails are still there. Its a very large project (40,000 items) so it takes a very long time and slows down my machine considerably. I know that the drive where the project is held is local, as are all the images; I had recently had this same problem but fixed it on my own by moving the Tropy file into a local drive. Should I run the test you suggested to the other user?

Thanks!
Beth Manley

[inukshuk]

So, just to confirm, all the thumbnails for your photos show in Tropy and you can also view the full photos in item view, but they keep getting consolidated? Does this happen every time you restart Tropy? And are all photos affected this way? That is, if you scroll a few pages through the project and then pause for a one or two seconds, Tropy will increasingly try to consolidate more photos. Or does this only affect a certain number of photos which get consolidated over and over again?

Here are a few more questions which might help us figure out what’s going on:

• Could you post a recent log file? Otherwise, can you tell us which version of Tropy this is and what’s your operating system.
• Do you use only a single Tropy project or do you routinely access multiple project files?
• Do you regularly access the photos in another application? Or is there another application which might for some reason access (and modify) the photo files?

[emanley1]

Thanks for replying!

All the thumbnails show, but in some (not all) items the images won’t actually show in item view (I get the triangle with exclamation point icon) and it will prompt me to consolidate the item, and then ask to consolidate the entire library, which then starts running a consolidation for the entire library of 45,000 items. It seems random when it does it, but I generally keep Tropy open all the time. I am using a single Tropy project (not multiple) and have now moved older versions of this project to a different folder to try to fix the problem. I also do not generally access the photos through another application.

I just tried uploading the Tropy log and I am wondering if that might be what is causing the problem - I can see the log file when I select “Show Log Files” but then is inaccessible (can’t find the pathway) when I try to find it for the upload - if that makes any sense? I am running 1.11.1 on a Mac OS 12.1 Monterey.

Thanks!

[inukshuk]

OK thanks, I think this is becoming much clearer. For most image types, Tropy will try to open the original file in item view. When this fails it typically means that either the file was moved or renamed (i.e., the path Tropy has for the file is wrong) or that there are permission errors when accessing the file. You’re then prompted to select the file again; only if this succeeds should Tropy prompt you whether to consolidate the entire library. My guess is that Tropy has insufficient permissions to access some of the files in your library – this is something that could have happened due to a macOS update. When you consolidate the single photo, you select it manually and pass it to Tropy: macOS notices this and grants Tropy access to that one file, but not the other files and that’s why the auto-consolidation fails.

This is just a guess though. When you save the log file after trying to open one of the missing photos, we should be able to tell (basically, the log file should indicate whether the file was not found or not opened due to insufficient permissions). I’m not sure why the log folder should be inaccessible. On macOS, you could try opening Terminal.app and see if you can make it show by entering the command: open Library/Logs/Tropy.

Alternatively, you could try granting Tropy full disk access in the ‘Security & Privacy’ section of the System Preferences (in the ‘Privacy’ Tab).

[emanley1]

It does seem like it might be a permissions issue, since I am not changing the location or path of the original items. I was able to use the Terminal app, and it does pull up the log file (see first attachment). A similar screen appears when I select “Show Log file” in Tropy (second attachment). However, when I seek to replicate that pathway in finder or when uploading a document, it disappears (see third attachment).

I did get Tropy full disc access. I restarted, and now its doing the update all 47,000 items (after prompting me as you indicated it would; fourth attachment).

Is that at all helpful? I’ve also managed to upload a copy of the log file.

tropy.log (9.5 KB)

[inukshuk]

Did full consolidation work after granting the full disk access permissions?

According to the log file, the photo that required consolidation initially was not found. It was looking for the file at:

 /Users/elizabethmanley/Dropbox/PrimaryDocs-Research/MISC_Tourism/TourismGuides/Haiti/1950/IMG_8655.JPG

Did you find the file at this location or was it moved?

[emanley1]

That file is very much still in the same place. I started the consolidation process about an hour ago - it is still going, and (as it does) slowing down my computer quite a bit . .

[inukshuk]

Can you check the log file again to see if there have been any errors reported (particularly permission denieds)? If there are then I would cancel the consolidation instead of waiting.

If you could identify another file that Tropy fails to open, could you check its extended file attributes in Terminal.app. For example, this would print the attributes of the file you consolidated manually:

xattr "/Users/elizabethmanley/Dropbox/PrimaryDocs-Research/MISC_Tourism/TourismGuides/Haiti/1950/IMG_8655.JPG"

If you do this on a file that fails to open, are there any attributes shown? Particularly, com.apple.quarantine? These might have been added if you downloaded/restored the Dropbox folder after updating macOS or something along those lines. If the flag is set, you can try removing it like this:

xattr -rd com.apple.quarantine /Users/elizabethmanley/Dropbox/PrimaryDocs-Research

[emanley1]

Looking at the log again - there seem to be a bunch of items that are getting the tag “Error: ENOENT” in the attempt to consolidate and these items seem to have an erroneous path. Should I try to fix those first before going to one (Guide to Haiti) that we’ve been using as an example? And if so, can I do that in a batched way - rather than one by one? Thanks so so much!

[inukshuk]

Right, ENOENT is the system error we normally receive when a file does not exist – i.e., when it was moved or renamed. Since the files are still there I suspect it’s a masked permission issue, namely, that Tropy isn’t even allowed to see that file exists by the operating system.

Anyway, if you’ve found such a file, I’d suggest to look at only this one file to figure out what’s going on. I’m assuming that once we know how to fix it, we’ll be able to fix all the others in one go – since the paths themselves haven’t changed this means you might not even have to consolidate them afterwards at all.

In any case, please run the xattr command on one of these ‘missing’ files. If the com.apple.quarantine flag is indeed set, then you should be able to run the xattr -rd com.apple.quarantaine command on the entire folder and, thanks to the -r switch it should recursively remove the attribute from all files in it.

[emanley1]

Ok, I ran the xattr command on one of the “missing” files. Here’s the result -

MacBook-Air-3:~ elizabethmanley$ xattr “/Users/elizabethmanley/Dropbox/PrimaryDocs-Research/MISC_Tourism/TourismGuides/Haiti/1952-3/IMG_8714.JPG”
com.dropbox.attrs

No quarantine flag, but this dropbox attribute. Do I need to try to remove that?

[emanley1]

Ok, before I waste any more of your time, I might have figured out the problem. I changed the folder name in a tiny way (from 1950 to 1950-1 and from 1952 to 1952-3) and I believe that is what screwed the whole thing up. That could have made the difference, yes?

[inukshuk]

Yes, exactly, this changes the path so Tropy will not find the file anymore. The way we suggest to handle this is to:

1. Rename the path
2. Open Tropy and find one of the photos in the path and consolidate it
3. Let Tropy consolidate the whole library with the additional info of the file you consolidated in step 2 – with this Tropy should be able to fix all other photos that are affected by the path you changed.

The downside of step 3 is that it has to check all the files in your library, which takes considerable time if your library is large. But in your case there seems to be a different problem, because the auto-consolidation doesn’t seem to fix all the missing files. This could be for two reasons: either there’s a permission issue after all (I’ll have to research if the dropbox attribute could cause this, though I doubt it) or your paths have changed in ways that requires you to consolidate multiple times. For example, if you have multiple folders in the Haiti folder that all changed, you’d have to consolidate one photo in each of the changed folders.

[emanley1]

Ok, thanks. I realized there were a few files that I “cleaned up” with more precise names like that, forgetting it was going to screw the whole system up.

I am going to go through and check and do as you suggest - which may take a minute - and then I will be back in touch with the remaining issue.

Thank you again so much for your help - and I will be in touch!