Failed to Consolidate Photos and Unhandled Error After Update to 17.1

Troubleshooting
1

Hi inukshuk/Tropy Support Team,

We’ve recently updated Tropy to version 17.1. Previously, when we accessed the application from our desktop, it was always opening the latest Tropy project we used. Indeed, it’s still trying to do that as expected. However now, while we can see the project scans and their tags displayed for a very brief moment, the flash suddenly disappears, and 2 error dialogs appear on top of one another, both respectively titled “Failed to Consolidate Photos — TypeError: Cannot read properties of undefined (reading ‘length’)” and “Unhandled Error — TypeError: Cannot read properties of undefined (reading ‘match’)”.

Please, refer to the attached zip file for the details (embeds 2 screenshots and the tropy.log file).

Any idea what’s going on and how it can be fixed? Should we downgrade back to 1.17 or perhaps wait till 1.17.2 is available? Note that to the best of our knowledge, none of our source files (photos etc) have been modified or moved lately.

Thank you,
Moggy
moggy-tropy-1.17.1-errors.zip (78.0 KB)

2

Erratum — Sorry for my typo about the version number. When I mentioned our update to Tropy 17.1, I meant of course Tropy 1.17.1 :wink: Apologies :folded_hands:

3

Thanks for the logs! We’ll likely make the 1.17.2 release in the coming week, and we’ll look into including a fix for this.

1 Like
4

Could you share the .tpy file with us?

5

Sure, inukshuk. Here’s a zipped version of it.
moggy-tropy-1.17.1-tpy-file.zip (43.2 KB)

6

Thanks! Unfortunately it looks like the file is completely damaged. Can you check the file size of the file on your disk? The file in the zip file is 304K which is relatively small so I just want to make sure the complete file was sent. Your log file suggests that the file on your disk contains more data, because Tropy can at least start to open the file, so I hope there’s just been an issue when zipping the file.

7

OK, I just checked. I confirm this particular tpy file I sent to you is indeed 304 KB (311,296 bytes) on my disk. The project file is of course not standalone, as it is “pointing” to various files, mainly regrouped in 2 folders: one for a 2-volume journal embedding raw scans, high-resolution versions for download and consultation, and low-resolution thumbnails for parsing (total: 2.05 GB); another one much smaller containing a series of line art sketches (total: 31.7 MB).

I wonder what could have caused this tpy file to get damaged :thinking: I considered recreating the project from scratch, which remains doable of course, but we’ve been concerned that it could cause the loss of all the metadata and tags we had created and gathered for the occasion. To my knowledge, this information hasn’t been saved separately and remains presumably embedded in the tpy file except some fragments that have been duplicated as PDF metadata when we do possess a PDF version of some of the documents.

8

Right, the .tpy file is a SQLite database file that contains all the information about the project (metadata, tags, lists, notes) but not the image files themselves. SQLite is a very robust file format, the main cause how I’ve seen files get damaged is in combination with network shares or cloud backup/sync folders. That said, 300 KB is suspiciously low for a project of your size, which might suggest that the file was truncated somehow. Do you keep it on an internal hard drive? Or is this a network share of some kind or was the file routinely copied or moved or something like that? Most importantly, do you have a backup or previous copy of the file?

9

Yes, the tpy file and the media files it references are on an internal hard drive. We also keep a back-up copy on the cloud. Based on what we have on the cloud, I can see the tpy file has not been modified since October 2023, the latest modifications on the journal scans were made in November 2023 with the addition of a transcript draft in early September this year (and Tropy was still definitely opening the project then until early this month). As for the line art, the last modifications were in March this year, but again between March and early this month, we could still load those files in Tropy.

OK hmm, here’s what I’m gonna try: on a different machine, or even the same one where the issue has been detected the first time, I’ll revert back to Tropy 1.17.0, then will download the tpy file and its companion media from our back-up copy on the cloud, will reconstruct the relative file hierarchy, and finally will try again to open the project. I’ll let you know the outcome.

10

Hi inukshuk,

OK, we’re making progress here :slightly_smiling_face: So, I reverted to 1.17.0 and based on your advice overrode our latest tpy file with the one we had on the cloud despite the last modified dates and sizes matching — and lo and behold, the project opened this time with no error messages! Then, I re-upgraded to 1.17.1 and again the project could open again without any obvious error message and with our tags and metadata fully preserved, so your intuition that the tpy file got somehow corrupted lately was probably right. Thank God, we had that back-up!!

Of course, that couldn’t be that straightforward :wink: For example, some photos could not be consolidated. For a majority of those, the explanation was simple: their root folder had been renamed causing the photos not to be found, so I had to click the warning icon to relink Tropy to their right location and after doing this to only one, the application even asked me if this updated root location should be used from now on to try to locate other missing pictures. I accepted the offer, and that resolved 95% of the consolidation issues, yay :smiley:

Now, our remaining issues are about a couple of TIFF scans that Tropy can show as the original file or open in an external viewer, and yet stubbornly shows their thumbnail as pitch dark and refuses to consolidate them. I am clueless as to exactly why, especially as all the other TIFFs located in the same directory and part of the same set are correctly processed by Tropy as expected, and they can be accessed otherwise without any problem. I tried to remove and re-add the scans, but to no avail. Any idea what I could try next?

I’ve re-attached a zip file embedding:

  • the recovered tpy file (which is still not that much bigger than 300 KB though),
  • the Tropy log file,
  • a screenshot,
  • a sample of those problematic TIFFs.

Hopefully, that should help you understand how I’m experiencing this consolidation issue.

Cheers.

moggy-tropy-1.17.1-tiff-consolidation-issue.zip (15.5 MB)

11

Hi inukshuk,

OK, all our problems are solved finally! Here’s what we decided to do: all the scans Tropy could not consolidate and generate a proper thumbnail for have been resaved as TIFFs (possibly using a different algorithm, as we used a different version of our image editing application for this). Then, we removed the problematic images from Tropy and replaced them with the newly generated equivalent. And this time, it worked! Everything got consolidated successfully, we have the expected thumbnails, image sets, tags and metadata everywhere — our project has finally been restored to its original pristine state (tpy file size: 324 KB) :slight_smile:

Hopefully, our upcoming update to Tropy 1.17.2 will go without further complications :crossed_fingers:
(EDIT: we’ve just tried with 1.17.2, it works great; no particular problem to report with our project.)

Do you have any recommendation for future projects, so that sort of issues would not happen again? For example:

  • Is there a particular way we should preserve our back-up copies?

  • Is there a typical file hierarchy Tropy users should follow?

  • Will there be options in future versions allowing users to export their tags and also metadata separately without losing the context, just in case (God forbid) similar situations would happen again, and remapping images to tags and metadata would need to be rebuilt? (But perhaps, selecting the project in Tropy and then File > Export > JSON-LD can just do that?)

  • Finally, is it usually preferable to embed everything within a Tropy project instead of having it pointing to external files? Of course, I guess the downside is, we could sometimes end up with an extremely large file as a result…

Thank you.

12

I looked at the sample TIFF and there is indeed an issue with the color channels: Tropy fails when trying to extract the dominant color for the image. Since it works fine with other TIFF files I’m assuming that it is indeed an issue with how the file was encoded, but we can look at this in more detail in case it happens more frequently.

Tropy stores all the project data in the project.tpy file. This is a SQLite file, a stable format that is well suited for archival purposes. So in general it should be fine to just keep backup copies of that file. To be even more safe, you could also make a backup copy in a text-based format, either JSON or CSV, because these will be easier to access with other tools. For a JSON backup you can either select all items and export to JSON (or use copy/paste) or you could use the Archive plugin: this will save all metadata in a JSON file together with all the linked photos in a ZIP file. Alternatively, you can use the CSV plugin to export the metadata to a CSV file which is compatible with the usual spreadsheet applications.

For advanced projects the file hierarchy is up to you. To make an advanced project ‘portable’ it’s good practice to make sure that all the linked files and the project file are in a shared root folder: this way you can configure the project to store links to the files using relative paths.

1 Like
13

Brilliant. We’ll keep your recommendations in mind for future projects and will definitely have a closer look at the plugins you’ve mentioned too. I guess you can close that ticket now. Thanks so much for your patience, help and insight inukshuk :folded_hands: Sin é.