Hi,
I’m using tropy for about a month now and I’m repeatedly facing the SQLITE_CORRUPT: database disk image is malformed error message (about once every week). Since I’m saving the .tpy file on the cloud it’s relatively easy for me to restore a working version, but I would like to reduce the frequency of this accidents. Any guesses why this is happening so often?
I’m running tropy on a windows x64 installation.
Thanks in advance,
Luca
Yes, this sounds bad, we should definitely try to figure out why this is happening.
When this error occurs, does it happen just when you start Tropy (or open the project), or does it happen while you’re working in it, that is, when you opened and used the project for some time already? If it is the latter, does it happen suddenly while you’re working or after a longer break, that is, you had the project open in Tropy for a while, maybe you were using different applications, or even put the computer to sleep, etc., and then the error occurs when you go back to Tropy and resume your work?
One possibility I can think of is that another application alters the file while Tropy is accessing it, or that there are some issues at the file-system level that manifest if Tropy has the file open for a long period of time (though Tropy should close the connections after some idle time). Especially since this is happening at a fixed interval, like once a week, it could be caused by some tool used for backups or maintenance, or the way the network share or cloud storage app is configured.
You could also post your log file here when the error occurs the next time. This might give us additional clues.
Many thanks for your thoughts!
The error occurs suddenly while editing an object, usually when I was filling out metadata fields but once also after I added a photo.
I am using different applications (Zotero, note taking apps) while working with tropy and I’m constantly switching between the apps.
In my cloud storage app I realized that a lot of rewriting of the .tpy file is taking place. For every metadata-field I edit a new version of the .tpy file is saved in the cloud. This easilly sums up to 10 new versions a Minute. Could this be the problem? Nevertheless, the error occurs also when I edit very slowly, filling out about one metadata field per minute.
I will post my log file when the error occurs the next time.
Yes, Tropy opens the file and writes changes to it as they take place. So, the file will change whenever you change a metadata field, add a tag, write a note, etc. It’s possible that the frequent updates are causing issues for your cloud storage app – you could try turning the cloud sync off temporarily, while you work on the file, to see if you can still reproduce the file corruption this way.
Hi,
unfortunately, I haven’t solved the problem. I’m not sure whether syncing less frequently helps. One day it happened while I was syncing manually.
Can you deduce something from the error messages posted below? Could it be a Windows 10 related issue or do you still think that it’s caused by my cloud storage app?
Thanks!
today
{“msg”:“unhandled error: SQLITE_IOERR: disk I/O error”,“stack”:“Error: SQLITE_IOERR: disk I/O error”,“system”:“Windows_NT 10.0.19043 (x64)”,“time”:1645544918840,“version”:“1.11.1”}
{“msg”:“unhandled error: payload is not iterable”,“stack”:“TypeError: payload is not iterable\n at Object.touch (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:1785:18)\n at itemsReducer (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\views\project.js:192:19)\n at combination (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:862:29)\n at dispatch (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:764:22)\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:4184:22\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:360:12\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:320:12\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:928:16\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:330:51\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\form-2f299276.js:32:51\n at dispatch (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:905:28)\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:2939:12\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3544:54\n at exec$1 (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3255:5)\n at flush (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3285:5)\n at asap (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3264:5)”,“system”:“Windows_NT 10.0.19043 (x64)”,“time”:1645544955082,“version”:“1.11.1”}
some days ago
{"msg":“unhandled error: SQLITE_ERROR: cannot rollback - no transaction is active”,“stack”:“Error: SQLITE_ERROR: cannot rollback - no transaction is active”,“system”:“Windows_NT 10.0.19043 (x64)”,“time”:1644597757180,“version”:“1.11.1”}
{“msg”:“unhandled error: payload is not iterable”,“stack”:“TypeError: payload is not iterable\n at Object.touch (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:1785:18)\n at itemsReducer (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\views\project.js:192:19)\n at combination (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:862:29)\n at dispatch (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:764:22)\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:4184:22\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:360:12\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:320:12\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:928:16\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:330:51\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\form-2f299276.js:32:51\n at dispatch (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\main-798bc84f.js:905:28)\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:2939:12\n at C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3544:54\n at exec$1 (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3255:5)\n at flush (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3285:5)\n at asap (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\project-caffde5c.js:3264:5)”,“system”:“Windows_NT 10.0.19043 (x64)”,“time”:1644597977720,“version”:“1.11.1”}
The first error is a fundamental disk I/O error while Tropy is writing to the project file (the second error looks like a follow-up error, I guess there was probably another disk I/O error before it; this might point to an issue with Tropy that we should address, but it’s just something that fails while trying to recover from the original error). Such errors are often caused by real hardware issues (typically by a faulty hard drive) but they can spuriously happen for other reasons, for example when your system is under heavy load or if the file handle held by Tropy is interfered with. We haven’t seen any reports like this before so I doubt that it is a general issue with Tropy on Windows 10. Furthermore, if this were an actual hardware issue, I’d expect you to notice file access issues elsewhere too, so my best guess is that this is caused by another process accessing the file while Tropy is writing to it.
If you’re sure that this happened while the file was not being accessed by the cloud storage app then I think we can rule that out. Could you still tell me which cloud storage this is? Similarly, are you aware of any other service that might have been accessing the file? Do you have more than one hard drive? If so, it would be helpful to know if the error is reproducible when you access the file from another drive (using a external drive or removable media would also serve this purpose).
Thanks again for your time!
Actually, I have similar issues in other apps. Sometimes when I’m writing notices in Zotero the pragraph i just typed disappears. In Zotero this doesn’t lead to a file-corruption, though. My Windows One Note-app also occasionally shuts itself down (also without corruption). Does this indicate that there is a hardware issue/ general issue?
No other service is accessing the file and I have only one hard drive. As a cloud storage I’m using a nextcloud-installtion run by my University (tubcloud).
I don’t think this necessarily points towards hardware issue, but I also would not rule it out. If similar issues happen elsewhere, I’d definitely suggest to run some diagnostics on your hard drive and file system. I’d also check if the device driver used is the appropriate one for your hard drive manufacturer/model and if it is up to date.
I’m sorry to bother you again. I thought updating my OS might help but unfortunately the problem occurred again. Now the app shows a little exclamation mark and suggests to check the database manually. Could you take a look at it? https://tubcloud.tu-berlin.de/s/RnMxmxiE8gQgMKy
Here’s also the logfile of the last corruption.
{“level”:30,“time”:1646052643786,“type”:“browser”,“name”:“main”,“args”:[],“opts”:{“env”:“production”,“autoUpdates”:true,“webgl”:false,“debug”:false,“trace”:false,“data”:“C:\Users\lucaa\AppData\Roaming\Tropy”,“cache”:“C:\Users\lucaa\AppData\Roaming\Tropy\cache”,“logs”:“C:\Users\lucaa\AppData\Roaming\Tropy\log”},“version”:“1.11.1”,“msg”:“main.init 1.11.1 Windows_NT 10.0.22000 (x64)”}
{“level”:30,“time”:1646052643833,“type”:“browser”,“name”:“main”,“msg”:“plugins scanned: 0”}
{“level”:30,“time”:1646052643835,“type”:“browser”,“name”:“main”,“msg”:“app state restored”}
{“level”:30,“time”:1646052643836,“type”:“browser”,“name”:“main”,“file”:“C:\Users\lucaa\tubCloud\Gegendrucke.tpy”,“msg”:“open new project window”}
{“level”:30,“time”:1646052644067,“type”:“renderer”,“name”:“project”,“dpx”:1.5,“opts”:{“env”:“production”,“documents”:“C:\Users\lucaa\Documents”,“pictures”:“C:\Users\lucaa\Pictures”,“theme”:“light”,“aqua”:false,“contrast”:false,“dark”:false,“scrollbars”:true,“vibrancy”:false,“file”:“C:\Users\lucaa\tubCloud\Gegendrucke.tpy”,“recent”:[“C:\Users\lucaa\tubCloud\Gegendrucke.tpy”,“C:\Users\lucaa\Documents\Gegendrucke.tpy”,“C:\Users\lucaa\tubCloud\Test.tpy”,“C:\Users\lucaa\Documents\Test.tpy”],“data”:“C:\Users\lucaa\AppData\Roaming\Tropy”,“debug”:false,“dev”:false,“cache”:“C:\Users\lucaa\AppData\Roaming\Tropy\cache”,“plugins”:“C:\Users\lucaa\AppData\Roaming\Tropy\plugins”,“fontSize”:“13px”,“frameless”:false,“level”:“info”,“locale”:“de”,“log”:“C:\Users\lucaa\AppData\Roaming\Tropy\log\tropy.log”,“uuid”:“ba1aec20-3bc7-11ec-ac6f-8d2e638a4b68”,“update”:{},“version”:“1.11.1”,“webgl”:true,“zoom”:1,“maximizable”:true,“minimizable”:true},“msg”:“project.init”}
{“level”:30,“time”:1646052644911,“type”:“renderer”,“name”:“project”,“msg”:“plugins scanned: 0”}
{“level”:30,“time”:1646052644911,“type”:“renderer”,“name”:“project”,“msg”:“plugins loaded: 0”}
{“level”:30,“time”:1646052644913,“type”:“browser”,“name”:“main”,“msg”:“ready after 338.322021484375ms”}
{“level”:30,“time”:1646052645241,“type”:“renderer”,“name”:“project”,“msg”:“restoring recent@tropy”}
{“level”:30,“time”:1646052645242,“type”:“renderer”,“name”:“project”,“msg”:“restoring settings@tropy”}
{“level”:30,“time”:1646052645242,“type”:“renderer”,“name”:“project”,“msg”:“restoring ui@tropy”}
{“level”:30,“time”:1646052645266,“type”:“renderer”,“name”:“project”,“mode”:“w+”,“msg”:“open db C:\Users\lucaa\AppData\Roaming\Tropy\ontology.db”}
{“level”:30,“time”:1646052645407,“type”:“renderer”,“name”:“project”,“msg”:“Esper.instance created using webgl renderer”}
{“level”:30,“time”:1646052645445,“type”:“renderer”,“name”:“project”,“mode”:“w”,“msg”:“open db C:\Users\lucaa\tubCloud\Gegendrucke.tpy”}
{“level”:30,“time”:1646052645451,“type”:“renderer”,“name”:“project”,“msg”:“db query took 157ms”,“query”:"\n SELECT count(*) AS count FROM sqlite_schema",“ms”:157}
{“level”:30,“time”:1646052645455,“type”:“renderer”,“name”:“project”,“msg”:“project ready 1586ms [dom:994ms win:48ms req:328ms]”}
{“level”:30,“time”:1646052645464,“type”:“renderer”,“name”:“project”,“msg”:“restoring project.watch@f80a1ec1-de9f-4d3b-a31c-19c1128d6ecc”}
{“level”:30,“time”:1646052645521,“type”:“renderer”,“name”:“project”,“msg”:“restoring nav@f80a1ec1-de9f-4d3b-a31c-19c1128d6ecc”}
{“level”:30,“time”:1646052645522,“type”:“renderer”,“name”:“project”,“msg”:“restoring notepad@f80a1ec1-de9f-4d3b-a31c-19c1128d6ecc”}
{“level”:30,“time”:1646052645522,“type”:“renderer”,“name”:“project”,“msg”:“restoring esper@f80a1ec1-de9f-4d3b-a31c-19c1128d6ecc”}
{“level”:30,“time”:1646052645522,“type”:“renderer”,“name”:“project”,“msg”:“restoring imports@f80a1ec1-de9f-4d3b-a31c-19c1128d6ecc”}
{“level”:30,“time”:1646052645523,“type”:“renderer”,“name”:“project”,“msg”:“restoring sidebar@f80a1ec1-de9f-4d3b-a31c-19c1128d6ecc”}
{“level”:30,“time”:1646052645523,“type”:“renderer”,“name”:“project”,“msg”:“restoring panel@f80a1ec1-de9f-4d3b-a31c-19c1128d6ecc”}
{“level”:30,“time”:1646052645548,“type”:“renderer”,“name”:“project”,“mode”:“w”,“msg”:“open db C:\Users\lucaa\tubCloud\Gegendrucke.tpy”}
{“level”:30,“time”:1646052645550,“type”:“renderer”,“name”:“project”,“mode”:“w”,“msg”:“open db C:\Users\lucaa\tubCloud\Gegendrucke.tpy”}
{“level”:30,“time”:1646052703849,“type”:“browser”,“name”:“main”,“msg”:“checking for updates…”}
{“level”:30,“time”:1646052706550,“type”:“browser”,“name”:“main”,“msg”:“no updates available”}
{“level”:40,“time”:1646053397595,“type”:“renderer”,“name”:“project”,“msg”:“SLOW: photo.create#182”}
{“level”:40,“time”:1646053416073,“type”:“renderer”,“name”:“project”,“stack”:“SyntaxError: Unexpected token u in JSON at position 0\n at JSON.parse ()\n at handlePaste (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\views\project.js:10960:25)”,“text”:“unten rechts auf der Stufe, mit Rötel: P. v. Lint D L a Roma [unleserlich]; verso, mit Graphit: 177”,“msg”:“pasted unsupported text”}
{“level”:40,“time”:1646053698073,“type”:“renderer”,“name”:“project”,“msg”:“SLOW: photo.create#277”}
{“level”:40,“time”:1646053714666,“type”:“renderer”,“name”:“project”,“stack”:“SyntaxError: Unexpected token T in JSON at position 0\n at JSON.parse ()\n at handlePaste (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\views\project.js:10960:25)”,“text”:“Teil einer Serie von 21 Vogeldarstellungen bei denen es sich vermutlich um Blätter nach demselben Design handelt, wie bei den Vogelmotiven aus der Werkstatt von J. Teyler, zu finden z.B. in dem sogenannten Teyler-Band siehe Rijksprentenkabinet, Rijksmuseum, Amsterdam (Inv. Nr. RP-P-1955-251). Die Vögel in beiden Gruppen weisen große Ähnlichkeit in der Ausführung auf, erscheinen aber nie in der gleichen Zusammenstellung und sind zum Teil im Gegensinn gestochen. Abdruck in Rot-Braun.”,“msg”:“pasted unsupported text”}
{“level”:40,“time”:1646053724967,“type”:“renderer”,“name”:“project”,“stack”:“SyntaxError: Unexpected token V in JSON at position 0\n at JSON.parse ()\n at handlePaste (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\views\project.js:10960:25)”,“text”:“Vergleiche den Kupferstich à la poupée aus der Werkstatt von J. Teyler (gegenseitig); Amsterdam, Rijksmuseum, Rijksprentenkabinet (Inv. Nr. RP-P-1939-1344)”,“msg”:“pasted unsupported text”}
{“level”:40,“time”:1646054032909,“type”:“renderer”,“name”:“project”,“msg”:“SLOW: photo.create#381”}
{“level”:40,“time”:1646054380429,“type”:“renderer”,“name”:“project”,“stack”:“SyntaxError: Unexpected token R in JSON at position 0\n at JSON.parse ()\n at handlePaste (C:\Users\lucaa\AppData\Local\tropy\app-1.11.1\resources\app.asar\lib\views\project.js:10960:25)”,“text”:“RP-P-2014-23”,“msg”:“pasted unsupported text”}
{“level”:40,“time”:1646054777631,“type”:“renderer”,“name”:“project”,“action”:“metadata.save”,“meta”:{“done”:true,“rel”:523,“was”:1646054777592,“seq”:527,“now”:1646054777630},“msg”:“metadata.save failed: SQLITE_CORRUPT: database disk image is malformed”,“stack”:“Error: SQLITE_CORRUPT: database disk image is malformed”}
{“level”:40,“time”:1646054789736,“type”:“renderer”,“name”:“project”,“stack”:“Error: SQLITE_CORRUPT: database disk image is malformed”,“msg”:“failed to import “C:\Users\lucaa\tubCloud\Bilder\283.jpg””}
{“level”:40,“time”:1646054789736,“type”:“renderer”,“name”:“project”,“msg”:“SLOW: photo.create#528”}
Thanks! I looked at the project file and it really looks like a file corruption caused by a failure while writing to the file. The database is looks good, but there is some malformed or missing data in the indices related to metadata value. So my guess is there was an I/O error while saving a change you made to your metadata.
Here is a link to the recovered version of the project file. Have you tried moving the file to a different drive? Unless we can reproduce this issue on a different drive, my guess is that there is an underlying issue with the drive in question.
Hi inukshuk,
unfortunately, the file got corrupted again. Could you restore it again? Does it still point towards an I/O error? I will move the file to a different drive, now.
https://tubcloud.tu-berlin.de/s/85cjLn42fC2cftz
Thanks!