Actions
Demande #6407
ferméSauvegardes en échec pour la VM pouet
Début:
04/05/2024
Echéance:
% réalisé:
10%
Temps estimé:
Description
Alerte de supervision depuis quelques jours concernant les sauvegardes de pouet.
Je viens de lancer une réparation :
BORG_RSH="ssh -p 2242 -A" borg check --repair backup@backup.chapril.org:/srv/backups/$(hostname --fqdn)
C'est une opération que j'ai eu besoin d'effectuer sur plusieurs VMs Chapril ces dernières semaines. Comme si les disques dur de felicette commençaient à montrer des signes de fatigue...
En tout cas, sur pouet, il y a semble-t-il déjà eu des cas où les sauvegardes ont eu des soucis et se sont auto-réparées par la suite on dirait :
=(^-^)=root@pouet:~# ls -1tr /var/log/borgmatic.log* | tail | xargs zcat -f | grep -e 'Successfully ran configuration file' -e CRITICAL
2024-03-31T05:13:13.286417+02:00 pouet borgmatic: CRITICAL ...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430282...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430283...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430284...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430285...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430286...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430287...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430288...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430289...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430290...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430291...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430292...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430293...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430294...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430295...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430296...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/343/3430297...#012Remote: finished segment check at segment 3430297#012Remote: Starting repository index check#012Remote: Index object count mismatch.#012Remote: committed index: 565911 objects#012Remote: rebuilt index: 565912 objects#012Remote: ID: 7d8218df880642842b5670361cb6d48de51cc1b84cd2a4b2c881156e3dce80cc rebuilt index: (2911500, 5177638) committed index: <not found>#012Remote: Finished full repository check, errors found.#012RemoteRepository: 211 B bytes sent, 209.19 MB bytes received, 3 messages sent#012terminating with warning status, rc 1
2024-03-31T05:13:13.286508+02:00 pouet borgmatic: CRITICAL Command 'borg check --last 2 --glob-archives 20* --debug --show-rc ssh://backup@backup.chapril.org/srv/backups/{fqdn}' returned non-zero exit status 1.
2024-03-31T05:13:13.286545+02:00 pouet borgmatic: CRITICAL
2024-03-31T05:13:13.286578+02:00 pouet borgmatic: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
2024-04-01T04:23:37.524610+02:00 pouet borgmatic[1706512]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-02T04:25:25.407814+02:00 pouet borgmatic[2907783]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-03T03:27:19.897343+02:00 pouet borgmatic[4131473]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-04T04:29:37.155047+02:00 pouet borgmatic[1327000]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-05T04:23:01.035596+02:00 pouet borgmatic[2659182]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-06T03:47:21.565270+02:00 pouet borgmatic[3912555]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-07T04:11:53.502045+02:00 pouet borgmatic[830624]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-08T03:46:31.195854+02:00 pouet borgmatic[1897460]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-09T03:38:45.710451+02:00 pouet borgmatic[3190765]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-10T03:51:29.105384+02:00 pouet borgmatic[309119]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-11T04:11:51.786687+02:00 pouet borgmatic[1620947]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-12T03:45:52.293655+02:00 pouet borgmatic[2963050]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-13T04:04:23.123669+02:00 pouet borgmatic[131629]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-14T03:48:07.648094+02:00 pouet borgmatic[1239955]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-15T04:53:55.855124+02:00 pouet borgmatic[2311275]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-16T04:56:16.957306+02:00 pouet borgmatic[3524109]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-17T04:18:34.575256+02:00 pouet borgmatic[506974]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-18T04:13:57.552185+02:00 pouet borgmatic[1711532]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-19T03:39:04.027455+02:00 pouet borgmatic[3032321]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-20T03:34:39.258558+02:00 pouet borgmatic[187763]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-21T04:14:26.023068+02:00 pouet borgmatic[1322038]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-22T04:17:10.850712+02:00 pouet borgmatic[2463472]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-23T03:41:45.856047+02:00 pouet borgmatic[3813928]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-24T04:02:56.471748+02:00 pouet borgmatic[983991]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-25T04:50:44.450402+02:00 pouet borgmatic[2312918]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-26T04:16:07.221873+02:00 pouet borgmatic[3648898]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-27T04:19:07.764071+02:00 pouet borgmatic[795540]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-28T03:59:15.259324+02:00 pouet borgmatic[1958227]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-29T04:05:07.611799+02:00 pouet borgmatic[3167955]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-04-30T03:34:38.302288+02:00 pouet borgmatic[238595]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-05-01T03:45:04.620326+02:00 pouet borgmatic[1633357]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
2024-05-02T05:05:57.885560+02:00 pouet borgmatic: CRITICAL ssh://backup@backup.chapril.org/srv/backups/{fqdn}: Error running actions for repository
2024-05-02T05:05:57.885696+02:00 pouet borgmatic: CRITICAL Command 'borg check --last 2 --glob-archives 20* --debug --show-rc ssh://backup@backup.chapril.org/srv/backups/{fqdn}' returned non-zero exit status 1.
2024-05-02T05:05:57.887265+02:00 pouet borgmatic: CRITICAL /etc/borgmatic.d/root.yaml: Error running configuration file
2024-05-02T05:05:57.887359+02:00 pouet borgmatic: CRITICAL
2024-05-02T05:05:57.887415+02:00 pouet borgmatic: CRITICAL summary:
2024-05-02T05:05:57.887465+02:00 pouet borgmatic: CRITICAL /etc/borgmatic.d/root.yaml: Error running configuration file
2024-05-02T05:05:57.887484+02:00 pouet borgmatic: CRITICAL ssh://backup@backup.chapril.org/srv/backups/{fqdn}: Error running actions for repository
2024-05-02T05:05:57.887885+02:00 pouet borgmatic: CRITICAL ...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449847...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449848...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449849...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449850...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449851...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449852...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449853...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449854...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449855...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449856...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449857...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/344/3449858...#012Remote: finished segment check at segment 3449858#012Remote: Starting repository index check#012Remote: Index object count mismatch.#012Remote: committed index: 565333 objects#012Remote: rebuilt index: 565338 objects#012Remote: ID: 910af220ee980bad5956f10ede538edf974a6b7ccdf4d6c9132084ad4916e62d rebuilt index: (3129635, 8) committed index: <not found>#012Remote: ID: 7d8218df880642842b5670361cb6d48de51cc1b84cd2a4b2c881156e3dce80cc rebuilt index: (2911500, 5177638) committed index: <not found>#012Remote: ID: 28fe3f5035177033b5678f85bc4ea8981496fa1eca473826d7659e4deeb8a1fd rebuilt index: (3385598, 4847974) committed index: <not found>#012Remote: ID: 1300fd6afa1a3ba47feea89009bbcf380493d7c78402a08711db014e09b8d6df rebuilt index: (3240361, 8) committed index: <not found>#012Remote: ID: 387cbd9e40bee8f6ca6f66b14cb1bce7a2df65574bf245619f17d66060072c8d rebuilt index: (3383403, 8) committed index: <not found>#012Remote: Finished full repository check, errors found.#012RemoteRepository: 211 B bytes sent, 209.23 MB bytes received, 3 messages sent#012terminating with warning status, rc 1
2024-05-02T05:05:57.887956+02:00 pouet borgmatic: CRITICAL Command 'borg check --last 2 --glob-archives 20* --debug --show-rc ssh://backup@backup.chapril.org/srv/backups/{fqdn}' returned non-zero exit status 1.
2024-05-02T05:05:57.887977+02:00 pouet borgmatic: CRITICAL
2024-05-02T05:05:57.887997+02:00 pouet borgmatic: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
2024-05-03T04:42:20.296966+02:00 pouet borgmatic: CRITICAL ssh://backup@backup.chapril.org/srv/backups/{fqdn}: Error running actions for repository
2024-05-03T04:42:20.297054+02:00 pouet borgmatic: CRITICAL Command 'borg check --last 2 --glob-archives 20* --debug --show-rc ssh://backup@backup.chapril.org/srv/backups/{fqdn}' returned non-zero exit status 1.
2024-05-03T04:42:20.298290+02:00 pouet borgmatic: CRITICAL /etc/borgmatic.d/root.yaml: Error running configuration file
2024-05-03T04:42:20.298376+02:00 pouet borgmatic: CRITICAL
2024-05-03T04:42:20.298411+02:00 pouet borgmatic: CRITICAL summary:
2024-05-03T04:42:20.298481+02:00 pouet borgmatic: CRITICAL /etc/borgmatic.d/root.yaml: Error running configuration file
2024-05-03T04:42:20.298853+02:00 pouet borgmatic: CRITICAL ssh://backup@backup.chapril.org/srv/backups/{fqdn}: Error running actions for repository
2024-05-03T04:42:20.298974+02:00 pouet borgmatic: CRITICAL ...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450426...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450427...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450428...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450429...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450430...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450431...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450432...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450433...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450434...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450435...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450436...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3450437...#012Remote: finished segment check at segment 3450437#012Remote: Starting repository index check#012Remote: Index object count mismatch.#012Remote: committed index: 565836 objects#012Remote: rebuilt index: 565841 objects#012Remote: ID: 910af220ee980bad5956f10ede538edf974a6b7ccdf4d6c9132084ad4916e62d rebuilt index: (3129635, 8) committed index: <not found>#012Remote: ID: 7d8218df880642842b5670361cb6d48de51cc1b84cd2a4b2c881156e3dce80cc rebuilt index: (2911500, 5177638) committed index: <not found>#012Remote: ID: 28fe3f5035177033b5678f85bc4ea8981496fa1eca473826d7659e4deeb8a1fd rebuilt index: (3385598, 4847974) committed index: <not found>#012Remote: ID: 1300fd6afa1a3ba47feea89009bbcf380493d7c78402a08711db014e09b8d6df rebuilt index: (3240361, 8) committed index: <not found>#012Remote: ID: 387cbd9e40bee8f6ca6f66b14cb1bce7a2df65574bf245619f17d66060072c8d rebuilt index: (3383403, 8) committed index: <not found>#012Remote: Finished full repository check, errors found.#012RemoteRepository: 211 B bytes sent, 209.24 MB bytes received, 3 messages sent#012terminating with warning status, rc 1
2024-05-03T04:42:20.299051+02:00 pouet borgmatic: CRITICAL Command 'borg check --last 2 --glob-archives 20* --debug --show-rc ssh://backup@backup.chapril.org/srv/backups/{fqdn}' returned non-zero exit status 1.
2024-05-03T04:42:20.299076+02:00 pouet borgmatic: CRITICAL
2024-05-03T04:42:20.299101+02:00 pouet borgmatic: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
2024-05-04T04:31:16.590176+02:00 pouet borgmatic: CRITICAL ssh://backup@backup.chapril.org/srv/backups/{fqdn}: Error running actions for repository
2024-05-04T04:31:16.590245+02:00 pouet borgmatic: CRITICAL Command 'borg check --last 2 --glob-archives 20* --debug --show-rc ssh://backup@backup.chapril.org/srv/backups/{fqdn}' returned non-zero exit status 1.
2024-05-04T04:31:16.591526+02:00 pouet borgmatic: CRITICAL /etc/borgmatic.d/root.yaml: Error running configuration file
2024-05-04T04:31:16.591608+02:00 pouet borgmatic: CRITICAL
2024-05-04T04:31:16.591646+02:00 pouet borgmatic: CRITICAL summary:
2024-05-04T04:31:16.591663+02:00 pouet borgmatic: CRITICAL /etc/borgmatic.d/root.yaml: Error running configuration file
2024-05-04T04:31:16.592123+02:00 pouet borgmatic: CRITICAL ssh://backup@backup.chapril.org/srv/backups/{fqdn}: Error running actions for repository
2024-05-04T04:31:16.592207+02:00 pouet borgmatic: CRITICAL ...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451005...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451006...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451007...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451008...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451009...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451010...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451011...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451012...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451013...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451014...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451015...#012Remote: checking segment file /srv/backups/pouet.cluster.chapril.org/data/345/3451016...#012Remote: finished segment check at segment 3451016#012Remote: Starting repository index check#012Remote: Index object count mismatch.#012Remote: committed index: 566447 objects#012Remote: rebuilt index: 566452 objects#012Remote: ID: 910af220ee980bad5956f10ede538edf974a6b7ccdf4d6c9132084ad4916e62d rebuilt index: (3129635, 8) committed index: <not found>#012Remote: ID: 7d8218df880642842b5670361cb6d48de51cc1b84cd2a4b2c881156e3dce80cc rebuilt index: (2911500, 5177638) committed index: <not found>#012Remote: ID: 28fe3f5035177033b5678f85bc4ea8981496fa1eca473826d7659e4deeb8a1fd rebuilt index: (3385598, 4847974) committed index: <not found>#012Remote: ID: 1300fd6afa1a3ba47feea89009bbcf380493d7c78402a08711db014e09b8d6df rebuilt index: (3240361, 8) committed index: <not found>#012Remote: ID: 387cbd9e40bee8f6ca6f66b14cb1bce7a2df65574bf245619f17d66060072c8d rebuilt index: (3383403, 8) committed index: <not found>#012Remote: Finished full repository check, errors found.#012RemoteRepository: 211 B bytes sent, 209.25 MB bytes received, 3 messages sent#012terminating with warning status, rc 1
2024-05-04T04:31:16.592273+02:00 pouet borgmatic: CRITICAL Command 'borg check --last 2 --glob-archives 20* --debug --show-rc ssh://backup@backup.chapril.org/srv/backups/{fqdn}' returned non-zero exit status 1.
2024-05-04T04:31:16.592292+02:00 pouet borgmatic: CRITICAL
2024-05-04T04:31:16.592314+02:00 pouet borgmatic: CRITICAL Need some help? https://torsion.org/borgmatic/#issues
Actions
#1
Mis à jour par pitchum . il y a 8 mois
- Statut changé de En cours de traitement à Résolu
- Assigné à mis à pitchum .
Ce matin, tout est OK.
root@pouet:~# tail /var/log/borgmatic.log 2024-05-05T06:32:32.750444+02:00 pouet borgmatic[2855398]: Writing check time at /root/.borgmatic/checks/0159fe6c5a51e9d83bd3d3f5722197e16a5803450954376049ec4b430e16e07b/repository 2024-05-05T06:32:32.750678+02:00 pouet borgmatic[2855398]: Writing check time at /root/.borgmatic/checks/0159fe6c5a51e9d83bd3d3f5722197e16a5803450954376049ec4b430e16e07b/archives 2024-05-05T06:32:32.750713+02:00 pouet borgmatic[2855398]: /etc/borgmatic.d/root.yaml: Running command for post-check hook 2024-05-05T06:32:32.750739+02:00 pouet borgmatic[2855398]: echo "Succeeded root checks at $(date -Iseconds)" 2024-05-05T06:32:32.752181+02:00 pouet borgmatic[2855398]: Succeeded root checks at 2024-05-05T06:32:32+02:00 2024-05-05T06:32:32.752226+02:00 pouet borgmatic: WARNING Succeeded root checks at 2024-05-05T06:32:32+02:00 2024-05-05T06:32:32.752271+02:00 pouet borgmatic[2855398]: /etc/borgmatic.d/root.yaml: No commands to run for post-actions hook 2024-05-05T06:32:32.752295+02:00 pouet borgmatic[2855398]: /etc/borgmatic.d/root.yaml: No commands to run for post-everything hook 2024-05-05T06:32:32.752369+02:00 pouet borgmatic[2855398]: summary: 2024-05-05T06:32:32.752391+02:00 pouet borgmatic[2855398]: /etc/borgmatic.d/root.yaml: Successfully ran configuration file
Actions