Project

General

Profile

Anomalie #5094

Opération de mantenance sur base de donnée

Added by Quentin Gibeaux 11 months ago. Updated 10 months ago.

Status:
Fermé
Priority:
Normale
Start date:
01/06/2021
Due date:
% Done:

0%

Estimated time:

Description

Lors de la mise à jour vers 3.3.0 et la migration de la base, il y avait un warning concernant la consistance des données, mais un message disait que c'était pas forcément grave.

Après relecture des notes de release je vois qu'une commande "fix-duplicates" a été intégrée à tootctl dans la version 3.2.2 qui était passée inaperçue entre deux release candidates.

Tout le détail est ici : https://docs.joinmastodon.org/admin/troubleshooting/index-corruption/

History

#1

Updated by Quentin Gibeaux 11 months ago

J'ai procédé à un backup de la db et ait lancé la commande à chaud, j'ai eu des erreurs de status dupliqués, puis j'ai vu qu'il fallait la lancer offline (le helper le précisait pas, seule la page https://docs.joinmastodon.org/admin/troubleshooting/index-corruption/ le précise).

mastodon@pouet:~/live$ RAILS_ENV=production bin/tootctl maintenance fix-duplicates

(…)
Restoring statuses indexes…
Traceback (most recent call last):
        24: from bin/tootctl:8:in `<main>'
        23: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
        22: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        21: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        20: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        19: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:243:in `block in subcommand'
        18: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:116:in `invoke'
        17: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        16: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        15: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        14: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:159:in `fix_duplicates'
        13: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:445:in `deduplicate_statuses!'
        12: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/schema_statements.rb:466:in `add_index'
        11: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:73:in `execute'
        10: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:571:in `log'
         9: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
         8: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `block in log'
         7: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `synchronize'
         6: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:581:in `block (2 levels) in log'
         5: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:74:in `block in execute'
         4: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
         3: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
         2: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
         1: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute'  
/home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `exec': ERREUR:  n'a pas pu créer l'index unique « index_statuses_on_uri » (PG::UniqueViolation)
DETAIL:  La clé (uri)=(https://framapiaf.org/users/Yahiko/statuses/105508116838601114) est dupliquée.
        24: from bin/tootctl:8:in `<main>'
        23: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
        22: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        21: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        20: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        19: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:243:in `block in subcommand'
        18: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:116:in `invoke'
        17: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        16: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        15: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        14: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:159:in `fix_duplicates'
        13: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:445:in `deduplicate_statuses!'
        12: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/schema_statements.rb:466:in `add_index'
        11: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:73:in `execute'
        10: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:571:in `log'
         9: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
         8: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `block in log'
         7: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `synchronize'
         6: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:581:in `block (2 levels) in log'
         5: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:74:in `block in execute'
         4: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
         3: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
         2: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
         1: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute'  
/home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `exec': ERREUR:  n'a pas pu créer l'index unique « index_statuses_on_uri » (PG::UniqueViolation)
DETAIL:  La clé (uri)=(https://framapiaf.org/users/Yahiko/statuses/105508116838601114) est dupliquée.
        24: from bin/tootctl:8:in `<main>'
        23: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/base.rb:485:in `start'
        22: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        21: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        20: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        19: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:243:in `block in subcommand'
        18: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:116:in `invoke'
        17: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor.rb:392:in `dispatch'
        16: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/invocation.rb:127:in `invoke_command'
        15: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/thor-1.0.1/lib/thor/command.rb:27:in `run'
        14: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:159:in `fix_duplicates'
        13: from /home/mastodon/live/lib/mastodon/maintenance_cli.rb:445:in `deduplicate_statuses!'
        12: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/schema_statements.rb:466:in `add_index'
        11: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:73:in `execute'
        10: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:571:in `log'
         9: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
         8: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `block in log'
         7: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:580:in `synchronize'
         6: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/abstract_adapter.rb:581:in `block (2 levels) in log'
         5: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:74:in `block in execute'
         4: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
         3: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
         2: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
         1: from /home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `block (2 levels) in execute'  
/home/mastodon/live/vendor/bundle/ruby/2.7.0/gems/activerecord-5.2.4.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:75:in `exec': PG::UniqueViolation: ERREUR:  n'a pas pu créer l'index unique « index_statuses_on_uri » (ActiveRecord::RecordNotUnique)
DETAIL:  La clé (uri)=(https://framapiaf.org/users/Yahiko/statuses/105508116838601114) est dupliquée.
: CREATE UNIQUE INDEX  "index_statuses_on_uri" ON "statuses"  ("uri")
mastodon@pouet:~/live$

J'ai donc prévenu les utilisateurs, coupé mastodon, relancé la commande de maintenance, et cette fois-ci elle est arrivée à son terme.

mastodon@pouet:~/live$ RAILS_ENV=production bin/tootctl maintenance fix-duplicates
This task will take a long time to run and is potentially destructive.
Please make sure to stop Mastodon and have a backup.
Continue? Yes
Deduplicating accounts… for local accounts, you will be asked to chose which account to keep unchanged.
Restoring index_accounts_on_username_and_domain_lower…
Deduplicating user records…
Restoring users indexes…
Removing duplicate account domain blocks…
Restoring account domain blocks indexes…
Removing duplicate account identity proofs…
Restoring account identity proofs indexes…
Removing duplicate account identity proofs…
Restoring announcement_reactions indexes…
Deduplicating conversations…
Restoring conversations indexes…
Deduplicating custom_emojis…
Restoring custom_emojis indexes…
Deduplicating custom_emoji_categories…
Restoring custom_emoji_categories indexes…
Deduplicating domain_allows…
Restoring domain_allows indexes…
Deduplicating domain_allows…
Restoring domain_blocks indexes…
Deduplicating unavailable_domains…
Restoring domain_allows indexes…
Deduplicating email_domain_blocks…
Restoring email_domain_blocks indexes…
Deduplicating media_attachments…
Restoring media_attachments indexes…
Deduplicating preview_cards…
Restoring preview_cards indexes…
Deduplicating statuses…
Restoring statuses indexes…
Deduplicating tags…  
Restoring tags indexes…
Deduplicating webauthn_credentials…
Restoring webauthn_credentials indexes…
Finished!

J'ai aussi lancé les deux commandes postgresql de la page ci dessus, qui analysent la db pour vérifier qu'elle n'est pas concernée par le problème, et je n'ai pas vu d'erreur.

=(^-^)=root@pouet:~# su - postgres
postgres@pouet:~$ psql mastodon-production

mastodon-production=# CREATE EXTENSION IF NOT EXISTS amcheck;
SELECT bt_index_check(c.oid)
FROM pg_index i
JOIN pg_class c ON i.indexrelid = c.oid
WHERE c.relname IN ('index_account_domain_blocks_on_account_id_and_domain',
  'index_account_proofs_on_account_and_provider_and_username',
  'index_accounts_on_username_and_domain_lower', 'index_accounts_on_uri',
  'index_accounts_on_url', 'index_conversations_on_uri',
  'index_custom_emoji_categories_on_name',
  'index_custom_emojis_on_shortcode_and_domain',
  'index_devices_on_access_token_id', 'index_domain_allows_on_domain',
  'index_domain_blocks_on_domain', 'index_email_domain_blocks_on_domain',
  'index_invites_on_code', 'index_markers_on_user_id_and_timeline',
  'index_media_attachments_on_shortcode', 'index_preview_cards_on_url',
  'index_statuses_on_uri', 'index_tags_on_name_lower',
  'index_tombstones_on_uri', 'index_unavailable_domains_on_domain',
  'index_users_on_email', 'index_webauthn_credentials_on_external_id'
);
CREATE EXTENSION
 bt_index_check
----------------
(22 lignes)

mastodon-production=#
mastodon-production=# CREATE EXTENSION IF NOT EXISTS amcheck;
SELECT bt_index_parent_check(c.oid)
FROM pg_index i
JOIN pg_class c ON i.indexrelid = c.oid
WHERE c.relname IN ('index_account_domain_blocks_on_account_id_and_domain',
  'index_account_proofs_on_account_and_provider_and_username',
  'index_accounts_on_username_and_domain_lower', 'index_accounts_on_uri',
  'index_accounts_on_url', 'index_conversations_on_uri',
  'index_custom_emoji_categories_on_name',
  'index_custom_emojis_on_shortcode_and_domain',
  'index_devices_on_access_token_id', 'index_domain_allows_on_domain',
  'index_domain_blocks_on_domain', 'index_email_domain_blocks_on_domain',
  'index_invites_on_code', 'index_markers_on_user_id_and_timeline',
  'index_media_attachments_on_shortcode', 'index_preview_cards_on_url',
  'index_statuses_on_uri', 'index_tags_on_name_lower',
  'index_tombstones_on_uri', 'index_unavailable_domains_on_domain',
  'index_users_on_email', 'index_webauthn_credentials_on_external_id'
);
NOTICE:  l'extension « amcheck » existe déjà, poursuite du traitement
CREATE EXTENSION

 bt_index_parent_check
-----------------------
(22 lignes)

mastodon-production=#

J'ai relancé le service mastodon.

NB: ils préviennent qu'il est possible qu'il faille relancer ce fix-duplicates à chaque màj de libc

#2

Updated by Quentin Gibeaux 11 months ago

  • Status changed from Nouveau to Résolu
#3

Updated by Quentin Gibeaux 10 months ago

  • Status changed from Résolu to Fermé
#4

Updated by Christian P. Momon 10 months ago

  • Target version changed from Backlog to Sprint 2021 janvier

Also available in: Atom PDF