:title: Gerrit .. _gerrit: Gerrit ###### Gerrit is the code review system used by the OpenStack project. For a full description of how the system fits into the OpenStack workflow, see `the development workflow guide `_. This section describes how Gerrit is configured for use in the OpenStack project and the tools used to manage that configuration. At a Glance =========== :Hosts: * https://review.opendev.org :Ansible: * :git_file:`playbooks/service-review.yaml` * :git_file:`playbooks/roles/gerrit` :Configuration: * :git_file:`playbooks/roles/gerrit/templates/projects.ini.j2` * :config:`gerrit/projects.yaml` :Projects: * http://code.google.com/p/gerrit/ :Bugs: * https://storyboard.openstack.org/#!/project/715 * http://code.google.com/p/gerrit/issues/list :Resources: * `Gerrit Documentation `_ Installation ============ Gerrit is installed and configured by Ansible, using a Docker container that contains the Java WAR file. Cinder Volumes -------------- The Gerrit installation at /home/gerrit2 is located on a Cinder volume. See :ref:`cinder` for details on volume management. Note that SSD volumes are used (and they have a minimum size of 100G). Groups ------ A number of system-wide groups are configured in Gerrit (rather than via Puppet). When installing a new Gerrit, you should create these by hand (and capture their UUID - you will need them to setup the ACLs later). The `Project Bootstrappers` group grants all the permissions needed to set up a new project. Normally, the OpenStack Project Creater account is the only member of this group, but members of the `Administrators` group may temporarily add themselves in order to correct problems with automatic project creation. The `Third-Party CI` group is used to grant +/-1 Verified access to external testing tools on a sandbox project. The `Voting Third-Party CI` group is used to grant +/-1 Verified access to external testing tools for all projects. The `Continuous Integration Tools` group contains Zuul and any other CI tools that get +2/-2 access on reviews. Users ----- The first user to log in becomes an administrator. Be sure to set an account name and add ssh keys - you'll need those. Once you've created your groups you should create the ``openstack-project-creator`` account by hand (the account name is referenced from :git_file:`playbooks/roles/gerrit/templates/projects.ini.j2`) using .. code-block:: shell-session $ cat $pubkey | ssh -p 29418 $USER@$HOST gerrit create-account \ --group "'Project Bootstrappers'" \ --group Administrators \ --full-name "'Project Creator'" \ --email openstack-infra@lists.openstack.org \ --ssh-key - openstack-project-creator .. _gerrit_github_integration: GitHub Integration ================== Gerrit replicates to GitHub by pushing to a standard Git remote. The GitHub projects are configured to allow only the Gerrit user to push. Pull requests can not be disabled for a project in Github, so instead we have a script that runs from cron to close any open pull requests with instructions to use Gerrit. These are both handled automatically by :ref:`jeepyb`. Note that the user running Gerrit will need to accept the GitHub host keys. e.g. .. code-block:: shell-session $ sudo su - gerrit2 $ ssh github.com Troubleshooting --------------- When creating a new project, there can be times where the :ref:`jeepyb` automation to create the GitHub project can fail, and leave the project improperly configured. This can cause replication to GitHub to fail. The project in GitHub will be created, but will appear empty. When trying replication from Gerrit, it will show a `Permission denied` error when trying to push content. To solve that, following steps are needed: #. Login into ``github.com``, using ``openstack-project-creator`` user. #. Navigate to the failed repository, and enter on ``Settings > Collaborators & teams`` option. #. Add Gerrit as Team member to that project. After the team has been added, project will start replicating successfully to GitHub. Gerrit IRC Bot ============== Gerritbot consumes the Gerrit event stream and announces relevant events on IRC. :ref:`gerritbot` is an ``OpenDev`` project and is also available on Pypi. Launchpad Bug Integration ========================= In addition to the hyperlinks provided by the regex in ``gerrit.config``, we use a Gerrit hook to update Launchpad bugs when changes referencing them are applied. This is managed by the :ref:`jeepyb` project. Storyboard Integration ====================== We use the Gerrit its-storyboard_ plugin to update :ref:`storyboard` stories and tasks when changes referencing them are applied. .. _its-storyboard: https://review.opendev.org/plugins/its-storyboard/Documentation/index.html New Project Creation ==================== Gerrit project creation is now managed through changes to the ``openstack/project-config`` repository. :ref:`jeepyb` handles automatically creating any new projects defined in the configuration files. .. _acl: Access Controls =============== High level goals: #. Anonymous users can read all projects. #. All registered users can perform informational code review (+/-1) on any project. #. Zuul can perform verification (blocking or approving: +/-2). #. Third Party CI systems can perform informational verification (+/-1). #. All registered users can create changes. #. Members of ``$PROJECT-core`` group can perform full code review (blocking or approving: +/- 2), and submit changes to be merged. #. Drivers (PTL and delegates) of client library projects should be able to add tags (which are automatically used to trigger releases). The global Gerrit permissions set out the high level goals (and manage-projects can then override this on a per project basis as needed). To setup the global permissions, first create the groups covered above under Groups. You need to grant yourself enough access to replace the ACLs over ssh (we use SSH because it's fast, and it gets syntax checked). #. Visit ``https://$HOST/#/admin/projects/All-Projects,access`` and click on Edit. #. Look for the reference to ``refs/meta/config``, click on the drop-box for ``add permission`` and choose ``PUSH``. #. Type in Administrators as the group name #. Click on Add #. Click on Save Changes Then... we need to fetch the All-Projects ACLs, update them, then push the updates back into Gerrit .. code-block:: shell-session $ export USER=$your_gerrit_user $ export HOST=$your_gerrit_host $ cd $anywhereyoulike $ mkdir All-Projects-ACLs $ cd All-Projects-ACLs $ git init $ git remote add gerrit ssh://$USER@$HOST:29418/All-Projects.git $ git fetch gerrit +refs/meta/*:refs/remotes/gerrit-meta/* $ git checkout -b config remotes/gerrit-meta/config There will be two interesting files, ``groups`` and ``project.config``. ``groups`` contains UUIDs and names of groups that will be referenced in ``project.config``. UUIDs can be found on the group page in Gerrit. Next, edit ``project.config`` to look like:: [access "refs/*"] create = group Project Bootstrappers forgeAuthor = group Registered Users forgeCommitter = group Project Bootstrappers push = +force group Project Bootstrappers pushMerge = group Project Bootstrappers pushSignedTag = group Project Bootstrappers pushTag = group Continuous Integration Tools pushTag = group Project Bootstrappers read = group Anonymous Users editTopicName = group Registered Users [access "refs/drafts/*"] push = block group Registered Users [access "refs/for/refs/*"] push = group Registered Users [access "refs/for/refs/zuul/*"] pushMerge = group Continuous Integration Tools [access "refs/heads/*"] label-Code-Review = -2..+2 group Project Bootstrappers label-Code-Review = -1..+1 group Registered Users label-Verified = -2..+2 group Continuous Integration Tools label-Verified = -2..+2 group Project Bootstrappers label-Verified = -1..+1 group Continuous Integration Tools Development label-Verified = -1..+1 group Voting Third-Party CI label-Workflow = -1..+0 group Change Owner label-Workflow = -1..+1 group Project Bootstrappers rebase = group Registered Users submit = group Continuous Integration Tools submit = group Project Bootstrappers [access "refs/meta/config"] read = group Project Owners [access "refs/meta/openstack/*"] create = group Continuous Integration Tools push = group Continuous Integration Tools read = group Continuous Integration Tools [access "refs/zuul/*"] create = group Continuous Integration Tools push = +force group Continuous Integration Tools pushMerge = group Continuous Integration Tools [capability] accessDatabase = group Administrators administrateServer = group Administrators createProject = group Project Bootstrappers emailReviewers = deny group Third-Party CI priority = batch group Service Users runAs = group Project Bootstrappers streamEvents = group Registered Users [contributor-agreement "ICLA"] accepted = group CLA Accepted - ICLA agreementUrl = static/cla.html autoVerify = group CLA Accepted - ICLA description = OpenStack Individual Contributor License Agreement [contributor-agreement "System CLA"] accepted = group System CLA agreementUrl = static/system-cla.html description = DON'T SIGN THIS: System CLA (externally managed) [contributor-agreement "USG CLA"] accepted = group USG CLA agreementUrl = static/usg-cla.html description = DON'T SIGN THIS: U.S. Government CLA (externally managed) [label "Code-Review"] abbreviation = R copyCondition = changekind:TRIVIAL_REBASE OR is:MIN function = NoBlock value = -2 Do not merge value = -1 This patch needs further work before it can be merged value = 0 No score value = +1 Looks good to me, but someone else must approve value = +2 Looks good to me (core reviewer) [submit-requirement "Code-Review"] description = Code reviewed submittableIf = label:Code-Review=MAX AND -label:Code-Review=MIN canOverrideInChildProjects = true [label "Verified"] function = NoBlock value = -2 Fails value = -1 Doesn't seem to work value = 0 No score value = +1 Works for me value = +2 Verified [submit-requirement "Verified"] description = Code verified by Zuul submittableIf = label:Verified=MAX AND -label:Verified=MIN [label "Workflow"] function = NoBlock value = -1 Work in progress value = 0 Ready for reviews value = +1 Approved [submit-requirement "Workflow"] description = Approved by core member submittableIf = label:Workflow=MAX AND -label:Workflow=MIN [plugin "its-storyboard"] enabled = true [project] description = Rights inherited by all other projects [receive] rejectImplicitMerges = true Now edit the groups file. The format is:: #UUID Group Name 1234567890123456789012345678901234567890 group-foo Each of the groups listed above under 'Groups' should have an entry as well as the built in groups such as 'Service Users' which may or may not be present in the initial groups file. You can find the UUID values by navigating to Admin -> Groups -> Group Name -> General in the Web UI. Finally, commit the changes and push the config back up to Gerrit .. code-block:: shell-session $ git commit -am "Initial All-Projects config" $ git push gerrit HEAD:refs/meta/config Manual Administrative Tasks =========================== The following sections describe tasks that individuals with root access may need to perform on rare occasions. Renaming a Project ------------------ Renaming a project is not automated and is disruptive to developers, so it should be avoided. Allow for an hour of downtime for the project in question, and about 10 minutes of downtime for all of Gerrit. All Gerrit changes, merged and open, will carry over, so in-progress changes do not need to be merged before the move. To rename a project: #. Prepare a change to the project-config repo to update things like projects.yaml, Gerrit ACLs, zuul and gerritbot for the new name. #. Prepare a yaml file called repos.yaml that has a single dictionary called `repos` with a list of dictionaries each having an old and new entry. Optionally also add a `gerrit_groups` dict of the same form if groups are being renamed:: repos: - old: stackforge/awesome-repo new: openstack/awesome-repo - old: openstack/foo new: openstack/bar gerrit_groups: - old: old-core-group new: new-core-group Add this file to the ``renames/`` directory in the ``opendev/project-config`` repository. #. An hour in advance of the maintenance (if possible), put ``review02.opendev.org``, ``gitea01-8.opendev.org``, and ``storyboard01.opendev.org`` into the emergency file on bridge. #. Check that all servers involved in the rename playbook (review, zuul-scheduler, storyboard, storyboard-dev, and the giteas) are responding to ssh to ensure the next step can run successfully. #. Run the ansible rename repos playbook, passing in the path to your yaml file .. code-block:: shell-session $ sudo ansible-playbook -f 10 /home/zuul/src/opendev.org/opendev/system-config/playbooks/rename_repos.yaml -e repolist=ABSOLUTE_PATH_TO_VARS_FILE #. :ref:`Force-merge ` the prepared configuration changes. #. Wait for the changes merged above to replicate to the giteas. .. warning:: Not waiting at this step can cause manage-projects to run with our old pre rename state causing the project to be created under its old name. #. Remove ``review02.opendev.org``, ``gitea01-8.opendev.org``, and ``storyboard01.opendev.org`` from the emergency file. #. Ensure that the next manage-projects run does not update the giteas or review servers. It should be a noop. Developers will either need to re-clone a new copy of the repository, or manually update their remotes with something like .. code-block:: shell-session $ git remote set-url origin https://opendev.org/$ORG/$PROJECT Third-Party Testing Access -------------------------- The command to add an account for an automated system which gets -1/+1 code verify voting rights (as outlined in :ref:`third-party-testing`) looks like: .. code-block:: shell-session $ ssh -p 29418 review.opendev.org "gerrit create-account --group 'Third-Party CI' --full-name 'Some CI Bot' --email ci-bot@third-party.org --ssh-key 'ssh-rsa AAAAB3Nz...zaUCse1P ci-bot@third-party.org' some-ci-bot" Details on the create-account_ command can be found in the Gerrit API documentation. .. _create-account: https://review.opendev.org/Documentation/cmd-create-account.html Deleting Accounts in Gerrit --------------------------- We can not delete accounts. They can be made inactive. Duplicate Accounts in Gerrit ---------------------------- If a user has two accounts, we can not combine them. We can only deactivate one of them. For example, user ``foo`` has an account ``foo@company.com`` and moves to a new job, creating a new account ``foo@new.com``. They log-in with ``foo@new.com``, but then realise what they really wanted to do was *add* this new address to their existing account (i.e. ``foo@company.com``). The first step to resolve this is to confirm the ID of the unwanted account, . As an admin user with a HTTP password set, search for the new account: .. code-block:: shell-session $ curl -u you.admin -i -H "Accept: application/json" 'https://review.opendev.org/a/accounts/foo@new.com' That will return an ``_acount_id``. For this example, assume it is ``12345``. The user should check in their settings they are *not* using this account. Clone ``All-Users`` to modify the account, and checkout the account config, which is sharded by the last two digits of the ``_account_id``. .. code-block:: shell-session $ git clone ssh://you.admin@review.opendev.org:29418/All-Users $ git fetch origin refs/users/45/12345 $ git checkout FETCH_HEAD Edit the ``[account]`` section of ``account.conf`` to remove ``preferredEmail`` and have a line ``active = false``. Put your admin account into ``Project Bootstrappers`` (see :ref:`sysadmin`) and commit this .. code-block:: shell-session $ git commit -m "Make duplicate account inactive" --author $ git push origin HEAD:refs/users/45/12345 There will still be an OpenID external ID associated with this now inactive account. This will prevent adding ``foo@new.com`` to another account until this is removed. Check this via the API with .. code-block:: shell-session $ curl -u you.admin -i -H "Accept: application/json" https://review.opendev.org/a/accounts/12345/external.ids This will give a json result with an ``identity`` URL like ``"identity":"https://login.ubuntu.com/+id/RaND0m``. Use this to delete the record with another call .. code-block:: shell-session $ curl -XPOST -u you.admin -i -H "Content-Type: application/json" -d '["https://login.ubuntu.com/+id/RaND0m"]' https://review.opendev.org/a/accounts/12345/external.ids:delete If the user has added email addresses, there may also be ``mailto:`` identity entries for emails the user now wishes to use on their other account. You should remove these with a ``:delete`` call as above. Note that ``username`` external-ids cannot be deleted (Gerrit will error), so new accounts can not reuse the username of old accounts. The user should now be able to add ``foo@new.com`` to their old account. Deactivating a Gerrit account ----------------------------- To deactivate a Gerrit account (use case can be a failing Third Party CI), you must follow that steps: 1. Identify the account ID of the Third Party CI you need to deactivate. Third-Party CI members can be found on: https://review.opendev.org/#/admin/groups/270,members That will give you the name and email of all members. Then you can get the matching numerical account ID with the help of REST API .. code-block:: shell-session $ curl -i -H "Accept: application/json" --digest --user <>:<> -X GET https://review.opendev.org/a/accounts/{email} This will return a JSON dictionary, that will contain _account_id field. 2. Mark the account as inactive using gerrit ssh api, with .. code-block:: shell-session $ ssh -p 29418 review.opendev.org gerrit set-account --inactive {account-id} Alternatively you can use REST API, sending a DELETE for .. code-block:: shell-session $ curl -i -H "Accept: application/json" --digest --user <>:<> -X DELETE https://review.opendev.org/a/accounts/{account-id}/active 3. Check if there are active gerrit ssh connections .. code-block:: shell-session $ ssh -p 29418 review.opendev.org gerrit show-connections -n | grep {account-id} And kill all of them with subsequent .. code-block:: shell-session $ ssh -p 29418 review.opendev.org gerrit close-connection {connection-id} 4. You can check if the account is properly marked as inactive using REST API, sending a GET for .. code-block:: shell-session $ curl -i -H "Accept: application/json" --digest --user <>:<> -X GET https://review.opendev.org/a/accounts/{account-id}/active A 200 return code means the account is active, and 204 means account inactive. Deleting Messages or Comments ----------------------------- Review messages or comments can not be completely deleted, but they can have their content completely replaced with text indicating that they have been deleted. The process is described below, and the actual deletion command requires a reason. You may want to use something like "removed by user request" or other similar text to indicate why the comment was deleted. To manually delete a review comment or message: #. Get an HTTP password for your Gerrit admin account: .. code-block:: shell-session $ ssh -p 29418 username.admin@review.opendev.org gerrit set-account --generate-http-password username.admin #. Use that along with the :git_file:`tools/gerrit-delete-comment.py` script to delete the desired message or comment. Use the script to list the messages or comments in order to obtain their internal IDs, then run it again to delete the specified message or comment. #. To reduce the attack surface, clear your HTTP password: .. code-block:: shell-session $ ssh -p 29418 username.admin@review.opendev.org gerrit set-account --clear-http-password username.admin Generating a Thread Dump for Debugging -------------------------------------- We removed the Java Melody plugin the wake of the Log4Shell vulnerability. This removed an easy way to acquire a thread dump but dumping threads is still possible with java command line tools. You may find yourself wanting to do this if Gerrit is suffering from poor performance or you are trying to debug odd Gerrit behavior. To run ``jstack`` and produce a thread dump do .. code-block:: shell-session root@review02 # docker exec -it gerrit-compose_gerrit_1 bash gerrit@review02 $ ps -ef | grep java # find the Gerrit java process PID gerrit@review02 $ jstack ${PID} > /tmp/dump.yearmonthday Debugging Failed OpenID Logins ------------------------------ OpenID logins can fail for a number of reasons. This document does not aim to comprehensively cover all possibilities, but does try to address some common cases. Contact Site Administrator Failures ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Login failures that redirect users to ``/SignInFailure,SIGN_IN,Contact+site+administrator`` occur for two common reasons. The first is that the account has been disabled. It will need to be reenabled before login can succeed. Second, there may be an email address conflict between multiple accounts. This can happen if users end up with a new OpenID url with the same email address as an existing Gerrit account. The existing Gerrit account may have this email address set as a preferred email address or as an external id. Addressing this usually involves disabling the old account and removing the conflicting email address from the old account. Local Signature Verification Failed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We have seen this occur when Gerrit ends up with what appears to be an invalid OpenID association with our OpenID provider. Inspecting the Apache access logs at ``/var/log/apache2/gerrit-ssl-access.log`` we can confirm the symptoms of this issue. The first thing to look for is a new ``assoc_handle`` value in the URLs logged by apache. Second you should see all login attempts redirect to ``/SignInFailure,SIGN_IN,Local+signature+verification+failed`` after the ``assoc_handle`` update. If these symptoms are present then restarting Gerrit should force Gerrit to generate a new association with the OpenID provider. In theory this new association will be functional and logins will continue working again. We are unsure of why this happens in the first place so it is theoretically possible multiple restarts will be required as we may have consecutive errors.