Integration

Это часть лекции CI/CD за один день,

основная статья - http://wiki.sirmax.noname.com.ua/index.php/CI_CD_1_day

Before start development, we need to configure Jenkins.

Jenkins Job Builder (JJB) takes simple descriptions of Jenkins jobs in YAML or JSON format and uses them to configure Jenkins. You can keep your job descriptions in human readable text format in a version control system to make changes and auditing easier.
It also has a flexible template system, so creating many similarly configured jobs is easy.

For our Jenkins jobs we will provide the following workflow:

User creates JenkinsJob (yaml format)
User commit job into gerrit cicd-jenkins-job repo and send it to review
Jenkins trigger detects "review-created" event in gerrit and run syntax check job wich check yaml syntax
- If job success: Voite +1
- If job fails: Voite -1

In case of "+1" human can review and add +1
Merge
Jenkins trigger detects "merge" event in gerrit and run update job

As result, job defined in yaml and commited to repo will be tested and created in Jenkins.

Create repo in gerrit

Create project, which inherits all permission:

Install, configure and test JJB

Install

Install pip and libs:

apt-get install python-pip python-yaml

pip install  PyYAML python-jenkins

Install JJB

git clone https://git.openstack.org/openstack-infra/jenkins-job-builder
cd jenkins-job-builder/
python setup.py install

Now JJB is installed

# jenkins-jobs

usage: jenkins-jobs [-h] [--conf CONF] [-l LOG_LEVEL] [--ignore-cache]
                    [--flush-cache] [--version] [--allow-empty-variables]
                    {update,test,delete,delete-all} ...
jenkins-jobs: error: too few arguments

Configure JJB

JJB uses config file (defined with --conf option)
For jenkins I'd like to have separate user with separate key, so let's generate rsa key, create user in ldap and add this user to group

Generate key:

#su - jenkins

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/var/lib/jenkins/.ssh/id_rsa):
Created directory '/var/lib/jenkins/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/lib/jenkins/.ssh/id_rsa.
Your public key has been saved in /var/lib/jenkins/.ssh/id_rsa.pub.
The key fingerprint is:
99:6c:fb:38:1e:5c:05:c9:ba:d6:cf:9b:f2:db:f0:a6 jenkins@cicd
The key's randomart image is:
+--[ RSA 2048]----+
|         ...     |
|          o.     |
|         .  .    |
|       ..o .     |
|        So.      |
|       ooo.      |
|       .+  o.    |
|        .+. o=.  |
|       .o..oE=o  |
+-----------------+

Create user in ldap (same password as for all other users):
- Jenkins user:

dn: cn=Jenkins User,ou=cicd,dc=demo
changetype: add
objectClass: top
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
cn: Jenkins User
ou: cicd
uid: jenkins-user
givenName: Jenkins
sn: user1
userPassword: {SSHA}5oLdx/TJdGrRb3Jaz/9JWuFsj59pPoPt
mail: jenkins@jenkins.com

- Add to group:

dn: cn=cicd  administrators,ou=cicd,dc=demo
changetype: modify
add: memberUid
memberUid: jenkins-user

- Run commands:

# ldapmodify < jenkins-user
adding new entry "cn=Jenkins User,ou=cicd,dc=demo"

# ldapmodify < add_jenkins_to_group
modifying entry "cn=cicd  administrators,ou=cicd,dc=demo"

Check in LDAP

Check Login in Jenkins:

Log-in to Gerrit and add key

On this step w have user, which can manage jenkins, pull data from gerrit and vote for pathces

Test JJB

Create config jenkins_jobs.ini

[job_builder]
ignore_cache=True
keep_descriptions=False
include_path=.:scripts:~/git/
recursive=False
exclude=.*:manual:./development
allow_duplicates=False

[jenkins]
user=jenkins-user
password=r00tme
url=http://192.168.56.102:8080
query_plugins_info=True
##### This is deprecated, use job_builder section instead
#ignore_cache=True

Clone cicd-jenkins-job

$ git clone ssh://jenkins-user@192.168.56.102:29418/cicd-jenkins-job
Cloning into 'cicd-jenkins-job'...
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 2 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (2/2), done.
Checking connectivity... done.

Create test job doing nothing just show env
- test.yaml

- job:
    name: 'test'
    project-type: freestyle
    description: |
        Test Job to test JJB dooing nothing
    concurrent: true
    wrappers:
      - timeout:
          timeout: 30
          fail: true
      - timestamps
      - ansicolor:
          colormap: xterm
    logrotate:
      daysToKeep: 1
      numToKeep: -1
      artifactDaysToKeep: -1
      artifactNumToKeep: -1
    properties:
      - heavy-job:
          weight: 1
    parameters:
      - string:
          name: PARAMETER1
          default: master
          description: |
             TEST_PARAMETER
    builders:
      - shell:
          !include-raw './builders/test.sh'
    publishers:
      - archive:
          allow-empty: true
          artifacts: 'artifacts/*.txt'
          latest-only: false

mkdir ./builders

./builders/test.sh

#!/bin/bash



mkdir -p artifacts

env > artifacts/env.txt

Test syntax

jenkins-jobs --conf jenkins_jobs.ini test cicd-jenkins-job/demo/test.yaml

$ jenkins-jobs --conf jenkins_jobs.ini update  cicd-jenkins-job/demo/test.yaml

INFO:root:Updating jobs in ['cicd-jenkins-job/demo/test.yaml'] ([])
/usr/local/lib/python2.7/dist-packages/jenkins/__init__.py:530: DeprecationWarning: get_plugins_info() is deprecated, use get_plugins()
  DeprecationWarning)
WARNING:jenkins_jobs.local_yaml:tag '!include-raw' is deprecated, switch to using '!include-raw:'
INFO:jenkins_jobs.local_yaml:Including file './builders/test.sh' from path '/var/lib/jenkins/tmp/cicd-jenkins-job/demo'
WARNING:root:logrotate is deprecated on jenkins>=1.637, use the property build-discarder on newer jenkins instead
INFO:jenkins_jobs.builder:Number of jobs generated:  1
INFO:jenkins_jobs.builder:Creating jenkins job test
INFO:jenkins_jobs.builder:Cache saved
INFO:root:Number of jobs updated: 1
INFO:jenkins_jobs.builder:Cache saved

Check Jenkins:

Automate IT!

Next step is to automate jenkins update.

Configure Jenkins

Install Gerrit Trigger Plugin and Git Plugin

Restart Jenkins after plugin installation

We need to fix Gerrit Trigger Plugin configuration, it is designed for older versions of Geerrit

Go to trigger configuration

Add server:

No streaming right?

Go to the All-Projects->Access and under Global Capabilities add Stream Events to the LDAP Users group

Test it:

$ ssh  jenkins-user@192.168.56.102 -p 29418 gerrit stream-events
<empty stream!>

Set up +1 and -1 both for Verified and Code Review.
Go to 'Advanced Settings'

Option 'Verified' is deprecated now, so we need to fix plugin configuration.

Replace

 --verified <VERIFIED>

with :

--label 'Verified=<VERIFIED>'

Configure user jenkins-user with ssh key, we need to access to repos

Create Test Job

Define job which should 'test job' before update it

File: test-jenkins-jobs.yaml

- job:
    name: 'test-jenkins-jobs'
    description: 'Test jenkins jobs configuration'
    concurrent: false
    node: 'master'
    builders:
      - shell:
          !include-raw: builders/test-jenkins-jobs/test-jenkins-jobs.sh
    properties:
      - heavy-job:
          weight: 1
    publishers:
      - postbuildscript:
          builders:
            - shell: 'rm jenkins_jobs.ini'
          script-only-if-succeeded: False
    parameters:
      - string:
          name: JJB_USER
          default: 'jenkins-user'
      - string:
          name: JJB_PASS
          default: 'r00tme'
      - string:
          name: JENKINS_URL
          default: 'http://192.168.56.102:8080'
    scm:
      - git:
          branches:
            - $GERRIT_REFSPEC
          remotes:
            - gerrit:
                refspec: $GERRIT_REFSPEC
                url: 'ssh://jenkins-user@192.168.56.102:29418/cicd-jenkins-job'
                credentials-id: 'cf14c3fb-2df7-45b6-9f8f-3f630e1f2c24'
          choosing-strategy: gerrit
          skip-tag: true
    triggers:
      - gerrit:
          trigger-on:
            - patchset-created-event:
                exclude-drafts: true
                exclude-trivial-rebase: true
                exclude-no-code-change: true
          projects:
            - project-compare-type: PLAIN
              project-pattern: 'cicd-jenkins-job'
              branches:
                - branch-compare-type: PLAIN
                  branch-pattern: 'master'
    wrappers:
      - timestamps
      - inject-passwords:
          global: true

Pay your attention on

credentials-id: '37227586-df68-4882-906c-52e3691f7eb2' , it is link to existing credential and differs from on installation to another.

Define builder script

mkdir test-jenkins-jobs

cat test-jenkins-jobs.sh

#!/bin/bash

set -ex


#source /home/jenkins/jjb/bin/activate

cat > jenkins_jobs.ini << EOF
[jenkins]
user=${JJB_USER}
password=${JJB_PASS}
url=${JENKINS_URL}
[job_builder]
ignore_cache=True
recursive=True
EOF

for D in `find .  -maxdepth 1 -type d   ! -path './.git*' ! -path '.'`;
do
  jenkins-jobs --conf jenkins_jobs.ini test ${D}  ${JOBS_LIST}
done

$ git add -A

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

	new file:   builders/test-jenkins-jobs/test-jenkins-jobs.sh
	new file:   builders/test.sh
	new file:   test-jenkins-jobs.yaml
	new file:   test.yaml

Create .gitreview file

[gerrit]
host=192.168.56.102
port=29418
project= cicd-jenkins-job

# git review

remote: Processing changes: new: 1, refs: 1, done
remote:
remote: New Changes:
remote:   http://192.168.56.102:8081/r/6 test jobs
remote:
To ssh://mmaxur@192.168.56.102:29418/cicd-jenkins-job
 * [new branch]      HEAD -> refs/publish/master

Push job to Jenkins Server

# jenkins-jobs --conf jenkins_jobs.ini update  cicd-jenkins-job/demo/test-jenkins-jobs.yaml

ssh  jenkins-user@192.168.56.102 -p 29418 gerrit stream-events






{"uploader":{"name":"Max Mazur","email":"mmaxur@mirantis.com","username":"mmaxur"},"patchSet":{"number":"1","revision":"141ba5020057cc0827f16788ee5a53f3df8faa59","parents":["6e67c6fdac4410088cc5e80a896f474275e52f87"],"ref":"refs/changes/08/8/1","uploader":{"name":"Max Mazur","email":"mmaxur@mirantis.com","username":"mmaxur"},"createdOn":1458875515,"author":{"name":"Max Mazur","email":"mmaxur@mirantis.com","username":"mmaxur"},"isDraft":false,"kind":"REWORK","sizeInsertions":1,"sizeDeletions":0},"change":{"project":"cicd-jenkins-job","branch":"master","id":"Ie7a6cdf7b942c7378b74e65380fab818516ad0fb","number":"8","subject":"test commit 1","owner":{"name":"Max Mazur","email":"mmaxur@mirantis.com","username":"mmaxur"},"url":"http://192.168.56.102:8081/r/8","commitMessage":"test commit 1\n\nChange-Id: Ie7a6cdf7b942c7378b74e65380fab818516ad0fb\n","status":"NEW"},"type":"patchset-created","eventCreatedOn":1458875516}

Check created job:

Job Results:

Create Update Job

Flow it the same as for test job

Configure

Create YAML: update-jenkins-jobs.yaml

- job:
    name: 'update-jenkins-jobs'
    description: 'Update jenkins jobs configuration on jenkins-jobs merge or manually'
    concurrent: true
    node: 'master'
    builders:
      - shell:
          !include-raw: builders/update-jenkins-jobs/update-jenkins-jobs.sh
    parameters:
      - string:
          name: GERRIT_REFSPEC
          default: 'refs/heads/master'
      - string:
          name: JOBS_LIST
          description: 'Space separated list of jobs to update. Will update all jobs if empty'
      - string:
          name: JJB_USER
          default: 'jenkins-user'
      - string:
          name: JJB_PASS
          default: 'r00tme'
      - string:
          name: JENKINS_URL
          default: 'http://192.168.56.102:8080'
    properties:
      - heavy-job:
          weight: 1
    publishers:
      - postbuildscript:
          builders:
            - shell: 'rm jenkins_jobs.ini'
          script-only-if-succeeded: False
    scm:
      - git:
          branches:
            - $GERRIT_REFSPEC
          remotes:
            - gerrit:
                refspec: $GERRIT_REFSPEC
                url: 'ssh://jenkins-demo@192.168.56.102:29418/cicd-jenkins-job'
                credentials-id: 'cf14c3fb-2df7-45b6-9f8f-3f630e1f2c24'
          choosing-strategy: gerrit
          skip-tag: true
    triggers:
      - gerrit:
          trigger-on:
            - change-merged-event
          projects:
            - project-compare-type: PLAIN
              project-pattern: 'cicd-jenkins-job'
              branches:
                - branch-compare-type: PLAIN
                  branch-pattern: 'master'
    wrappers:
      - timestamps
      - inject-passwords:
          global: true

Create builder: update-jenkins-jobs/update-jenkins-jobs.sh

#!/bin/bash
set -ex



cat > jenkins_jobs.ini << EOF
[jenkins]
user=${JJB_USER}
password=${JJB_PASS}
url=${JENKINS_URL}
[job_builder]
ignore_cache=True
recursive=True
EOF

for D in `find .  -maxdepth 1 -type d   ! -path './.git*' ! -path '.'`;
do
  jenkins-jobs --conf jenkins_jobs.ini update ${D}
done

Add 'Update' job to Jenkins:

jenkins-jobs --conf jenkins_jobs.ini update  cicd-jenkins-job/demo/update-jenkins-jobs.yaml

Test

Test update job
- Create test commit
- See how test syntax job works (should set +1)
- Merge
- See how test.sh builder updated

Change, commit, send to review

# git add test.sh

# git commit -m 'test update job'
[master c80dc6a] test update job
 1 file changed, 1 insertion(+), 1 deletion(-)

# git review
remote: Processing changes: new: 1, refs: 1, done
remote:
remote: New Changes:
remote:   http://192.168.56.102:8081/r/10 test update job
remote:
To ssh://mmaxur@192.168.56.102:29418/cicd-jenkins-job
 * [new branch]      HEAD -> refs/publish/master

See created review

Merge it!

Update job was triggered, see console log.

Check changes in gerrit.

See, job is updated, same changes as committed.

Jenkin Gerrit JJB

Содержание