Drone.io cache strategy after v0.5

Drone provided the standard cache function until v0.4.
Cache function is to use build artifacts which saved in the end of previous build.

It reduce the build time to preserve node_modules for nodejs, bundler gems for ruby.From dron v0.5 disastablished standard cache function, instead provides plugin. Plugin function produces some problems. This article introduces alternative cache function.

Volume Cache Plugin

When you expected to the same behaviour as always, volume-cache plugin is available.

http://plugins.drone.io/drillster/drone-volume-cache/

This plugin is able to save the build artifacts on the arbitrary path which is running agent.

Enabling cache is to put the step of cache restore and rebuild between build as follows.

Because this flag is able to set by administrator only, management will be heavy task. Also trusted flag open docker socket to plugins, so the user can mount on any paths on the agent host.

It is the best choice in small scale usage.

S3 Cache Plugin

s3-cache plugin which uses AWS S3 as cache storage is also available.

http://plugins.drone.io/drone-plugins/drone-s3-cache/

When your drone is hosted on AWS, this plugin is convenient choice, If not, you can choice S3 compatible storages. Minio is the best for you own hosted drone.

https://minio.io/

Minio is able to launch on docker immediately, works fine with drone.
In docker-compose, the settings are as follows.

Minio uses 9000 port as default, however the drone v0.8 uses 9000 port for gRPC to communicate with agent.
To resolve port conflict, launch on other node, or listen port 80.

Minio is simple storage for single user. Because of this, the access and secret key is need to share with all users. But this is not going to matter as much for cache storage.

Setting example is as follows, but the url option uses on document for S3 endpoint setting, “endpoint” is required instead of url option.

Default cache path is ///. Basically it generate the cache on each branch, it reduce effectiveness of cache when in workflow creating branch on every feature requests like a Github Flow. If you specify the path option with a fixed value as below, the cache can be used the same artifact every time.

Organization name less then 3 characters

If your drone connected to github and organization name is less than 3 characters, rebuild-cache will fail because the bucket name violates the validation rule of 3 or more characters. (It will deadlock and die on the rebuild step)
In this case, using path option to avoid this problem.

flush

flush is the function to erase old cache.
Since minio can be seen regular files from the host side, if tmpwatch target include the minio mount path, the user do not have to explicitly write flush_cache.

Conclutions

The cache function has been deleted standard features in v0.5 but now users can freely delete rotten caches by the plugin. Try more drone.

Measuring pending tests for OSS drone.io v0.4

Number of executable tests at same time are as same as registered number of docker nodes. When exceeds the limit, rest of the test will be delayed. Measuring the load is important, because long task will easily block the other many tests.

$ sqlite3 /var/lib/drone/drone.sqlite "select count(*) from jobs where job_status = 'pending';"
2

if you want to get more detailed information,

select repo_owner, repo_name, repo_private, build_branch from jobs inner join builds on jobs.job_build_id = builds.build_id inner join repos on builds.build_repo_id = repos.repo_id where job_status = 'pending'

Job status list and schema are available on github.

I created the bot that is watching the pending tests, and notify to the hipchat.

Encrypting secrets in OSS drone.io v0.4

OSS drone.io works with other services in notifications and deployments.

But once you commit a password or authentication token for other services writes in drone.yml, your sensitive data will be public.
The drone provides “secrets” that encrypting your sensitive data. the official documents describe the way using command line tool, however you can generate the secrets on the Web UI.

Generating secrets

At first, open your repository setting page on the drone, and select the “SECRETS” tab.

Input your secrets under environment node as yaml files like below.

drone-secrets

Generate and copy the output text into “.drone.sec” file on the top level of repository.

Refer the secrets

You can refer the secrets using $$ in .drone.yml. If hipchat notification settings

notify:
  hipchat:
    auth_token: $$HIPCHAT_TOKEN
    room_id_or_name: 'test'
    notify: true

OSS drone.io v0.3 behind reverse proxy

OSS drone is a open source platform version of drone.io

drone v0.4 has breaking changes. Without converting .drone.yml, all tests must fail. One of mitigation is a parallel run with Name Based Virtual Host.

Apache2 setting sample is below.

<VirtualHost *:443>
...snip...
  ProxyRequests Off # NO FORWARD PROXY
  ProxyPass /api/stream/ wss://localhost:8080/api/stream/
  ProxyPassReverse /api/stream/ wss://localhost:8080/api/stream/
  ProxyPass / https://localhost:8080/
  ProxyPassReverse / https://localhost:8080
  ProxyPreserveHost On
</VirtualHost>

drone v0.3 does not support the reverse proxy. X-Forwaeded-Host cannot be interpreted. ProxyReserveHost will pass the Host header from the incoming request to the drone. Authentication callback can be return to the proper url (not localhost).

drone uses the websocket. apache2 needs mod_proxy_wstunnel
if ubuntu

sudo a2enmod proxy_wstunnel

Restarting the apache, v0.3 and v0.4 run on same host.

Reset the webhooks for Drone – Open Source CI Server

Drone is a useful Open Source CI Server. You can host the private and dedicate CI environement.

Once you delete or modify the webhooks on the github (enterprise), never reset the setting from Web UI. NOT YET IMPLEMENTED.

But, we have APIs.

First of all, get the api key from https://your.ci.server/account/profile

my_profile

Let’s curl

$ curl -X POST https://your.ci.server/api/repos/github.com/ashphy/baku?access_token=<INSERT YOUR API KEY>

API Reference: http://readme.drone.io/api/repos/post/

Deactivate is also available.