About me

我叫Pham Vu Tuan,我是新加坡的最后一年本科生。这是我第一次参加Google Summer of Code,并为开源组织做出贡献。我很高兴今年夏天做出贡献。


I have GSoC mentors who help me in this projectOleg NenashevSupun Wanniarachchi. Besides that, I also receive great support from developers in remoting projectDevin NusbaumJeff Thompson.


Current versions of Jenkins Remoting are based on the TCP protocol. If it fails, the agent connection and the build fails as well. There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability.

This project aims to develop a plugin in order to add support of a popular message queue/bus technology (Kafka) as a fault-tolerant communication layer in Jenkins.

Why Kafka?

When planning for this project, we want to use traditional message queue system such asActiveMQ或者RabbitMQ. However, after some discussion, we decided to have a try withKafkawith more suitable features with this project:

  • Kafka itself is not a queue like ActiveMQ or RabbitMQ, it is a distributed, replicated commit log. This helps to remove message delivery complexity we have in traditional queue system.

  • We need to support data streaming as a requirement, and Kafka is good at this aspect, which RabbitMQ is lack of.

  • 据说卡夫卡(Kafka)具有更好的可扩展性和良好的支持。

Current State

The project is reaching the end of the first phase and here are things we have achieved so far:

  • Setup project as a set of Docker Compose components: Kafka cluster, Jenkins controller (with plugin) and a custom agent (JAR).

  • Create a PoC with new command transport implementation to support Kafka, which involves of command invocation, RMI, classloading and data streaming.

  • 对远程和詹金斯核心进行必要的更改,以使其可扩展到该项目的使用。必威国际有限公司

  • Decide to use Kafka as a suitable final implementation.

We planned to release an alpha version of this plugin by the end of this phase, but decided to move this release to the second phase because we need to wait for remoting and core patches to be released.

Architecture Overview

The project consists of multiple components:

  • Kafka Client Library - new command transport implementation, producer and consumer client logic.

  • Remoting Kafka Plugin - plugin implementation with KafkaGlobalConfiguration and KafkaComputerLauncher.

  • Remoting Kafka Agent - A custom JAR agent with remoting JAR packaged together with a custom Engine implementation to setup a communication channel with Kafka.

  • All the components are packaged together with Docker Compose.

The below diagram is the overview of the current architecture:远程kafka架构

With this design, controller is not communicating with agent using direct TCP communication anymore, all the communication commands are transferred with Kafka.


1. Kafka Global Configuration


2. Custom agent start up as a JAR

User can start running an agent with the following command:start agent

3. Launch agents with Kafka

launch agent kafka

4. Commands transferred between controller and agent over Kafka

kafka commands

Remoting operations are being executed over Kafka. In the log you may see:

  • Classloading (Classloader.fetch())

  • Log streaming (Pipe.chunk())

5. Run jobs with remoting Kafka

It is possible to run jobs on Agents connected over Kafka远程kafka run job

Next Phase Plan


How to run demo

You can try to run a demo of the plugin by following theinstruction.

About the Author
Pham Vu Tuan

Pham Vu Tuan is a developer from Singapore. He starts contributing to Jenkins from Google Summer of Code 2018 forJenkins Remoting over Message Bus/Queue