Baidu trillion traffic forwarding engine BFE open source

BFE (Baidu Front End, Baidu Unified Front End) is Baidu’s unified seven-tier traffic forwarding platform. The BFE platform is now connected to most of Baidu’s traffic, with daily forwarding requests of nearly 1 trillion, with a peak of more than 10 million QPS. In the 2019 Baidu Spring Festival Red Envelope activities, BFE platform in the super-large user pressure, several flow peaks under the smooth operation, to ensure the smooth progress of the Spring Festival hongbao activities.

As a comprehensive traffic forwarding platform, the BFE platform integrates the following four functions:

Traffic access and forwarding: support slots on HTTP, HTTPS, HTTP/2, QUIC and other protocols, and support strong application layer routing capabilities

Traffic Global Scheduling: Supports global traffic scheduling system, which is composed of extranet traffic scheduling and internal network traffic scheduling

Security and anti-attack: supports a variety of anti-attack capabilities such as blacklisting, fine current limiting and application layer firewall (WAF)

Real-time data analysis: ultra-high dimensional time series reports that support minutes

Baidu trillion traffic forwarding engine BFE open source

As a core component of the BFE platform, the BFE forwarding engine was developed in 2012 and refactored in the Go language in 2014.

BfE offers the following advantages due to its Go language-based, nginx open source software, which is commonly used in the industry:

High research and development efficiency: Go language development efficiency is much higher than the C language (and Lua), but also in terms of code maintainability also has a huge advantage.

The system is highly secure and stable: the Go language does not have the buffer overflow risk inherent in the C language, avoiding a large number of stability and security risks, and for exceptions can be caught, to ensure that the program in the rapid iteration of the situation does not collapse.

There is reason to believe that, in the long run, software systems based on more advanced programming languages will gradually gain competitive advantage. The price of hardware resources such as the CPU will continue to fall rapidly, and development costs, project development risks, system stability/security will become more important decision-making considerations. In this respect, Nginx, which is mainly based on C, will gradually decline, while software based on more advanced programming languages, such as BFE, will gradually become mainstream.

In addition, BFE in the design, in particular, also increased the consideration of enterprise-class application scenarios:

Direct support for forwarding scenarios: Unlike Nginx, which has an evolutionary path from Web Server to Proxy, BFE is designed directly for forwarding scenes, more responsive to the needs of forwarding scenes in terms of forwarding models and forwarding configurations

Multi-tenant support: Multi-tenant reuse is a common demand in the cloud computing scenario. In BFE’s design, built-in multi-tenant support is provided

Structured configuration: BFE configuration design, extensive use of JSON, such as structured approach, easy and related configuration management system docking

Rich monitoring probe: As an industrial-grade software, the requirements of on-line monitoring are fully taken into account in the design of BFE, which exposes thousands of internal state variables out of http

Baidu trillion traffic forwarding engine BFE open source

In order to facilitate the exchange and development of load balancing technology, the BFE forwarding engine was officially open sourced in the summer of 2019.

(https://github.com/baidu/bfe)

BFE is now open source and supports the following important capabilities:

1, mainstream network protocol access

HTTP/HTTPS/SPDY/HTTP2/WebSocket, etc.

Support for TLS/HTTP/WebSocket reverse proxy mode

2. Extensible plug-in framework

Rapidly customize development extensions to meet business customization needs with scalable plug-in frameworks

Please

Rich plug-ins such as built-in rewrites, redirects, traffic modifications, blocking, etc.

3, based on the diversion of requested content

Custom traffic forwarding for complex business scenarios based on domain-specific language shunt rules

Supports a complete set of shunt condition primitives, including conditional primitives based on the requested content (URI/Header/Cookie, etc.) and the request context (IP, protocol, tags, time, etc.).

4, flexible load balancing strategy

Support for cluster-level load balancing and instance-level load balancing for multi-AZ disaster tolerance and overload protection

Built-in weighted polling, weighted minimum number of connections strategy, user session hold based on IP or request content recognition

For details of the BFE open source version and follow-up roadmap, interested users are asked to follow

https://github.com/baidu/bfe, or sign in github to search bfE directly.

BFE forwarding engine research and development process, adhering to Baidu’s excellent research and development tradition, after many years of technology accumulation. BFE has been running steadily at Baidu for many years, and has undergone many baptisms of heavy traffic. Contributing to the community with open source is an important way for Baidu to embody the value of technology. Hope to take the opportunity of BFE open source, and colleagues to learn technology, and build the field of network access open source technology ecology.

Add a Comment

Your email address will not be published. Required fields are marked *